You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "sarvekshayr (via GitHub)" <gi...@apache.org> on 2026/07/25 04:52:02 UTC
[PR] HDDS-15935. Extract ExportFileManager and document container export directory layout [ozone]
sarvekshayr opened a new pull request, #10866:
URL: https://github.com/apache/ozone/pull/10866
## What changes were proposed in this pull request?
- Document on-disk export layout under `{scm.db.dirs}/exports`.
- Centralize markers, TAR paths, startup orphan cleanup, failure cleanup in `ExportFileManager`
Part **3** in splitting [HDDS-15496](https://issues.apache.org/jira/browse/HDDS-15496) https://github.com/apache/ozone/pull/10673 container ID export.
https://github.com/apache/ozone/pull/10813 will rebase to delegate file operations here after this merges.
## What is the link to the Apache JIRA
[HDDS-15935](https://issues.apache.org/jira/browse/HDDS-15935)
## How was this patch tested?
Added tests in `TestExportFileManager`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15935. Extract ExportFileManager and document container export directory layout [ozone]
Posted by "szetszwo (via GitHub)" <gi...@apache.org>.
szetszwo commented on code in PR #10866:
URL: https://github.com/apache/ozone/pull/10866#discussion_r3659716958
##########
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/export/ExportFileManager.java:
##########
@@ -0,0 +1,202 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hdds.scm.container.export;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+import java.util.Objects;
+import java.util.UUID;
+import org.apache.commons.io.FileUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Manages on-disk paths and artifacts for container ID export jobs.
+ *
+ * <p>The export directory ({@code exportDirectory}, typically {@code {scm.db.dirs}/exports})
+ * uses the layout below. The manager gzip-compresses the archive ({@code .tar.gz}) so operators
+ * can stream entries with {@code zcat}
+ *
+ * <pre>
+ * {exportDirectory}/
+ * ├── {jobId}.in-progress
+ * ├── container-ids-{scope}-{timestamp}-{jobId}.tar.gz
+ * └── export_{jobId}/
+ * ├── container-ids-{scope}-{timestamp}-part001.txt
+ * └── ...
+ * </pre>
+ *
+ * <p>{@code export_{jobId}/} holds shard text files while the job appends them into the archive.
+ *
+ * <p><b>When {@code export_{jobId}/} is deleted:</b> the export manager deletes it after the
+ * archive closes successfully, or during {@link #cleanupFailedArtifacts} on failure or cancel.
+ * On startup, {@link #start()} deletes a leftover {@code export_{jobId}/} when no in-progress
+ * marker remains. If the marker still exists, {@link #start()} deletes {@code export_{jobId}/}
+ * together with the marker and any partial archive for that job id.
+ *
+ * <p><b>When {@code .tar.gz} is deleted:</b> {@link #cleanupFailedArtifacts} deletes partial
+ * archives for failed or cancelled jobs. {@link #start()} deletes partial archives for jobs that
+ * still have an in-progress marker. Completed archives remain on disk until the export manager
+ * evicts the job from memory ({@code maxTerminalJobs} in {@code ContainerExportManager}) or an
+ * operator deletes them manually. After SCM restart, in-memory eviction state is lost, so
+ * completed archives persist until manual cleanup.
+ *
+ * <p><b>SCM restart while a job runs:</b> the in-progress marker and {@code export_{jobId}/}
+ * remain on disk, but in-memory job status is lost. {@link #start()} treats the job as incomplete,
+ * removes the marker, workspace, and any partial {@code .tar.gz} for that job id, and the
Review Comment:
We should not allow a partial {@code .tar.gz}.
1. When the job is running, it writes to export_{jobId}/
2. When the job is completed, it creates gz for export_{jobId}/
Step 2 should be:
- create .tar.gz.tmp
- atomic rename tar.gz.tmp to tar.gz.
We may use org.apache.ratis.util.AtomicFileOutputStream.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15935. Extract ExportFileManager and document container export directory layout [ozone]
Posted by "szetszwo (via GitHub)" <gi...@apache.org>.
szetszwo commented on code in PR #10866:
URL: https://github.com/apache/ozone/pull/10866#discussion_r3678476177
##########
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/export/ExportFileManager.java:
##########
@@ -0,0 +1,275 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hdds.scm.container.export;
+
+import java.io.File;
+import java.io.IOException;
+import java.io.RandomAccessFile;
+import java.nio.channels.FileLock;
+import java.nio.channels.OverlappingFileLockException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.List;
+import java.util.Objects;
+import java.util.UUID;
+import org.apache.commons.io.FileUtils;
+import org.apache.ratis.util.AtomicFileOutputStream;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Manages on-disk paths and artifacts for container ID export jobs.
+ *
+ * <p>The export directory ({@code exportDirectory}, typically {@code {scm.db.dirs}/exports})
+ * uses the layout below. The manager gzip-compresses the archive ({@code .tar.gz}) so operators
+ * can stream entries with {@code zcat}.
+ *
+ * <p>While a job runs, shard text files are written under {@code export_{jobId}/}. The archive is
+ * created only after all shards are written. The export manager writes
+ * {@code container-ids-{scope}-{timestamp}.tar.gz.tmp} and atomically renames it to
+ * {@code .tar.gz} on close ({@link AtomicFileOutputStream}), so a partial {@code .tar.gz} is
+ * never visible. {@link #lock()} is used to exclude concurrent writers.
+ *
+ * <pre>
+ * {exportDirectory}/
+ * ├── in_use.lock
+ * ├── {jobId}.in-progress
Review Comment:
The {jobId}.in-progress marker seems not useful. On startup
- Just delete all tar.gz.tmp and export_{jobId}/
- Load all tar.gz and rebuild the state (suppose it has the JobId.)
##########
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/export/ExportFileManager.java:
##########
@@ -0,0 +1,275 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hdds.scm.container.export;
+
+import java.io.File;
+import java.io.IOException;
+import java.io.RandomAccessFile;
+import java.nio.channels.FileLock;
+import java.nio.channels.OverlappingFileLockException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.List;
+import java.util.Objects;
+import java.util.UUID;
+import org.apache.commons.io.FileUtils;
+import org.apache.ratis.util.AtomicFileOutputStream;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Manages on-disk paths and artifacts for container ID export jobs.
+ *
+ * <p>The export directory ({@code exportDirectory}, typically {@code {scm.db.dirs}/exports})
+ * uses the layout below. The manager gzip-compresses the archive ({@code .tar.gz}) so operators
+ * can stream entries with {@code zcat}.
+ *
+ * <p>While a job runs, shard text files are written under {@code export_{jobId}/}. The archive is
+ * created only after all shards are written. The export manager writes
+ * {@code container-ids-{scope}-{timestamp}.tar.gz.tmp} and atomically renames it to
+ * {@code .tar.gz} on close ({@link AtomicFileOutputStream}), so a partial {@code .tar.gz} is
+ * never visible. {@link #lock()} is used to exclude concurrent writers.
+ *
+ * <pre>
+ * {exportDirectory}/
+ * ├── in_use.lock
+ * ├── {jobId}.in-progress
+ * ├── container-ids-{scope}-{timestamp}.tar.gz
+ * ├── container-ids-{scope}-{timestamp}.tar.gz.tmp
+ * └── export_{jobId}/
+ * ├── container-ids-{scope}-{timestamp}-part001.txt
+ * └── ...
+ * </pre>
+ *
+ * <p><b>When {@code export_{jobId}/} is deleted:</b> the export manager deletes it after the
+ * archive is committed, or during {@link #cleanupFailedArtifacts} on failure or cancel.
+ * On startup, {@link #start()} deletes a leftover {@code export_{jobId}/} when no in-progress
+ * marker remains. If the marker still exists, {@link #start()} deletes {@code export_{jobId}/}
+ * together with the marker and any {@code .tar.gz.tmp} for that incomplete job.
+ *
+ * <p><b>When {@code .tar.gz.tmp} is deleted:</b> only the temporary file is removed for
+ * incomplete work; a partial {@code .tar.gz} is never written. {@link #cleanupFailedArtifacts}
+ * deletes {@code .tar.gz.tmp} for failed or cancelled jobs. {@link #start()} deletes
+ * {@code .tar.gz.tmp} for jobs that still have an in-progress marker.
+ *
+ * <p><b>When completed {@code .tar.gz} is deleted:</b> completed archives remain on disk until
+ * the export manager evicts them ({@code maxTerminalJobs} in {@code ContainerExportManager}).
+ *
+ * <p><b>SCM restart:</b> in-memory job status is lost and {@code jobId} cannot be recovered from
+ * the archive file name. {@link #listCompletedArchivePaths()} returns existing {@code tarPath}
Review Comment:
Why not just add the jobId to the name? It is actually useful for debugging.
- container-ids-{scope}-{timestamp}_job{jobId}.tar.gz
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15935. Extract ExportFileManager and document container export directory layout [ozone]
Posted by "szetszwo (via GitHub)" <gi...@apache.org>.
szetszwo commented on code in PR #10866:
URL: https://github.com/apache/ozone/pull/10866#discussion_r3653530786
##########
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/export/ExportFileManager.java:
##########
@@ -0,0 +1,179 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hdds.scm.container.export;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+import java.util.Objects;
+import java.util.UUID;
+import org.apache.commons.io.FileUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Manages on-disk paths and artifacts for container ID export jobs.
+ * Layout under the export directory ({@code {exportDirectory}}, typically {@code {scm.db.dirs}/exports}):
+ * <p>
+ * {exportDirectory}/
+ * {jobId}.in-progress // marker while a job is running
+ * container-ids-{scope}-{timestamp}-{jobId}.tar // completed export archive
+ * export-{jobId}/ // per-job workspace (removed on success)
Review Comment:
- Use `export_{jobId}` since jobId is an uuid.
- When this directory will be deleted?
- What if the SCM is restarted when a job is running?
##########
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/export/ExportFileManager.java:
##########
@@ -0,0 +1,179 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hdds.scm.container.export;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+import java.util.Objects;
+import java.util.UUID;
+import org.apache.commons.io.FileUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Manages on-disk paths and artifacts for container ID export jobs.
+ * Layout under the export directory ({@code {exportDirectory}}, typically {@code {scm.db.dirs}/exports}):
+ * <p>
+ * {exportDirectory}/
Review Comment:
Let's draw it similar the `tree` commend output:
```
% tree
.
├── dev-support
│ ├── checks
│ │ ├── _diffoscope.sh
│ │ ├── _mvn_check.sh
│ │ ├── build.sh
│ │ ├── repro.sh
│ │ └── unit.sh
│ ├── find_maven.sh
│ └── make_rc.sh
```
##########
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/export/ExportFileManager.java:
##########
@@ -0,0 +1,179 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hdds.scm.container.export;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+import java.util.Objects;
+import java.util.UUID;
+import org.apache.commons.io.FileUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Manages on-disk paths and artifacts for container ID export jobs.
+ * Layout under the export directory ({@code {exportDirectory}}, typically {@code {scm.db.dirs}/exports}):
+ * <p>
+ * {exportDirectory}/
+ * {jobId}.in-progress // marker while a job is running
+ * container-ids-{scope}-{timestamp}-{jobId}.tar // completed export archive
+ * export-{jobId}/ // per-job workspace (removed on success)
+ * work/
+ * container-ids-{scope}-{timestamp}-part001.txt
+ * ...
Review Comment:
Why the work directory is needed? Does export_{jobId} contain only `work`?
##########
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/export/ExportFileManager.java:
##########
@@ -0,0 +1,179 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hdds.scm.container.export;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+import java.util.Objects;
+import java.util.UUID;
+import org.apache.commons.io.FileUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Manages on-disk paths and artifacts for container ID export jobs.
+ * Layout under the export directory ({@code {exportDirectory}}, typically {@code {scm.db.dirs}/exports}):
+ * <p>
Review Comment:
Use \<pre>
##########
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/export/ExportFileManager.java:
##########
@@ -0,0 +1,179 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hdds.scm.container.export;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+import java.util.Objects;
+import java.util.UUID;
+import org.apache.commons.io.FileUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Manages on-disk paths and artifacts for container ID export jobs.
+ * Layout under the export directory ({@code {exportDirectory}}, typically {@code {scm.db.dirs}/exports}):
+ * <p>
+ * {exportDirectory}/
+ * {jobId}.in-progress // marker while a job is running
+ * container-ids-{scope}-{timestamp}-{jobId}.tar // completed export archive
+ * export-{jobId}/ // per-job workspace (removed on success)
+ * work/
+ * container-ids-{scope}-{timestamp}-part001.txt
+ * ...
+ * <p>
Review Comment:
Use \</pre>
##########
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/export/ExportFileManager.java:
##########
@@ -0,0 +1,179 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hdds.scm.container.export;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+import java.util.Objects;
+import java.util.UUID;
+import org.apache.commons.io.FileUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Manages on-disk paths and artifacts for container ID export jobs.
+ * Layout under the export directory ({@code {exportDirectory}}, typically {@code {scm.db.dirs}/exports}):
+ * <p>
+ * {exportDirectory}/
+ * {jobId}.in-progress // marker while a job is running
+ * container-ids-{scope}-{timestamp}-{jobId}.tar // completed export archive
Review Comment:
- We should gz it; see GZIPOutputStream.
- Note that gz is better than zip in this case since gz support zcat but zip doesn't.
- When this file will be deleted?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15935. Extract ExportFileManager and document container export directory layout [ozone]
Posted by "szetszwo (via GitHub)" <gi...@apache.org>.
szetszwo commented on code in PR #10866:
URL: https://github.com/apache/ozone/pull/10866#discussion_r3659716958
##########
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/export/ExportFileManager.java:
##########
@@ -0,0 +1,202 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hdds.scm.container.export;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+import java.util.Objects;
+import java.util.UUID;
+import org.apache.commons.io.FileUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Manages on-disk paths and artifacts for container ID export jobs.
+ *
+ * <p>The export directory ({@code exportDirectory}, typically {@code {scm.db.dirs}/exports})
+ * uses the layout below. The manager gzip-compresses the archive ({@code .tar.gz}) so operators
+ * can stream entries with {@code zcat}
+ *
+ * <pre>
+ * {exportDirectory}/
+ * ├── {jobId}.in-progress
+ * ├── container-ids-{scope}-{timestamp}-{jobId}.tar.gz
+ * └── export_{jobId}/
+ * ├── container-ids-{scope}-{timestamp}-part001.txt
+ * └── ...
+ * </pre>
+ *
+ * <p>{@code export_{jobId}/} holds shard text files while the job appends them into the archive.
+ *
+ * <p><b>When {@code export_{jobId}/} is deleted:</b> the export manager deletes it after the
+ * archive closes successfully, or during {@link #cleanupFailedArtifacts} on failure or cancel.
+ * On startup, {@link #start()} deletes a leftover {@code export_{jobId}/} when no in-progress
+ * marker remains. If the marker still exists, {@link #start()} deletes {@code export_{jobId}/}
+ * together with the marker and any partial archive for that job id.
+ *
+ * <p><b>When {@code .tar.gz} is deleted:</b> {@link #cleanupFailedArtifacts} deletes partial
+ * archives for failed or cancelled jobs. {@link #start()} deletes partial archives for jobs that
+ * still have an in-progress marker. Completed archives remain on disk until the export manager
+ * evicts the job from memory ({@code maxTerminalJobs} in {@code ContainerExportManager}) or an
+ * operator deletes them manually. After SCM restart, in-memory eviction state is lost, so
+ * completed archives persist until manual cleanup.
+ *
+ * <p><b>SCM restart while a job runs:</b> the in-progress marker and {@code export_{jobId}/}
+ * remain on disk, but in-memory job status is lost. {@link #start()} treats the job as incomplete,
+ * removes the marker, workspace, and any partial {@code .tar.gz} for that job id, and the
Review Comment:
We should not allow a partial {@code .tar.gz}.
1. When the job is running, it writes to export_{jobId}/
2. When the job is completed, it creates gz for export_{jobId}/
Step 2 should be:
- create .tar.gz.tmp
- atomic rename tar.gz.tmp to tar.gz.
We may use AtomicFileOutputStream.
##########
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/export/ExportFileManager.java:
##########
@@ -0,0 +1,202 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hdds.scm.container.export;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+import java.util.Objects;
+import java.util.UUID;
+import org.apache.commons.io.FileUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Manages on-disk paths and artifacts for container ID export jobs.
+ *
+ * <p>The export directory ({@code exportDirectory}, typically {@code {scm.db.dirs}/exports})
+ * uses the layout below. The manager gzip-compresses the archive ({@code .tar.gz}) so operators
+ * can stream entries with {@code zcat}
+ *
+ * <pre>
+ * {exportDirectory}/
+ * ├── {jobId}.in-progress
Review Comment:
It should not include {jobId}. If different jobs write to a different file, we cannot make it atomic. We should use FileLock. For example, see https://github.com/apache/ratis/blob/master/ratis-server/src/main/java/org/apache/ratis/server/storage/RaftStorageDirectoryImpl.java#L191,L204
##########
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/export/ExportFileManager.java:
##########
@@ -0,0 +1,202 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hdds.scm.container.export;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+import java.util.Objects;
+import java.util.UUID;
+import org.apache.commons.io.FileUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Manages on-disk paths and artifacts for container ID export jobs.
+ *
+ * <p>The export directory ({@code exportDirectory}, typically {@code {scm.db.dirs}/exports})
+ * uses the layout below. The manager gzip-compresses the archive ({@code .tar.gz}) so operators
+ * can stream entries with {@code zcat}
+ *
+ * <pre>
+ * {exportDirectory}/
+ * ├── {jobId}.in-progress
+ * ├── container-ids-{scope}-{timestamp}-{jobId}.tar.gz
+ * └── export_{jobId}/
+ * ├── container-ids-{scope}-{timestamp}-part001.txt
+ * └── ...
+ * </pre>
+ *
+ * <p>{@code export_{jobId}/} holds shard text files while the job appends them into the archive.
+ *
+ * <p><b>When {@code export_{jobId}/} is deleted:</b> the export manager deletes it after the
+ * archive closes successfully, or during {@link #cleanupFailedArtifacts} on failure or cancel.
+ * On startup, {@link #start()} deletes a leftover {@code export_{jobId}/} when no in-progress
+ * marker remains. If the marker still exists, {@link #start()} deletes {@code export_{jobId}/}
+ * together with the marker and any partial archive for that job id.
+ *
+ * <p><b>When {@code .tar.gz} is deleted:</b> {@link #cleanupFailedArtifacts} deletes partial
+ * archives for failed or cancelled jobs. {@link #start()} deletes partial archives for jobs that
+ * still have an in-progress marker. Completed archives remain on disk until the export manager
+ * evicts the job from memory ({@code maxTerminalJobs} in {@code ContainerExportManager}) or an
+ * operator deletes them manually. After SCM restart, in-memory eviction state is lost, so
+ * completed archives persist until manual cleanup.
Review Comment:
When SCM restart, could it count the number of gz files and rebuild the in-memory eviction state?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15935. Extract ExportFileManager and document container export directory layout [ozone]
Posted by "szetszwo (via GitHub)" <gi...@apache.org>.
szetszwo merged PR #10866:
URL: https://github.com/apache/ozone/pull/10866
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15935. Extract ExportFileManager and document container export directory layout [ozone]
Posted by "szetszwo (via GitHub)" <gi...@apache.org>.
szetszwo commented on code in PR #10866:
URL: https://github.com/apache/ozone/pull/10866#discussion_r3692044912
##########
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/export/ExportFileManager.java:
##########
@@ -0,0 +1,230 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hdds.scm.container.export;
+
+import java.io.File;
+import java.io.IOException;
+import java.io.RandomAccessFile;
+import java.nio.channels.FileLock;
+import java.nio.channels.OverlappingFileLockException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.List;
+import java.util.Objects;
+import java.util.UUID;
+import org.apache.commons.io.FileUtils;
+import org.apache.ratis.util.AtomicFileOutputStream;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Manages on-disk paths and artifacts for container ID export jobs.
+ *
+ * <p>The export directory ({@code exportDirectory}, typically {@code {scm.db.dirs}/exports})
+ * uses the layout below. The manager gzip-compresses the archive ({@code .tar.gz}) so operators
+ * can stream entries with {@code zcat}.
+ *
+ * <p>While a job runs, shard text files are written under {@code export_{jobId}/}. The archive is
+ * created only after all shards are written. The export manager writes
+ * {@code container-ids-{scope}-{timestamp}_job{jobId}.tar.gz.tmp} and atomically renames it to
+ * {@code .tar.gz} on close ({@link AtomicFileOutputStream}), so a partial {@code .tar.gz} is
+ * never visible. {@link #lock()} uses {@code in_use.lock} to exclude concurrent writers.
+ *
+ * <pre>
+ * {exportDirectory}/
+ * ├── in_use.lock
+ * ├── container-ids-{scope}-{timestamp}_job{jobId}.tar.gz
+ * ├── container-ids-{scope}-{timestamp}_job{jobId}.tar.gz.tmp
+ * └── export_{jobId}/
+ * ├── container-ids-{scope}-{timestamp}-part001.txt
+ * └── ...
+ * </pre>
Review Comment:
The layout looks great now! Also, the javadoc is well written. 👍
##########
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/export/ExportFileManager.java:
##########
@@ -0,0 +1,230 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hdds.scm.container.export;
+
+import java.io.File;
+import java.io.IOException;
+import java.io.RandomAccessFile;
+import java.nio.channels.FileLock;
+import java.nio.channels.OverlappingFileLockException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.List;
+import java.util.Objects;
+import java.util.UUID;
+import org.apache.commons.io.FileUtils;
+import org.apache.ratis.util.AtomicFileOutputStream;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Manages on-disk paths and artifacts for container ID export jobs.
+ *
+ * <p>The export directory ({@code exportDirectory}, typically {@code {scm.db.dirs}/exports})
+ * uses the layout below. The manager gzip-compresses the archive ({@code .tar.gz}) so operators
+ * can stream entries with {@code zcat}.
+ *
+ * <p>While a job runs, shard text files are written under {@code export_{jobId}/}. The archive is
+ * created only after all shards are written. The export manager writes
+ * {@code container-ids-{scope}-{timestamp}_job{jobId}.tar.gz.tmp} and atomically renames it to
+ * {@code .tar.gz} on close ({@link AtomicFileOutputStream}), so a partial {@code .tar.gz} is
+ * never visible. {@link #lock()} uses {@code in_use.lock} to exclude concurrent writers.
+ *
+ * <pre>
+ * {exportDirectory}/
+ * ├── in_use.lock
+ * ├── container-ids-{scope}-{timestamp}_job{jobId}.tar.gz
+ * ├── container-ids-{scope}-{timestamp}_job{jobId}.tar.gz.tmp
+ * └── export_{jobId}/
+ * ├── container-ids-{scope}-{timestamp}-part001.txt
+ * └── ...
+ * </pre>
+ *
+ * <p><b>Incomplete work</b> ({@code export_{jobId}/} and {@code .tar.gz.tmp}) is removed by
+ * {@link #cleanupFailedJob(Path, File)} on failure or cancel, and by {@link #start()} for every
+ * leftover directory and temp file after SCM restart. Completed {@code .tar.gz} files are kept.
+ *
+ * <p><b>Completed {@code .tar.gz}</b> remains on disk until the export manager evicts it
+ * ({@code maxTerminalJobs} in {@code ContainerExportManager}) via {@link #deleteExportTar(String)}.
+ *
+ * <p><b>SCM restart:</b> in-memory job status is lost. {@link #start()} clears incomplete work;
+ * {@link #listCompletedArchivePaths()} returns existing {@code tarPath} values (oldest first);
+ * {@link #jobIdFromArchiveFileName(String)} parses {@code jobId} for terminal-job rebuild in
+ * {@code ContainerExportManager}.
+ */
+final class ExportFileManager {
+
+ private static final Logger LOG = LoggerFactory.getLogger(ExportFileManager.class);
+
+ static final String EXPORT_JOB_DIR_PREFIX = "export_";
+ static final String EXPORT_ARCHIVE_JOB_INFIX = "_job";
+ static final String EXPORT_ARCHIVE_SUFFIX = ".tar.gz";
+ static final String EXPORT_ARCHIVE_TMP_SUFFIX = EXPORT_ARCHIVE_SUFFIX + AtomicFileOutputStream.TMP_EXTENSION;
+ static final String EXPORT_LOCK_NAME = "in_use.lock";
+
+ private final String exportDirectory;
+ private FileLock exportDirectoryLock;
+
+ ExportFileManager(String exportDirectory) {
+ this.exportDirectory = Objects.requireNonNull(exportDirectory, "exportDirectory == null");
+ }
+
+ String getExportDirectory() {
+ return exportDirectory;
+ }
+
+ void start() throws IOException {
+ Files.createDirectories(Paths.get(exportDirectory));
+ removeIncompleteWorkOnStartup();
+ }
+
+ void lock() throws IOException {
+ if (exportDirectoryLock != null) {
+ return;
+ }
+ File lockFile = new File(exportDirectory, EXPORT_LOCK_NAME);
+ RandomAccessFile lockAccessFile = new RandomAccessFile(lockFile, "rws");
+ try {
+ FileLock lock = lockAccessFile.getChannel().tryLock();
+ if (lock == null) {
+ lockAccessFile.close();
+ throw new OverlappingFileLockException();
+ }
+ exportDirectoryLock = lock;
+ LOG.debug("Acquired container export directory lock {}", lockFile.getAbsolutePath());
+ } catch (OverlappingFileLockException | IOException e) {
+ lockAccessFile.close();
+ throw new IOException("Failed to lock container export directory " + exportDirectory, e);
+ }
+ }
+
+ void unlock() throws IOException {
+ if (exportDirectoryLock == null) {
+ return;
+ }
+ exportDirectoryLock.release();
+ exportDirectoryLock.channel().close();
+ exportDirectoryLock = null;
+ }
+
+ File resolveArchiveFile(ExportScope scope, String fileTimestamp, String jobId) {
Review Comment:
Create the ExportJob.Id class and use it here.
##########
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/export/ExportFileManager.java:
##########
@@ -0,0 +1,230 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hdds.scm.container.export;
+
+import java.io.File;
+import java.io.IOException;
+import java.io.RandomAccessFile;
+import java.nio.channels.FileLock;
+import java.nio.channels.OverlappingFileLockException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.List;
+import java.util.Objects;
+import java.util.UUID;
+import org.apache.commons.io.FileUtils;
+import org.apache.ratis.util.AtomicFileOutputStream;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Manages on-disk paths and artifacts for container ID export jobs.
+ *
+ * <p>The export directory ({@code exportDirectory}, typically {@code {scm.db.dirs}/exports})
+ * uses the layout below. The manager gzip-compresses the archive ({@code .tar.gz}) so operators
+ * can stream entries with {@code zcat}.
+ *
+ * <p>While a job runs, shard text files are written under {@code export_{jobId}/}. The archive is
+ * created only after all shards are written. The export manager writes
+ * {@code container-ids-{scope}-{timestamp}_job{jobId}.tar.gz.tmp} and atomically renames it to
+ * {@code .tar.gz} on close ({@link AtomicFileOutputStream}), so a partial {@code .tar.gz} is
+ * never visible. {@link #lock()} uses {@code in_use.lock} to exclude concurrent writers.
+ *
+ * <pre>
+ * {exportDirectory}/
+ * ├── in_use.lock
+ * ├── container-ids-{scope}-{timestamp}_job{jobId}.tar.gz
+ * ├── container-ids-{scope}-{timestamp}_job{jobId}.tar.gz.tmp
+ * └── export_{jobId}/
+ * ├── container-ids-{scope}-{timestamp}-part001.txt
+ * └── ...
+ * </pre>
+ *
+ * <p><b>Incomplete work</b> ({@code export_{jobId}/} and {@code .tar.gz.tmp}) is removed by
+ * {@link #cleanupFailedJob(Path, File)} on failure or cancel, and by {@link #start()} for every
+ * leftover directory and temp file after SCM restart. Completed {@code .tar.gz} files are kept.
+ *
+ * <p><b>Completed {@code .tar.gz}</b> remains on disk until the export manager evicts it
+ * ({@code maxTerminalJobs} in {@code ContainerExportManager}) via {@link #deleteExportTar(String)}.
+ *
+ * <p><b>SCM restart:</b> in-memory job status is lost. {@link #start()} clears incomplete work;
+ * {@link #listCompletedArchivePaths()} returns existing {@code tarPath} values (oldest first);
+ * {@link #jobIdFromArchiveFileName(String)} parses {@code jobId} for terminal-job rebuild in
+ * {@code ContainerExportManager}.
+ */
+final class ExportFileManager {
+
+ private static final Logger LOG = LoggerFactory.getLogger(ExportFileManager.class);
+
+ static final String EXPORT_JOB_DIR_PREFIX = "export_";
+ static final String EXPORT_ARCHIVE_JOB_INFIX = "_job";
+ static final String EXPORT_ARCHIVE_SUFFIX = ".tar.gz";
+ static final String EXPORT_ARCHIVE_TMP_SUFFIX = EXPORT_ARCHIVE_SUFFIX + AtomicFileOutputStream.TMP_EXTENSION;
+ static final String EXPORT_LOCK_NAME = "in_use.lock";
+
+ private final String exportDirectory;
+ private FileLock exportDirectoryLock;
+
+ ExportFileManager(String exportDirectory) {
+ this.exportDirectory = Objects.requireNonNull(exportDirectory, "exportDirectory == null");
+ }
+
+ String getExportDirectory() {
+ return exportDirectory;
+ }
+
+ void start() throws IOException {
+ Files.createDirectories(Paths.get(exportDirectory));
+ removeIncompleteWorkOnStartup();
+ }
+
+ void lock() throws IOException {
+ if (exportDirectoryLock != null) {
+ return;
+ }
+ File lockFile = new File(exportDirectory, EXPORT_LOCK_NAME);
+ RandomAccessFile lockAccessFile = new RandomAccessFile(lockFile, "rws");
+ try {
+ FileLock lock = lockAccessFile.getChannel().tryLock();
+ if (lock == null) {
+ lockAccessFile.close();
+ throw new OverlappingFileLockException();
+ }
+ exportDirectoryLock = lock;
+ LOG.debug("Acquired container export directory lock {}", lockFile.getAbsolutePath());
+ } catch (OverlappingFileLockException | IOException e) {
+ lockAccessFile.close();
+ throw new IOException("Failed to lock container export directory " + exportDirectory, e);
+ }
+ }
+
+ void unlock() throws IOException {
+ if (exportDirectoryLock == null) {
+ return;
+ }
+ exportDirectoryLock.release();
+ exportDirectoryLock.channel().close();
+ exportDirectoryLock = null;
+ }
+
+ File resolveArchiveFile(ExportScope scope, String fileTimestamp, String jobId) {
+ return new File(exportDirectory, String.format("container-ids-%s-%s%s%s%s",
+ scope.getValue(), fileTimestamp, EXPORT_ARCHIVE_JOB_INFIX, jobId, EXPORT_ARCHIVE_SUFFIX));
+ }
+
+ File resolveArchiveTempFile(ExportScope scope, String fileTimestamp, String jobId) {
+ return AtomicFileOutputStream.getTemporaryFile(resolveArchiveFile(scope, fileTimestamp, jobId));
+ }
+
+ /**
+ * Returns completed archive paths ({@code tarPath} in {@code ExportJob.Status}), oldest first.
+ */
+ List<String> listCompletedArchivePaths() {
+ File exportDir = new File(exportDirectory);
+ File[] matches = exportDir.listFiles((dir, fileName) -> fileName.endsWith(EXPORT_ARCHIVE_SUFFIX)
+ && !fileName.endsWith(EXPORT_ARCHIVE_TMP_SUFFIX));
+ if (matches == null || matches.length == 0) {
+ return Collections.emptyList();
+ }
+ Arrays.sort(matches, Comparator.comparingLong(File::lastModified));
+ List<String> archivePaths = new ArrayList<>(matches.length);
+ for (File archive : matches) {
+ archivePaths.add(archive.getAbsolutePath());
+ }
+ return archivePaths;
+ }
+
+ static String jobIdFromArchiveFileName(String fileName) {
+ if (!fileName.endsWith(EXPORT_ARCHIVE_SUFFIX)) {
+ return null;
+ }
+ String nameWithoutSuffix = fileName.substring(0, fileName.length() - EXPORT_ARCHIVE_SUFFIX.length());
+ int jobIndex = nameWithoutSuffix.lastIndexOf(EXPORT_ARCHIVE_JOB_INFIX);
+ if (jobIndex < 0) {
+ return null;
+ }
+ String jobId = nameWithoutSuffix.substring(jobIndex + EXPORT_ARCHIVE_JOB_INFIX.length());
+ return isUuidDirectoryName(jobId) ? jobId : null;
+ }
+
+ void deleteExportTar(String tarPath) {
+ if (tarPath == null) {
+ return;
+ }
+ File archive = new File(tarPath);
+ if (archive.isFile() && FileUtils.deleteQuietly(archive)) {
+ LOG.debug("Removed container export archive: {}", archive.getName());
+ }
+ FileUtils.deleteQuietly(AtomicFileOutputStream.getTemporaryFile(archive));
+ }
+
+ void cleanupFailedJob(Path jobDir, File archiveFile) {
+ if (jobDir != null) {
+ FileUtils.deleteQuietly(jobDir.toFile());
+ }
+ if (archiveFile != null) {
+ FileUtils.deleteQuietly(AtomicFileOutputStream.getTemporaryFile(archiveFile));
+ }
+ }
+
+ private void removeIncompleteWorkOnStartup() {
+ File exportDir = new File(exportDirectory);
+ File[] children = exportDir.listFiles();
+ if (children != null) {
+ for (File child : children) {
+ if (child.isDirectory() && jobIdFromExportDirName(child.getName()) != null) {
+ FileUtils.deleteQuietly(child);
+ LOG.debug("Removed incomplete container export job directory: {}", child.getAbsolutePath());
+ }
+ }
+ }
+ File[] tempFiles = exportDir.listFiles((dir, fileName) -> fileName.endsWith(EXPORT_ARCHIVE_TMP_SUFFIX));
+ if (tempFiles != null) {
+ for (File tempFile : tempFiles) {
+ if (FileUtils.deleteQuietly(tempFile)) {
+ LOG.debug("Removed incomplete container export archive temp file: {}", tempFile.getName());
+ }
+ }
+ }
+ }
+
+ static String exportJobDirName(String jobId) {
+ return EXPORT_JOB_DIR_PREFIX + jobId;
+ }
+
+ private static String jobIdFromExportDirName(String dirName) {
+ if (!dirName.startsWith(EXPORT_JOB_DIR_PREFIX)) {
+ return null;
+ }
+ String jobId = dirName.substring(EXPORT_JOB_DIR_PREFIX.length());
+ return isUuidDirectoryName(jobId) ? jobId : null;
+ }
+
+ private static boolean isUuidDirectoryName(String directoryName) {
+ try {
+ return directoryName.equals(UUID.fromString(directoryName).toString());
+ } catch (IllegalArgumentException e) {
+ return false;
+ }
+ }
Review Comment:
Move it to UUIDUtil.
##########
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/export/ExportScope.java:
##########
@@ -0,0 +1,74 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hdds.scm.container.export;
+
+import org.apache.hadoop.hdds.protocol.proto.HddsProtos.LifeCycleState;
+import org.apache.hadoop.hdds.scm.container.ContainerHealthState;
+
+/**
+ * Container listing filters for an export job.
+ * An export job filters containers by {@link ContainerHealthState}, {@link LifeCycleState} or both.
+ * Example archive name:
+ * {@code container-ids-health-MISSING_lifecycle-OPEN-20260101T120000Z_job{jobId}.tar.gz}
+ */
+public final class ExportScope {
+
+ private final LifeCycleState lifeCycleState;
+ private final ContainerHealthState healthState;
+ private final String value;
+
+ private ExportScope(LifeCycleState lifeCycleState, ContainerHealthState healthState, String value) {
+ this.lifeCycleState = lifeCycleState;
+ this.healthState = healthState;
+ this.value = value;
+ }
+
+ public static ExportScope of(LifeCycleState lifeCycleState, ContainerHealthState healthState) {
+ StringBuilder sb = new StringBuilder();
+ if (healthState != null) {
+ sb.append("health-").append(healthState.name());
+ }
+ if (lifeCycleState != null) {
+ if (sb.length() > 0) {
+ sb.append('_');
+ }
+ sb.append("lifecycle-").append(lifeCycleState.name());
+ }
Review Comment:
For null, let's use "ANY". Then, the string is much easier to parse.
##########
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/export/ExportFileManager.java:
##########
@@ -0,0 +1,230 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hdds.scm.container.export;
+
+import java.io.File;
+import java.io.IOException;
+import java.io.RandomAccessFile;
+import java.nio.channels.FileLock;
+import java.nio.channels.OverlappingFileLockException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.List;
+import java.util.Objects;
+import java.util.UUID;
+import org.apache.commons.io.FileUtils;
+import org.apache.ratis.util.AtomicFileOutputStream;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Manages on-disk paths and artifacts for container ID export jobs.
+ *
+ * <p>The export directory ({@code exportDirectory}, typically {@code {scm.db.dirs}/exports})
+ * uses the layout below. The manager gzip-compresses the archive ({@code .tar.gz}) so operators
+ * can stream entries with {@code zcat}.
+ *
+ * <p>While a job runs, shard text files are written under {@code export_{jobId}/}. The archive is
+ * created only after all shards are written. The export manager writes
+ * {@code container-ids-{scope}-{timestamp}_job{jobId}.tar.gz.tmp} and atomically renames it to
+ * {@code .tar.gz} on close ({@link AtomicFileOutputStream}), so a partial {@code .tar.gz} is
+ * never visible. {@link #lock()} uses {@code in_use.lock} to exclude concurrent writers.
+ *
+ * <pre>
+ * {exportDirectory}/
+ * ├── in_use.lock
+ * ├── container-ids-{scope}-{timestamp}_job{jobId}.tar.gz
+ * ├── container-ids-{scope}-{timestamp}_job{jobId}.tar.gz.tmp
+ * └── export_{jobId}/
+ * ├── container-ids-{scope}-{timestamp}-part001.txt
+ * └── ...
+ * </pre>
+ *
+ * <p><b>Incomplete work</b> ({@code export_{jobId}/} and {@code .tar.gz.tmp}) is removed by
+ * {@link #cleanupFailedJob(Path, File)} on failure or cancel, and by {@link #start()} for every
+ * leftover directory and temp file after SCM restart. Completed {@code .tar.gz} files are kept.
+ *
+ * <p><b>Completed {@code .tar.gz}</b> remains on disk until the export manager evicts it
+ * ({@code maxTerminalJobs} in {@code ContainerExportManager}) via {@link #deleteExportTar(String)}.
+ *
+ * <p><b>SCM restart:</b> in-memory job status is lost. {@link #start()} clears incomplete work;
+ * {@link #listCompletedArchivePaths()} returns existing {@code tarPath} values (oldest first);
+ * {@link #jobIdFromArchiveFileName(String)} parses {@code jobId} for terminal-job rebuild in
+ * {@code ContainerExportManager}.
+ */
+final class ExportFileManager {
+
+ private static final Logger LOG = LoggerFactory.getLogger(ExportFileManager.class);
+
+ static final String EXPORT_JOB_DIR_PREFIX = "export_";
+ static final String EXPORT_ARCHIVE_JOB_INFIX = "_job";
+ static final String EXPORT_ARCHIVE_SUFFIX = ".tar.gz";
+ static final String EXPORT_ARCHIVE_TMP_SUFFIX = EXPORT_ARCHIVE_SUFFIX + AtomicFileOutputStream.TMP_EXTENSION;
+ static final String EXPORT_LOCK_NAME = "in_use.lock";
+
+ private final String exportDirectory;
+ private FileLock exportDirectoryLock;
+
+ ExportFileManager(String exportDirectory) {
+ this.exportDirectory = Objects.requireNonNull(exportDirectory, "exportDirectory == null");
+ }
+
+ String getExportDirectory() {
+ return exportDirectory;
+ }
+
+ void start() throws IOException {
+ Files.createDirectories(Paths.get(exportDirectory));
+ removeIncompleteWorkOnStartup();
+ }
+
+ void lock() throws IOException {
+ if (exportDirectoryLock != null) {
+ return;
+ }
+ File lockFile = new File(exportDirectory, EXPORT_LOCK_NAME);
+ RandomAccessFile lockAccessFile = new RandomAccessFile(lockFile, "rws");
+ try {
+ FileLock lock = lockAccessFile.getChannel().tryLock();
+ if (lock == null) {
+ lockAccessFile.close();
+ throw new OverlappingFileLockException();
+ }
+ exportDirectoryLock = lock;
+ LOG.debug("Acquired container export directory lock {}", lockFile.getAbsolutePath());
+ } catch (OverlappingFileLockException | IOException e) {
+ lockAccessFile.close();
+ throw new IOException("Failed to lock container export directory " + exportDirectory, e);
+ }
+ }
+
+ void unlock() throws IOException {
+ if (exportDirectoryLock == null) {
+ return;
+ }
+ exportDirectoryLock.release();
+ exportDirectoryLock.channel().close();
+ exportDirectoryLock = null;
+ }
+
+ File resolveArchiveFile(ExportScope scope, String fileTimestamp, String jobId) {
+ return new File(exportDirectory, String.format("container-ids-%s-%s%s%s%s",
+ scope.getValue(), fileTimestamp, EXPORT_ARCHIVE_JOB_INFIX, jobId, EXPORT_ARCHIVE_SUFFIX));
+ }
+
+ File resolveArchiveTempFile(ExportScope scope, String fileTimestamp, String jobId) {
+ return AtomicFileOutputStream.getTemporaryFile(resolveArchiveFile(scope, fileTimestamp, jobId));
+ }
+
+ /**
+ * Returns completed archive paths ({@code tarPath} in {@code ExportJob.Status}), oldest first.
+ */
+ List<String> listCompletedArchivePaths() {
+ File exportDir = new File(exportDirectory);
+ File[] matches = exportDir.listFiles((dir, fileName) -> fileName.endsWith(EXPORT_ARCHIVE_SUFFIX)
+ && !fileName.endsWith(EXPORT_ARCHIVE_TMP_SUFFIX));
+ if (matches == null || matches.length == 0) {
+ return Collections.emptyList();
+ }
+ Arrays.sort(matches, Comparator.comparingLong(File::lastModified));
Review Comment:
It should use the timestamp in the filename.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15935. Extract ExportFileManager and document container export directory layout [ozone]
Posted by "szetszwo (via GitHub)" <gi...@apache.org>.
szetszwo commented on code in PR #10866:
URL: https://github.com/apache/ozone/pull/10866#discussion_r3695212689
##########
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/export/ExportFileManager.java:
##########
@@ -0,0 +1,234 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hdds.scm.container.export;
+
+import java.io.File;
+import java.io.IOException;
+import java.io.RandomAccessFile;
+import java.nio.channels.FileLock;
+import java.nio.channels.OverlappingFileLockException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.List;
+import java.util.Objects;
+import org.apache.commons.io.FileUtils;
+import org.apache.hadoop.ozone.util.UUIDUtil;
+import org.apache.ratis.util.AtomicFileOutputStream;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Manages on-disk paths and artifacts for container ID export jobs.
+ *
+ * <p>The export directory ({@code exportDirectory}, typically {@code {scm.db.dirs}/exports})
+ * uses the layout below. The manager gzip-compresses the archive ({@code .tar.gz}) so operators
+ * can stream entries with {@code zcat}.
+ *
+ * <p>While a job runs, shard text files are written under {@code export_{jobId}/}. The archive is
+ * created only after all shards are written. The export manager writes
+ * {@code container-ids-{scope}-{timestamp}_job{jobId}.tar.gz.tmp} and atomically renames it to
+ * {@code .tar.gz} on close ({@link AtomicFileOutputStream}), so a partial {@code .tar.gz} is
+ * never visible. {@link #lock()} uses {@code in_use.lock} to exclude concurrent writers.
+ *
+ * <pre>
+ * {exportDirectory}/
+ * ├── in_use.lock
+ * ├── container-ids-{scope}-{timestamp}_job{jobId}.tar.gz
+ * ├── container-ids-{scope}-{timestamp}_job{jobId}.tar.gz.tmp
+ * └── export_{jobId}/
+ * ├── container-ids-{scope}-{timestamp}-part001.txt
Review Comment:
Let's use `_`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15935. Extract ExportFileManager and document container export directory layout [ozone]
Posted by "sarvekshayr (via GitHub)" <gi...@apache.org>.
sarvekshayr commented on code in PR #10866:
URL: https://github.com/apache/ozone/pull/10866#discussion_r3694573350
##########
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/export/ExportFileManager.java:
##########
@@ -0,0 +1,234 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hdds.scm.container.export;
+
+import java.io.File;
+import java.io.IOException;
+import java.io.RandomAccessFile;
+import java.nio.channels.FileLock;
+import java.nio.channels.OverlappingFileLockException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.List;
+import java.util.Objects;
+import org.apache.commons.io.FileUtils;
+import org.apache.hadoop.ozone.util.UUIDUtil;
+import org.apache.ratis.util.AtomicFileOutputStream;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Manages on-disk paths and artifacts for container ID export jobs.
+ *
+ * <p>The export directory ({@code exportDirectory}, typically {@code {scm.db.dirs}/exports})
+ * uses the layout below. The manager gzip-compresses the archive ({@code .tar.gz}) so operators
+ * can stream entries with {@code zcat}.
+ *
+ * <p>While a job runs, shard text files are written under {@code export_{jobId}/}. The archive is
+ * created only after all shards are written. The export manager writes
+ * {@code container-ids-{scope}-{timestamp}_job{jobId}.tar.gz.tmp} and atomically renames it to
+ * {@code .tar.gz} on close ({@link AtomicFileOutputStream}), so a partial {@code .tar.gz} is
+ * never visible. {@link #lock()} uses {@code in_use.lock} to exclude concurrent writers.
+ *
+ * <pre>
+ * {exportDirectory}/
+ * ├── in_use.lock
+ * ├── container-ids-{scope}-{timestamp}_job{jobId}.tar.gz
+ * ├── container-ids-{scope}-{timestamp}_job{jobId}.tar.gz.tmp
+ * └── export_{jobId}/
+ * ├── container-ids-{scope}-{timestamp}-part001.txt
Review Comment:
For `scope` component, should health and lifecycle states be joined with `_` or `-`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15935. Extract ExportFileManager and document container export directory layout [ozone]
Posted by "sarvekshayr (via GitHub)" <gi...@apache.org>.
sarvekshayr commented on code in PR #10866:
URL: https://github.com/apache/ozone/pull/10866#discussion_r3654586432
##########
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/export/ExportFileManager.java:
##########
@@ -0,0 +1,179 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hdds.scm.container.export;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+import java.util.Objects;
+import java.util.UUID;
+import org.apache.commons.io.FileUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Manages on-disk paths and artifacts for container ID export jobs.
+ * Layout under the export directory ({@code {exportDirectory}}, typically {@code {scm.db.dirs}/exports}):
+ * <p>
+ * {exportDirectory}/
+ * {jobId}.in-progress // marker while a job is running
+ * container-ids-{scope}-{timestamp}-{jobId}.tar // completed export archive
+ * export-{jobId}/ // per-job workspace (removed on success)
+ * work/
+ * container-ids-{scope}-{timestamp}-part001.txt
+ * ...
Review Comment:
Removed `work` directory.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15935. Extract ExportFileManager and document container export directory layout [ozone]
Posted by "szetszwo (via GitHub)" <gi...@apache.org>.
szetszwo commented on code in PR #10866:
URL: https://github.com/apache/ozone/pull/10866#discussion_r3694513699
##########
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/export/ExportFileManager.java:
##########
@@ -0,0 +1,234 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hdds.scm.container.export;
+
+import java.io.File;
+import java.io.IOException;
+import java.io.RandomAccessFile;
+import java.nio.channels.FileLock;
+import java.nio.channels.OverlappingFileLockException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.List;
+import java.util.Objects;
+import org.apache.commons.io.FileUtils;
+import org.apache.hadoop.ozone.util.UUIDUtil;
+import org.apache.ratis.util.AtomicFileOutputStream;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Manages on-disk paths and artifacts for container ID export jobs.
+ *
+ * <p>The export directory ({@code exportDirectory}, typically {@code {scm.db.dirs}/exports})
+ * uses the layout below. The manager gzip-compresses the archive ({@code .tar.gz}) so operators
+ * can stream entries with {@code zcat}.
+ *
+ * <p>While a job runs, shard text files are written under {@code export_{jobId}/}. The archive is
+ * created only after all shards are written. The export manager writes
+ * {@code container-ids-{scope}-{timestamp}_job{jobId}.tar.gz.tmp} and atomically renames it to
+ * {@code .tar.gz} on close ({@link AtomicFileOutputStream}), so a partial {@code .tar.gz} is
+ * never visible. {@link #lock()} uses {@code in_use.lock} to exclude concurrent writers.
+ *
+ * <pre>
+ * {exportDirectory}/
+ * ├── in_use.lock
+ * ├── container-ids-{scope}-{timestamp}_job{jobId}.tar.gz
+ * ├── container-ids-{scope}-{timestamp}_job{jobId}.tar.gz.tmp
+ * └── export_{jobId}/
+ * ├── container-ids-{scope}-{timestamp}-part001.txt
Review Comment:
Since `-` is used within a component (e.g. uuid, healthState, etc.), it should use `_` for inter-components.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15935. Extract ExportFileManager and document container export directory layout [ozone]
Posted by "sarvekshayr (via GitHub)" <gi...@apache.org>.
sarvekshayr commented on code in PR #10866:
URL: https://github.com/apache/ozone/pull/10866#discussion_r3694573350
##########
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/export/ExportFileManager.java:
##########
@@ -0,0 +1,234 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hdds.scm.container.export;
+
+import java.io.File;
+import java.io.IOException;
+import java.io.RandomAccessFile;
+import java.nio.channels.FileLock;
+import java.nio.channels.OverlappingFileLockException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.List;
+import java.util.Objects;
+import org.apache.commons.io.FileUtils;
+import org.apache.hadoop.ozone.util.UUIDUtil;
+import org.apache.ratis.util.AtomicFileOutputStream;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Manages on-disk paths and artifacts for container ID export jobs.
+ *
+ * <p>The export directory ({@code exportDirectory}, typically {@code {scm.db.dirs}/exports})
+ * uses the layout below. The manager gzip-compresses the archive ({@code .tar.gz}) so operators
+ * can stream entries with {@code zcat}.
+ *
+ * <p>While a job runs, shard text files are written under {@code export_{jobId}/}. The archive is
+ * created only after all shards are written. The export manager writes
+ * {@code container-ids-{scope}-{timestamp}_job{jobId}.tar.gz.tmp} and atomically renames it to
+ * {@code .tar.gz} on close ({@link AtomicFileOutputStream}), so a partial {@code .tar.gz} is
+ * never visible. {@link #lock()} uses {@code in_use.lock} to exclude concurrent writers.
+ *
+ * <pre>
+ * {exportDirectory}/
+ * ├── in_use.lock
+ * ├── container-ids-{scope}-{timestamp}_job{jobId}.tar.gz
+ * ├── container-ids-{scope}-{timestamp}_job{jobId}.tar.gz.tmp
+ * └── export_{jobId}/
+ * ├── container-ids-{scope}-{timestamp}-part001.txt
Review Comment:
In `scope` component, should health and lifecycle states be joined with `_` or `-`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15935. Extract ExportFileManager and document container export directory layout [ozone]
Posted by "sarvekshayr (via GitHub)" <gi...@apache.org>.
sarvekshayr commented on code in PR #10866:
URL: https://github.com/apache/ozone/pull/10866#discussion_r3654595178
##########
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/export/ExportFileManager.java:
##########
@@ -0,0 +1,179 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hdds.scm.container.export;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+import java.util.Objects;
+import java.util.UUID;
+import org.apache.commons.io.FileUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Manages on-disk paths and artifacts for container ID export jobs.
+ * Layout under the export directory ({@code {exportDirectory}}, typically {@code {scm.db.dirs}/exports}):
+ * <p>
+ * {exportDirectory}/
+ * {jobId}.in-progress // marker while a job is running
+ * container-ids-{scope}-{timestamp}-{jobId}.tar // completed export archive
+ * export-{jobId}/ // per-job workspace (removed on success)
Review Comment:
> When this directory will be deleted?
<p><b>When {@code export_{jobId}/} is deleted:</b> the export manager deletes it after the
archive closes successfully, or during {@link #cleanupFailedArtifacts} on failure or cancel.
On startup, {@link #start()} deletes a leftover {@code export_{jobId}/} when no in-progress
marker remains. If the marker still exists, {@link #start()} deletes {@code export_{jobId}/}
together with the marker and any partial archive for that job id.
> What if the SCM is restarted when a job is running?
<p><b>SCM restart while a job runs:</b> the in-progress marker and {@code export_{jobId}/}
remain on disk, but in-memory job status is lost. {@link #start()} treats the job as incomplete,
removes the marker, workspace, and any partial {@code .tar.gz} for that job id, and the
operator re-submits the export on the new leader.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15935. Extract ExportFileManager and document container export directory layout [ozone]
Posted by "szetszwo (via GitHub)" <gi...@apache.org>.
szetszwo commented on code in PR #10866:
URL: https://github.com/apache/ozone/pull/10866#discussion_r3653541612
##########
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/export/ExportFileManager.java:
##########
@@ -0,0 +1,179 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hdds.scm.container.export;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+import java.util.Objects;
+import java.util.UUID;
+import org.apache.commons.io.FileUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Manages on-disk paths and artifacts for container ID export jobs.
+ * Layout under the export directory ({@code {exportDirectory}}, typically {@code {scm.db.dirs}/exports}):
+ * <p>
+ * {exportDirectory}/
Review Comment:
Let's draw it using the `tree` commend output format:
```
% tree
.
├── dev-support
│ ├── checks
│ │ ├── _diffoscope.sh
│ │ ├── _mvn_check.sh
│ │ ├── build.sh
│ │ ├── repro.sh
│ │ └── unit.sh
│ ├── find_maven.sh
│ └── make_rc.sh
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15935. Extract ExportFileManager and document container export directory layout [ozone]
Posted by "sarvekshayr (via GitHub)" <gi...@apache.org>.
sarvekshayr commented on code in PR #10866:
URL: https://github.com/apache/ozone/pull/10866#discussion_r3654590731
##########
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/export/ExportFileManager.java:
##########
@@ -0,0 +1,179 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hdds.scm.container.export;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+import java.util.Objects;
+import java.util.UUID;
+import org.apache.commons.io.FileUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Manages on-disk paths and artifacts for container ID export jobs.
+ * Layout under the export directory ({@code {exportDirectory}}, typically {@code {scm.db.dirs}/exports}):
+ * <p>
+ * {exportDirectory}/
+ * {jobId}.in-progress // marker while a job is running
+ * container-ids-{scope}-{timestamp}-{jobId}.tar // completed export archive
Review Comment:
> When this file will be deleted?
<p><b>When {@code .tar.gz} is deleted:</b> {@link #cleanupFailedArtifacts} deletes partial
archives for failed or cancelled jobs. {@link #start()} deletes partial archives for jobs that
still have an in-progress marker. Completed archives remain on disk until the export manager
evicts the job from memory ({@code maxTerminalJobs} in {@code ContainerExportManager}) or an
operator deletes them manually. After SCM restart, in-memory eviction state is lost, so
completed archives persist until manual cleanup.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15935. Extract ExportFileManager and document container export directory layout [ozone]
Posted by "sarvekshayr (via GitHub)" <gi...@apache.org>.
sarvekshayr commented on code in PR #10866:
URL: https://github.com/apache/ozone/pull/10866#discussion_r3654590731
##########
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/export/ExportFileManager.java:
##########
@@ -0,0 +1,179 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hdds.scm.container.export;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+import java.util.Objects;
+import java.util.UUID;
+import org.apache.commons.io.FileUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Manages on-disk paths and artifacts for container ID export jobs.
+ * Layout under the export directory ({@code {exportDirectory}}, typically {@code {scm.db.dirs}/exports}):
+ * <p>
+ * {exportDirectory}/
+ * {jobId}.in-progress // marker while a job is running
+ * container-ids-{scope}-{timestamp}-{jobId}.tar // completed export archive
Review Comment:
<p><b>When {@code .tar.gz} is deleted:</b> {@link #cleanupFailedArtifacts} deletes partial
archives for failed or cancelled jobs. {@link #start()} deletes partial archives for jobs that
still have an in-progress marker. Completed archives remain on disk until the export manager
evicts the job from memory ({@code maxTerminalJobs} in {@code ContainerExportManager}) or an
operator deletes them manually. After SCM restart, in-memory eviction state is lost, so
completed archives persist until manual cleanup.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org