You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "Siyao Meng (Jira)" <ji...@apache.org> on 2026/10/23 11:12:00 UTC

[jira] [Created] (HDDS-17389) Recon export cleanup deletes every directory and .tar file in ozone.recon.export.directory, not only files created by exports

Siyao Meng created HDDS-17389:
---------------------------------

             Summary: Recon export cleanup deletes every directory and .tar file in ozone.recon.export.directory, not only files created by exports
                 Key: HDDS-17389
                 URL: https://issues.apache.org/jira/browse/HDDS-17389
             Project: Apache Ozone
          Issue Type: Bug
            Reporter: Siyao Meng
         Attachments: CR-1-export-startup-cleanup-only-owned-entries.patch, TestBugCR1ExportDirSweep.java

h3. Mechanism
The {{ExportJobManager}} constructor removes leftovers of earlier exports from {{ozone.recon.export.directory}}. It lists the directory and calls {{FileUtils.deleteQuietly}} (recursive) on every subdirectory and on every file ending in {{.tar}}, without looking at the name. The manager itself only creates {{<jobId>}} working directories (a random UUID) and {{export_<state>_<millis>.tar}} files, so everything else it deletes was put there by someone else.

The configured value is used as is. Nothing checks that the directory is dedicated to exports or differs from {{ozone.recon.db.dir}}, and neither {{ozone-default.xml}} nor {{ReconServerConfigKeys}} says that its contents are removed. SCM's container ID export ({{ExportFileManager}}) clears its export directory on startup more narrowly: it removes only {{export_<UUID>}} directories and {{*.tar.gz.tmp}} files.

h3. Trigger
# Set {{ozone.recon.export.directory}} to a directory that holds other data, for example the Recon DB directory itself or a shared scratch directory.
# Start Recon. The cleanup runs when the {{ExportJobManager}} singleton is created, which happens once per Recon process on first use of the containers API (argued from the Guice binding, not run in a Recon server).

The default ({{ozone.recon.export.directory}} empty, resolved to {{<ozone.recon.db.dir>/exports}}) is not affected.

h3. Impact
* With the export directory equal to {{ozone.recon.db.dir}}: Recon's container key DB, OM snapshot DB, Recon SCM DB and Derby SQL DB directories are deleted, and so is {{recon/}} with Recon's storage {{current/VERSION}} file (run against stand-in directories with those names, see reproduction). Because the sweep runs on the first containers API request, Recon has these databases open at that point, so they are deleted under a running process (argued from code, not run).
* With a shared directory: other applications' directories and any {{.tar}} file (for example backups) are deleted permanently (run). Other files are kept.
* It repeats on every Recon start.

h3. Reproduction
[^TestBugCR1ExportDirSweep.java] ({{hadoop-ozone/recon}}, package {{org.apache.hadoop.ozone.recon.api}}) builds {{ExportJobManager}} through its public constructor with real configuration keys, as Guice does; only {{ContainerHealthSchemaManager}}, which the cleanup does not use, is mocked. The first test points the export directory at the Recon DB directory holding {{recon-container-key.db_...}}, {{om.snapshot.db_...}}, {{recon-scm.db}}, {{ozone_recon_derby.db}} and {{recon/current/VERSION}}; after construction the directory is empty. The second uses a shared directory with {{other-app-cache/}}, {{operator-backup-2026-10-01.tar}} and {{notes.txt}}; only {{notes.txt}} is left. Both tests pass on unmodified source, deterministic; a passing test means the defect is present.

h3. Patch
[^CR-1-export-startup-cleanup-only-owned-entries.patch], against fb2d0310a9c6ea37c72cefb5cc1d5b4db35a7ab1. It also applies to master at 9c3e5477ffb.

The cleanup now removes a directory only if its name is a valid UUID ({{UUIDUtil.isValidUuidString}}, as in SCM's {{ExportFileManager}}) and a file only if its name has the form {{export_<state>_<millis>.tar}}, the names {{submitJob}} and {{executeExport}} create. The state part accepts any letters, because {{submitJob}} lower cases it with the JVM default locale (under a Turkish locale {{MISSING}} gets a dotless i). Leftovers of an interrupted export are still removed and other files are kept. A directory whose name is a UUID is still removed, because the job manager names its own working directories that way. In the default directory, which only holds export files, the behaviour is unchanged.

Covered in the existing {{TestExportJobManager}}: the new {{startupCleanupKeepsEntriesNotCreatedByExports}} checks that {{backup.tar}}, {{export_backup.tar}} and a DB directory survive, and {{startupCleanupRemovesLeftoverTarsAndDirs}} now uses the names the manager really creates (it used {{stale_export.tar}} and {{stale_job_id/}} before), including a tar named with the Turkish lower case of {{MISSING}}, and still expects them removed. The test builds that name with an explicit locale and does not change the JVM default. Without the change the new test fails because {{backup.tar}} is deleted. With it {{TestExportJobManager}}, {{TestExportJob}} and {{TestContainerEndpoint}} (54 tests) pass and checkstyle is clean. With the patch both tests of the reproduction fail, because all entries are kept.

Found by code review of the Recon unhealthy container export job manager, as part of the TLA+ verification effort under HDDS-15926, on commit fb2d0310a9c6ea37c72cefb5cc1d5b4db35a7ab1. Checked against HDDS issues and apache/ozone pull requests for duplicates before filing. The attached patch is a proposal for review. Generated with Specula (Claude Opus 5.5).



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org