You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@commons.apache.org by "Oliver Zeigermann (JIRA)" <ji...@apache.org> on 2007/09/06 14:32:34 UTC
[jira] Commented: (TRANSACTION-26) applyDeletes in
commons-transaction-1.2 is very slow in case when large number of files
needs to be deleted from the directory
[ https://issues.apache.org/jira/browse/TRANSACTION-26?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12525408 ]
Oliver Zeigermann commented on TRANSACTION-26:
----------------------------------------------
What you propose is a reversal of a fix you proposed before: https://issues.apache.org/jira/browse/TRANSACTION-18 and https://issues.apache.org/jira/browse/TRANSACTION-19. AFAIK your previous fix was justified as it ensured correct behavior which is more important that speed. Please check that and report back.
> applyDeletes in commons-transaction-1.2 is very slow in case when large number of files needs to be deleted from the directory
> ------------------------------------------------------------------------------------------------------------------------------
>
> Key: TRANSACTION-26
> URL: https://issues.apache.org/jira/browse/TRANSACTION-26
> Project: Commons Transaction
> Issue Type: Improvement
> Affects Versions: 1.2
> Environment: Linux/JDK1.5
> Reporter: Bojan Vukojevic
> Assignee: Oliver Zeigermann
> Priority: Minor
> Fix For: 1.2
>
> Attachments: apply_delete_patch_1.2.txt
>
>
> method applyDeletes in FileResourceManager lists the directory after every file delete. In case when there is a large number of files in the directory that needs to be deleted, this can be very slow. I had a case where directory had ~100,000.00 files. It took 5 hours for this method to complete. Simple improvement can be done where the check for the empty directory is moved outside of the loop that is deleting the files, i.e.:
> Original:
> protected static void applyDeletes(File removeDir, File targetDir, File rootDir) throws IOException {
> if (removeDir.isDirectory() && targetDir.isDirectory()) {
> File[] files = removeDir.listFiles();
> for (int i = 0; i < files.length; i++) {
> File removeFile = files[i];
> File targetFile = new File(targetDir, removeFile.getName());
> if (removeFile.isFile()) {
> if (targetFile.exists()) {
> if (!targetFile.delete()) {
> throw new IOException("Could not delete file " + removeFile.getName()
> + " in directory targetDir");
> }
> }
> // indicate, this has been done
> removeFile.delete();
> } else {
> applyDeletes(removeFile, targetFile, rootDir);
> }
> // delete empty target directories, except root dir
> if (!targetDir.equals(rootDir) && targetDir.list().length == 0) {
> targetDir.delete();
> }
> }
> }
> }
> =======================
> Much Faster:
> protected static void applyDeletes(File removeDir, File targetDir, File rootDir) throws IOException {
> if (removeDir.isDirectory() && targetDir.isDirectory()) {
> File[] files = removeDir.listFiles();
> for (int i = 0; i < files.length; i++) {
> File removeFile = files[i];
> File targetFile = new File(targetDir, removeFile.getName());
> if (removeFile.isFile()) {
> if (targetFile.exists()) {
> if (!targetFile.delete()) {
> throw new IOException("Could not delete file " + removeFile.getName()
> + " in directory targetDir");
> }
> }
> // indicate, this has been done
> removeFile.delete();
> } else {
> applyDeletes(removeFile, targetFile, rootDir);
> }
> }
> // delete empty target directories, except root dir
> if (!targetDir.equals(rootDir) && targetDir.list().length == 0) {
> targetDir.delete();
> }
> }
> }
> Below is the patch generated based on the released 1.2 version:
> Index: //10.1.31.20/bvukojev/DS7.1_EclipseWorkspace/asf-commons-transaction-1.2/src/java/org/apache/commons/transaction/file/FileResourceManager.java
> ===================================================================
> --- //10.1.31.20/bvukojev/DS7.1_EclipseWorkspace/asf-commons-transaction-1.2/src/java/org/apache/commons/transaction/file/FileResourceManager.java (revision 573053)
> +++ //10.1.31.20/bvukojev/DS7.1_EclipseWorkspace/asf-commons-transaction-1.2/src/java/org/apache/commons/transaction/file/FileResourceManager.java (working copy)
> @@ -162,12 +162,12 @@
> removeFile.delete();
> } else {
> applyDeletes(removeFile, targetFile, rootDir);
> - }
> - // delete empty target directories, except root dir
> - if (!targetDir.equals(rootDir) && targetDir.list().length == 0) {
> - targetDir.delete();
> - }
> + }
> }
> + // delete empty target directories, except root dir
> + if (!targetDir.equals(rootDir) && targetDir.list().length == 0) {
> + targetDir.delete();
> + }
> }
> }
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.