You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@sqoop.apache.org by Eric Lin <er...@cloudera.com> on 2017/01/17 07:24:28 UTC

Re: Review Request 54528: SQOOP-3042 - Sqoop does not clear compile directory under /tmp/sqoop-/compile automatically

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/54528/
-----------------------------------------------------------

(Updated Jan. 17, 2017, 7:24 a.m.)


Review request for Sqoop and Szabolcs Vasas.


Bugs: SQOOP-3042
    https://issues.apache.org/jira/browse/SQOOP-3042


Repository: sqoop-trunk


Description
-------

After running sqoop, all the temp files generated by ClassWriter are left behind on disk, so anyone can check those JAVA files to see the schema of those tables that Sqoop has been interacting with. By default, the directory is under /tmp/sqoop-<username>/compile.

In class org.apache.sqoop.SqoopOptions, function getNonceJarDir(), I can see that we did add "deleteOnExit" on the temp dir:
    for (int attempts = 0; attempts < MAX_DIR_CREATE_ATTEMPTS; attempts++) {
      hashDir = new File(baseDir, RandomHash.generateMD5String());
      while (hashDir.exists()) {
        hashDir = new File(baseDir, RandomHash.generateMD5String());
      }

      if (hashDir.mkdirs()) {
        // We created the directory. Use it.
        // If this directory is not actually filled with files, delete it
        // when the JVM quits.
        hashDir.deleteOnExit();
        break;
      }
    }
However, I believe it failed to delete due to directory is not empty.


Diffs
-----

  src/java/org/apache/sqoop/SqoopOptions.java 30b4705 
  src/java/org/apache/sqoop/orm/ClassWriter.java e289429 
  src/java/org/apache/sqoop/orm/CompilationManager.java 0a2a87f 

Diff: https://reviews.apache.org/r/54528/diff/


Testing
-------

I have tested manually. I have checked with a couple of other Java developers and it turned out that it is not easy to add test for deleteOnExit, so I did not add any test cases. The code path I changed does not seem to have test coverage either. Let me know if I am wrong.

Thanks


Thanks,

Eric Lin


Re: Review Request 54528: SQOOP-3042 - Sqoop does not clear compile directory under /tmp/sqoop-/compile automatically

Posted by Eric Lin <er...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/54528/
-----------------------------------------------------------

(Updated June 11, 2017, 12:02 p.m.)


Review request for Sqoop and Szabolcs Vasas.


Changes
-------

added new argument "--delete-compile-dir" so that deletion of compile directory can be controlled by user


Bugs: SQOOP-3042
    https://issues.apache.org/jira/browse/SQOOP-3042


Repository: sqoop-trunk


Description
-------

After running sqoop, all the temp files generated by ClassWriter are left behind on disk, so anyone can check those JAVA files to see the schema of those tables that Sqoop has been interacting with. By default, the directory is under /tmp/sqoop-<username>/compile.

In class org.apache.sqoop.SqoopOptions, function getNonceJarDir(), I can see that we did add "deleteOnExit" on the temp dir:
    for (int attempts = 0; attempts < MAX_DIR_CREATE_ATTEMPTS; attempts++) {
      hashDir = new File(baseDir, RandomHash.generateMD5String());
      while (hashDir.exists()) {
        hashDir = new File(baseDir, RandomHash.generateMD5String());
      }

      if (hashDir.mkdirs()) {
        // We created the directory. Use it.
        // If this directory is not actually filled with files, delete it
        // when the JVM quits.
        hashDir.deleteOnExit();
        break;
      }
    }
However, I believe it failed to delete due to directory is not empty.


Diffs (updated)
-----

  src/java/org/apache/sqoop/SqoopOptions.java 2eb3d8a 
  src/java/org/apache/sqoop/orm/ClassWriter.java cdb2364 
  src/java/org/apache/sqoop/orm/CompilationManager.java 3322c8b 
  src/java/org/apache/sqoop/tool/BaseSqoopTool.java 1564bdc 
  src/test/com/cloudera/sqoop/TestSqoopOptions.java dbdd2f1 


Diff: https://reviews.apache.org/r/54528/diff/2/

Changes: https://reviews.apache.org/r/54528/diff/1-2/


Testing
-------

I have tested manually. I have checked with a couple of other Java developers and it turned out that it is not easy to add test for deleteOnExit, so I did not add any test cases. The code path I changed does not seem to have test coverage either. Let me know if I am wrong.

Thanks


Thanks,

Eric Lin