You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "Colin Patrick McCabe (JIRA)" <ji...@apache.org> on 2012/07/11 03:13:35 UTC

[jira] [Created] (HDFS-3634) Add self-contained, mavenized fuse_dfs test

Colin Patrick McCabe created HDFS-3634:
------------------------------------------

             Summary: Add self-contained, mavenized fuse_dfs test
                 Key: HDFS-3634
                 URL: https://issues.apache.org/jira/browse/HDFS-3634
             Project: Hadoop HDFS
          Issue Type: Test
          Components: contrib/fuse-dfs
    Affects Versions: 2.0.1-alpha
            Reporter: Colin Patrick McCabe
            Assignee: Colin Patrick McCabe
            Priority: Minor


We should have a self-contained, mavenized FUSE unit test which runs as part of the normal build and can detect problems.  Of course, because FUSE is an optional build component, the unit test won't run unless the user has FUSE installed.  However, it would be very useful in improving the quality of fuse_dfs and detecting regressions.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Updated] (HDFS-3634) Add self-contained, mavenized fuse_dfs test

Posted by "Colin Patrick McCabe (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HDFS-3634?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Colin Patrick McCabe updated HDFS-3634:
---------------------------------------

    Attachment: HDFS-3634.006.patch

* remove stray tabs
                
> Add self-contained, mavenized fuse_dfs test
> -------------------------------------------
>
>                 Key: HDFS-3634
>                 URL: https://issues.apache.org/jira/browse/HDFS-3634
>             Project: Hadoop HDFS
>          Issue Type: Test
>          Components: fuse-dfs
>    Affects Versions: 2.1.0-alpha
>            Reporter: Colin Patrick McCabe
>            Assignee: Colin Patrick McCabe
>            Priority: Minor
>         Attachments: HDFS-3634.002.patch, HDFS-3634.003.patch, HDFS-3634.004.patch, HDFS-3634.005.patch, HDFS-3634.006.patch
>
>
> We should have a self-contained, mavenized FUSE unit test which runs as part of the normal build and can detect problems.  Of course, because FUSE is an optional build component, the unit test won't run unless the user has FUSE installed.  However, it would be very useful in improving the quality of fuse_dfs and detecting regressions.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (HDFS-3634) Add self-contained, mavenized fuse_dfs test

Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HDFS-3634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13430674#comment-13430674 ] 

Hadoop QA commented on HDFS-3634:
---------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12539704/HDFS-3634.005.patch
  against trunk revision .

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 3 new or modified test files.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 eclipse:eclipse.  The patch built with eclipse:eclipse.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    -1 core tests.  The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs:

                  org.apache.hadoop.hdfs.TestDFSClientRetries

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2965//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2965//console

This message is automatically generated.
                
> Add self-contained, mavenized fuse_dfs test
> -------------------------------------------
>
>                 Key: HDFS-3634
>                 URL: https://issues.apache.org/jira/browse/HDFS-3634
>             Project: Hadoop HDFS
>          Issue Type: Test
>          Components: fuse-dfs
>    Affects Versions: 2.1.0-alpha
>            Reporter: Colin Patrick McCabe
>            Assignee: Colin Patrick McCabe
>            Priority: Minor
>         Attachments: HDFS-3634.002.patch, HDFS-3634.003.patch, HDFS-3634.004.patch, HDFS-3634.005.patch
>
>
> We should have a self-contained, mavenized FUSE unit test which runs as part of the normal build and can detect problems.  Of course, because FUSE is an optional build component, the unit test won't run unless the user has FUSE installed.  However, it would be very useful in improving the quality of fuse_dfs and detecting regressions.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Updated] (HDFS-3634) Add self-contained, mavenized fuse_dfs test

Posted by "Colin Patrick McCabe (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HDFS-3634?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Colin Patrick McCabe updated HDFS-3634:
---------------------------------------

    Status: Patch Available  (was: Open)
    
> Add self-contained, mavenized fuse_dfs test
> -------------------------------------------
>
>                 Key: HDFS-3634
>                 URL: https://issues.apache.org/jira/browse/HDFS-3634
>             Project: Hadoop HDFS
>          Issue Type: Test
>          Components: fuse-dfs
>    Affects Versions: 2.1.0-alpha
>            Reporter: Colin Patrick McCabe
>            Assignee: Colin Patrick McCabe
>            Priority: Minor
>         Attachments: HDFS-3634.002.patch
>
>
> We should have a self-contained, mavenized FUSE unit test which runs as part of the normal build and can detect problems.  Of course, because FUSE is an optional build component, the unit test won't run unless the user has FUSE installed.  However, it would be very useful in improving the quality of fuse_dfs and detecting regressions.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (HDFS-3634) Add self-contained, mavenized fuse_dfs test

Posted by "Andy Isaacson (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HDFS-3634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13429662#comment-13429662 ] 

Andy Isaacson commented on HDFS-3634:
-------------------------------------

{code}
+    req.tv_sec = rem.tv_sec;
+    req.tv_nsec = rem.tv_nsec;
{code}
This can simply be written {{req = rem;}}.
{code}
+  } while (rem.tv_sec || rem.tv_nsec);
{code}
I don't see anywhere in the docs that say {{rem}} is zeroed on successful sleep, nor that it isn't modified on successful sleep.  The docs say the return value will be 0 on successful sleep.  So we should do something like
{code}
   do {
      req = rem;
      ret = nanosleep(&req, &rem);
   } while (ret == -1 && errno == EINTR);
   if (ret == -1) {
       fprintf(stderr, "sleepNoSig: nanosleep: %s\n", strerror(errno));
   }
{code}

                
> Add self-contained, mavenized fuse_dfs test
> -------------------------------------------
>
>                 Key: HDFS-3634
>                 URL: https://issues.apache.org/jira/browse/HDFS-3634
>             Project: Hadoop HDFS
>          Issue Type: Test
>          Components: fuse-dfs
>    Affects Versions: 2.1.0-alpha
>            Reporter: Colin Patrick McCabe
>            Assignee: Colin Patrick McCabe
>            Priority: Minor
>         Attachments: HDFS-3634.002.patch, HDFS-3634.003.patch, HDFS-3634.004.patch
>
>
> We should have a self-contained, mavenized FUSE unit test which runs as part of the normal build and can detect problems.  Of course, because FUSE is an optional build component, the unit test won't run unless the user has FUSE installed.  However, it would be very useful in improving the quality of fuse_dfs and detecting regressions.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (HDFS-3634) Add self-contained, mavenized fuse_dfs test

Posted by "Hudson (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HDFS-3634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13431876#comment-13431876 ] 

Hudson commented on HDFS-3634:
------------------------------

Integrated in Hadoop-Hdfs-trunk-Commit #2631 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2631/])
    HDFS-3634. Add self-contained, mavenized fuse_dfs test. Contributed by Colin Patrick McCabe. (Revision 1371231)

     Result = SUCCESS
atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1371231
Files : 
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/CMakeLists.txt
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/native/fuse-dfs/CMakeLists.txt
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/native/fuse-dfs/test/fuse_workload.c
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/native/fuse-dfs/test/fuse_workload.h
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/native/fuse-dfs/test/test_fuse_dfs.c
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/expect.h
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/native_mini_dfs.c
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/native_mini_dfs.h
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/native/util/posix_util.c
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/native/util/posix_util.h

                
> Add self-contained, mavenized fuse_dfs test
> -------------------------------------------
>
>                 Key: HDFS-3634
>                 URL: https://issues.apache.org/jira/browse/HDFS-3634
>             Project: Hadoop HDFS
>          Issue Type: Test
>          Components: fuse-dfs
>    Affects Versions: 2.1.0-alpha
>            Reporter: Colin Patrick McCabe
>            Assignee: Colin Patrick McCabe
>            Priority: Minor
>             Fix For: 2.2.0-alpha
>
>         Attachments: HDFS-3634.002.patch, HDFS-3634.003.patch, HDFS-3634.004.patch, HDFS-3634.005.patch, HDFS-3634.006.patch
>
>
> We should have a self-contained, mavenized FUSE unit test which runs as part of the normal build and can detect problems.  Of course, because FUSE is an optional build component, the unit test won't run unless the user has FUSE installed.  However, it would be very useful in improving the quality of fuse_dfs and detecting regressions.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (HDFS-3634) Add self-contained, mavenized fuse_dfs test

Posted by "Aaron T. Myers (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HDFS-3634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13431853#comment-13431853 ] 

Aaron T. Myers commented on HDFS-3634:
--------------------------------------

+1, the latest patch looks good to me as well. I'm going to commit this momentarily.
                
> Add self-contained, mavenized fuse_dfs test
> -------------------------------------------
>
>                 Key: HDFS-3634
>                 URL: https://issues.apache.org/jira/browse/HDFS-3634
>             Project: Hadoop HDFS
>          Issue Type: Test
>          Components: fuse-dfs
>    Affects Versions: 2.1.0-alpha
>            Reporter: Colin Patrick McCabe
>            Assignee: Colin Patrick McCabe
>            Priority: Minor
>         Attachments: HDFS-3634.002.patch, HDFS-3634.003.patch, HDFS-3634.004.patch, HDFS-3634.005.patch, HDFS-3634.006.patch
>
>
> We should have a self-contained, mavenized FUSE unit test which runs as part of the normal build and can detect problems.  Of course, because FUSE is an optional build component, the unit test won't run unless the user has FUSE installed.  However, it would be very useful in improving the quality of fuse_dfs and detecting regressions.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Closed] (HDFS-3634) Add self-contained, mavenized fuse_dfs test

Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HDFS-3634?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Arun C Murthy closed HDFS-3634.
-------------------------------

    
> Add self-contained, mavenized fuse_dfs test
> -------------------------------------------
>
>                 Key: HDFS-3634
>                 URL: https://issues.apache.org/jira/browse/HDFS-3634
>             Project: Hadoop HDFS
>          Issue Type: Test
>          Components: fuse-dfs
>    Affects Versions: 2.0.0-alpha
>            Reporter: Colin Patrick McCabe
>            Assignee: Colin Patrick McCabe
>            Priority: Minor
>             Fix For: 2.0.2-alpha
>
>         Attachments: HDFS-3634.002.patch, HDFS-3634.003.patch, HDFS-3634.004.patch, HDFS-3634.005.patch, HDFS-3634.006.patch
>
>
> We should have a self-contained, mavenized FUSE unit test which runs as part of the normal build and can detect problems.  Of course, because FUSE is an optional build component, the unit test won't run unless the user has FUSE installed.  However, it would be very useful in improving the quality of fuse_dfs and detecting regressions.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HDFS-3634) Add self-contained, mavenized fuse_dfs test

Posted by "Andy Isaacson (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HDFS-3634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13429553#comment-13429553 ] 

Andy Isaacson commented on HDFS-3634:
-------------------------------------

{code}
+static int expectDirs(const struct dirent *de, void *v)
+{
+  const char **names = (const char **)v;
{code}
no need to cast a void* in C, just assign.
{code}
+ * @return                 0 on success; error code otherwise
{code}
The function returns a negative errno, may as well document that.
{code}
+    fprintf(stderr, "FUSE_TEST: failed to fork: error %d\n", ret);
{code}
please print {{strerror(errno)}} as well.  (Several instances of this pattern, IMO we should never print errno without also printing strerror.)
{code}
+      c = ((src[i + 1] - '0') << 16) |
+           ((src[i + 2] - '0') << 8) |
+            (src[i + 3] - '0');
{code}
That's not the right way to decode a 3-digit octal string, it yields 0x10503 given "0153".  I think you would want <<2 and <<5 but given that we're discussing it, clearly this needs a standalone function and a unit test; I'd copy to a temporary array and use {{strtol(buf,8,&p)}} then check p.

{code}
+  f = fopen("/proc/mounts", "r");
...
+    line = fgets(buf, sizeof(buf), f);
{code}
Please use {{getmntent(3)}} rather than rolling our own.
{code}
+  snprintf(scratch, sizeof(scratch), "%s", argv0);
{code}
strncpy is more idiomatic.
{code}
+  char mntTmp[PATH_MAX] = { 0 };
{code}
Initialize strings with strings, so this should be {code}char mntTmp[PATH_MAX] = "";{code} .
{code}
+int recursiveDeleteContents(const char *path)
{code}
Do we really need this (potentially dangerous) code in the testcase?  I'd hate to see a bug result in an accidental {{rm -rf $HOME}}.  (I've looked at the obvious cases and don't see any bugs, but that's small comfort.)  The target hdfs will be deleted afterwards so no need to delete there; the local target is pretty small so leaving it around is no big deal.  Some of the code seems to indicate that something will error out if you attempt to mount over a directory with contents, but that seems like just a bug?

{code}
+    if ((de->d_name[0] == '.') && (de->d_name[1] == '\0'))
+      continue;
+    if ((de->d_name[0] == '.') && (de->d_name[1] == '.') &&
+        (de->d_name[2] == '\0'))
{code}
These would be more idiomatic as {{if (!strcmp(de->d_name, "."))}}, IMHO.  But, your call.
{code}
+    // canonicalize non-abosolute TMPDIR
{code}
abosolute -> absolute
                
> Add self-contained, mavenized fuse_dfs test
> -------------------------------------------
>
>                 Key: HDFS-3634
>                 URL: https://issues.apache.org/jira/browse/HDFS-3634
>             Project: Hadoop HDFS
>          Issue Type: Test
>          Components: fuse-dfs
>    Affects Versions: 2.1.0-alpha
>            Reporter: Colin Patrick McCabe
>            Assignee: Colin Patrick McCabe
>            Priority: Minor
>         Attachments: HDFS-3634.002.patch, HDFS-3634.003.patch
>
>
> We should have a self-contained, mavenized FUSE unit test which runs as part of the normal build and can detect problems.  Of course, because FUSE is an optional build component, the unit test won't run unless the user has FUSE installed.  However, it would be very useful in improving the quality of fuse_dfs and detecting regressions.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (HDFS-3634) Add self-contained, mavenized fuse_dfs test

Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HDFS-3634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13429928#comment-13429928 ] 

Hadoop QA commented on HDFS-3634:
---------------------------------

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12539392/HDFS-3634.004.patch
  against trunk revision .

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 3 new or modified test files.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 eclipse:eclipse.  The patch built with eclipse:eclipse.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2960//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2960//console

This message is automatically generated.
                
> Add self-contained, mavenized fuse_dfs test
> -------------------------------------------
>
>                 Key: HDFS-3634
>                 URL: https://issues.apache.org/jira/browse/HDFS-3634
>             Project: Hadoop HDFS
>          Issue Type: Test
>          Components: fuse-dfs
>    Affects Versions: 2.1.0-alpha
>            Reporter: Colin Patrick McCabe
>            Assignee: Colin Patrick McCabe
>            Priority: Minor
>         Attachments: HDFS-3634.002.patch, HDFS-3634.003.patch, HDFS-3634.004.patch
>
>
> We should have a self-contained, mavenized FUSE unit test which runs as part of the normal build and can detect problems.  Of course, because FUSE is an optional build component, the unit test won't run unless the user has FUSE installed.  However, it would be very useful in improving the quality of fuse_dfs and detecting regressions.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (HDFS-3634) Add self-contained, mavenized fuse_dfs test

Posted by "Aaron T. Myers (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HDFS-3634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13430826#comment-13430826 ] 

Aaron T. Myers commented on HDFS-3634:
--------------------------------------

Looks like the patch has a few tab characters in it. Otherwise it looks good to me.

Andy, does Colin's latest patch look good to you? If so, I'll go ahead and commit this patch.
                
> Add self-contained, mavenized fuse_dfs test
> -------------------------------------------
>
>                 Key: HDFS-3634
>                 URL: https://issues.apache.org/jira/browse/HDFS-3634
>             Project: Hadoop HDFS
>          Issue Type: Test
>          Components: fuse-dfs
>    Affects Versions: 2.1.0-alpha
>            Reporter: Colin Patrick McCabe
>            Assignee: Colin Patrick McCabe
>            Priority: Minor
>         Attachments: HDFS-3634.002.patch, HDFS-3634.003.patch, HDFS-3634.004.patch, HDFS-3634.005.patch
>
>
> We should have a self-contained, mavenized FUSE unit test which runs as part of the normal build and can detect problems.  Of course, because FUSE is an optional build component, the unit test won't run unless the user has FUSE installed.  However, it would be very useful in improving the quality of fuse_dfs and detecting regressions.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (HDFS-3634) Add self-contained, mavenized fuse_dfs test

Posted by "Hudson (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HDFS-3634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13431878#comment-13431878 ] 

Hudson commented on HDFS-3634:
------------------------------

Integrated in Hadoop-Common-trunk-Commit #2566 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2566/])
    HDFS-3634. Add self-contained, mavenized fuse_dfs test. Contributed by Colin Patrick McCabe. (Revision 1371231)

     Result = SUCCESS
atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1371231
Files : 
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/CMakeLists.txt
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/native/fuse-dfs/CMakeLists.txt
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/native/fuse-dfs/test/fuse_workload.c
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/native/fuse-dfs/test/fuse_workload.h
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/native/fuse-dfs/test/test_fuse_dfs.c
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/expect.h
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/native_mini_dfs.c
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/native_mini_dfs.h
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/native/util/posix_util.c
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/native/util/posix_util.h

                
> Add self-contained, mavenized fuse_dfs test
> -------------------------------------------
>
>                 Key: HDFS-3634
>                 URL: https://issues.apache.org/jira/browse/HDFS-3634
>             Project: Hadoop HDFS
>          Issue Type: Test
>          Components: fuse-dfs
>    Affects Versions: 2.1.0-alpha
>            Reporter: Colin Patrick McCabe
>            Assignee: Colin Patrick McCabe
>            Priority: Minor
>             Fix For: 2.2.0-alpha
>
>         Attachments: HDFS-3634.002.patch, HDFS-3634.003.patch, HDFS-3634.004.patch, HDFS-3634.005.patch, HDFS-3634.006.patch
>
>
> We should have a self-contained, mavenized FUSE unit test which runs as part of the normal build and can detect problems.  Of course, because FUSE is an optional build component, the unit test won't run unless the user has FUSE installed.  However, it would be very useful in improving the quality of fuse_dfs and detecting regressions.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Updated] (HDFS-3634) Add self-contained, mavenized fuse_dfs test

Posted by "Colin Patrick McCabe (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HDFS-3634?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Colin Patrick McCabe updated HDFS-3634:
---------------------------------------

    Attachment: HDFS-3634.002.patch
    
> Add self-contained, mavenized fuse_dfs test
> -------------------------------------------
>
>                 Key: HDFS-3634
>                 URL: https://issues.apache.org/jira/browse/HDFS-3634
>             Project: Hadoop HDFS
>          Issue Type: Test
>          Components: fuse-dfs
>    Affects Versions: 2.1.0-alpha
>            Reporter: Colin Patrick McCabe
>            Assignee: Colin Patrick McCabe
>            Priority: Minor
>         Attachments: HDFS-3634.002.patch
>
>
> We should have a self-contained, mavenized FUSE unit test which runs as part of the normal build and can detect problems.  Of course, because FUSE is an optional build component, the unit test won't run unless the user has FUSE installed.  However, it would be very useful in improving the quality of fuse_dfs and detecting regressions.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (HDFS-3634) Add self-contained, mavenized fuse_dfs test

Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HDFS-3634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13423652#comment-13423652 ] 

Hadoop QA commented on HDFS-3634:
---------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12538100/HDFS-3634.002.patch
  against trunk revision .

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 3 new or modified test files.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 eclipse:eclipse.  The patch built with eclipse:eclipse.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    -1 core tests.  The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs:

                  org.apache.hadoop.hdfs.server.datanode.TestDirectoryScanner

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2917//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2917//console

This message is automatically generated.
                
> Add self-contained, mavenized fuse_dfs test
> -------------------------------------------
>
>                 Key: HDFS-3634
>                 URL: https://issues.apache.org/jira/browse/HDFS-3634
>             Project: Hadoop HDFS
>          Issue Type: Test
>          Components: fuse-dfs
>    Affects Versions: 2.1.0-alpha
>            Reporter: Colin Patrick McCabe
>            Assignee: Colin Patrick McCabe
>            Priority: Minor
>         Attachments: HDFS-3634.002.patch
>
>
> We should have a self-contained, mavenized FUSE unit test which runs as part of the normal build and can detect problems.  Of course, because FUSE is an optional build component, the unit test won't run unless the user has FUSE installed.  However, it would be very useful in improving the quality of fuse_dfs and detecting regressions.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Updated] (HDFS-3634) Add self-contained, mavenized fuse_dfs test

Posted by "Colin Patrick McCabe (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HDFS-3634?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Colin Patrick McCabe updated HDFS-3634:
---------------------------------------

    Attachment: HDFS-3634.003.patch

* be more careful to close all file descriptors on error -- to avoid hangs

* work around HDFS-2551 and add comment

* wait until fuse_dfs finishes mounting the filesystem before starting the test.  (Add waitForMount).

* put sleepNoSig into a common utility function.  Since we use alarm(), we can't use sleep().
                
> Add self-contained, mavenized fuse_dfs test
> -------------------------------------------
>
>                 Key: HDFS-3634
>                 URL: https://issues.apache.org/jira/browse/HDFS-3634
>             Project: Hadoop HDFS
>          Issue Type: Test
>          Components: fuse-dfs
>    Affects Versions: 2.1.0-alpha
>            Reporter: Colin Patrick McCabe
>            Assignee: Colin Patrick McCabe
>            Priority: Minor
>         Attachments: HDFS-3634.002.patch, HDFS-3634.003.patch
>
>
> We should have a self-contained, mavenized FUSE unit test which runs as part of the normal build and can detect problems.  Of course, because FUSE is an optional build component, the unit test won't run unless the user has FUSE installed.  However, it would be very useful in improving the quality of fuse_dfs and detecting regressions.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (HDFS-3634) Add self-contained, mavenized fuse_dfs test

Posted by "Andy Isaacson (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HDFS-3634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13429556#comment-13429556 ] 

Andy Isaacson commented on HDFS-3634:
-------------------------------------

{code}
+void sleepNoSig(int sec)
+{
+  struct timespec req, rem;
+
+  req.tv_sec = sec;
+  req.tv_nsec = 0;
+  memset(&rem, 0, sizeof(rem));
+  nanosleep(&req, &rem);
+}
{code}
Is this supposed to resume the sleep if interrupted?  If so we need a loop. If not, we can drop {{rem}} and just {{nanosleep(&req, 0);}}.
                
> Add self-contained, mavenized fuse_dfs test
> -------------------------------------------
>
>                 Key: HDFS-3634
>                 URL: https://issues.apache.org/jira/browse/HDFS-3634
>             Project: Hadoop HDFS
>          Issue Type: Test
>          Components: fuse-dfs
>    Affects Versions: 2.1.0-alpha
>            Reporter: Colin Patrick McCabe
>            Assignee: Colin Patrick McCabe
>            Priority: Minor
>         Attachments: HDFS-3634.002.patch, HDFS-3634.003.patch
>
>
> We should have a self-contained, mavenized FUSE unit test which runs as part of the normal build and can detect problems.  Of course, because FUSE is an optional build component, the unit test won't run unless the user has FUSE installed.  However, it would be very useful in improving the quality of fuse_dfs and detecting regressions.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Updated] (HDFS-3634) Add self-contained, mavenized fuse_dfs test

Posted by "Colin Patrick McCabe (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HDFS-3634?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Colin Patrick McCabe updated HDFS-3634:
---------------------------------------

    Attachment: HDFS-3634.005.patch

* improve sleepNoSig
                
> Add self-contained, mavenized fuse_dfs test
> -------------------------------------------
>
>                 Key: HDFS-3634
>                 URL: https://issues.apache.org/jira/browse/HDFS-3634
>             Project: Hadoop HDFS
>          Issue Type: Test
>          Components: fuse-dfs
>    Affects Versions: 2.1.0-alpha
>            Reporter: Colin Patrick McCabe
>            Assignee: Colin Patrick McCabe
>            Priority: Minor
>         Attachments: HDFS-3634.002.patch, HDFS-3634.003.patch, HDFS-3634.004.patch, HDFS-3634.005.patch
>
>
> We should have a self-contained, mavenized FUSE unit test which runs as part of the normal build and can detect problems.  Of course, because FUSE is an optional build component, the unit test won't run unless the user has FUSE installed.  However, it would be very useful in improving the quality of fuse_dfs and detecting regressions.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (HDFS-3634) Add self-contained, mavenized fuse_dfs test

Posted by "Hudson (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HDFS-3634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13431908#comment-13431908 ] 

Hudson commented on HDFS-3634:
------------------------------

Integrated in Hadoop-Mapreduce-trunk-Commit #2586 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2586/])
    HDFS-3634. Add self-contained, mavenized fuse_dfs test. Contributed by Colin Patrick McCabe. (Revision 1371231)

     Result = FAILURE
atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1371231
Files : 
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/CMakeLists.txt
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/native/fuse-dfs/CMakeLists.txt
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/native/fuse-dfs/test/fuse_workload.c
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/native/fuse-dfs/test/fuse_workload.h
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/native/fuse-dfs/test/test_fuse_dfs.c
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/expect.h
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/native_mini_dfs.c
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/native_mini_dfs.h
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/native/util/posix_util.c
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/native/util/posix_util.h

                
> Add self-contained, mavenized fuse_dfs test
> -------------------------------------------
>
>                 Key: HDFS-3634
>                 URL: https://issues.apache.org/jira/browse/HDFS-3634
>             Project: Hadoop HDFS
>          Issue Type: Test
>          Components: fuse-dfs
>    Affects Versions: 2.1.0-alpha
>            Reporter: Colin Patrick McCabe
>            Assignee: Colin Patrick McCabe
>            Priority: Minor
>             Fix For: 2.2.0-alpha
>
>         Attachments: HDFS-3634.002.patch, HDFS-3634.003.patch, HDFS-3634.004.patch, HDFS-3634.005.patch, HDFS-3634.006.patch
>
>
> We should have a self-contained, mavenized FUSE unit test which runs as part of the normal build and can detect problems.  Of course, because FUSE is an optional build component, the unit test won't run unless the user has FUSE installed.  However, it would be very useful in improving the quality of fuse_dfs and detecting regressions.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Updated] (HDFS-3634) Add self-contained, mavenized fuse_dfs test

Posted by "Colin Patrick McCabe (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HDFS-3634?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Colin Patrick McCabe updated HDFS-3634:
---------------------------------------

    Attachment: HDFS-3634.004.patch

* fix a few style issues

* use getmntent rather than reading from /proc/mounts directly.  This also means we don't need the code to parse octal escapes.

* don't call recursiveDeleteContents on mount point before mounting: instead, give the -ononempty option to FUSE.

* sleepNoSig: sleep the full period even in the presence of signals
                
> Add self-contained, mavenized fuse_dfs test
> -------------------------------------------
>
>                 Key: HDFS-3634
>                 URL: https://issues.apache.org/jira/browse/HDFS-3634
>             Project: Hadoop HDFS
>          Issue Type: Test
>          Components: fuse-dfs
>    Affects Versions: 2.1.0-alpha
>            Reporter: Colin Patrick McCabe
>            Assignee: Colin Patrick McCabe
>            Priority: Minor
>         Attachments: HDFS-3634.002.patch, HDFS-3634.003.patch, HDFS-3634.004.patch
>
>
> We should have a self-contained, mavenized FUSE unit test which runs as part of the normal build and can detect problems.  Of course, because FUSE is an optional build component, the unit test won't run unless the user has FUSE installed.  However, it would be very useful in improving the quality of fuse_dfs and detecting regressions.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (HDFS-3634) Add self-contained, mavenized fuse_dfs test

Posted by "Andy Isaacson (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HDFS-3634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13431386#comment-13431386 ] 

Andy Isaacson commented on HDFS-3634:
-------------------------------------

Thanks for addressing the feedback, Colin.  The .006 patch looks good to me.  I'm a bit concerned that we don't have any automated way to validate the interface, for example the JNI interface description strings, and I didn't have time to manually check them... but I suppose it'll blow up visibly if it's wrong, and this shouldn't break any existing code, so go ahead and check it in.  We can address any issues in a followup jira.
                
> Add self-contained, mavenized fuse_dfs test
> -------------------------------------------
>
>                 Key: HDFS-3634
>                 URL: https://issues.apache.org/jira/browse/HDFS-3634
>             Project: Hadoop HDFS
>          Issue Type: Test
>          Components: fuse-dfs
>    Affects Versions: 2.1.0-alpha
>            Reporter: Colin Patrick McCabe
>            Assignee: Colin Patrick McCabe
>            Priority: Minor
>         Attachments: HDFS-3634.002.patch, HDFS-3634.003.patch, HDFS-3634.004.patch, HDFS-3634.005.patch, HDFS-3634.006.patch
>
>
> We should have a self-contained, mavenized FUSE unit test which runs as part of the normal build and can detect problems.  Of course, because FUSE is an optional build component, the unit test won't run unless the user has FUSE installed.  However, it would be very useful in improving the quality of fuse_dfs and detecting regressions.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Updated] (HDFS-3634) Add self-contained, mavenized fuse_dfs test

Posted by "Aaron T. Myers (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HDFS-3634?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Aaron T. Myers updated HDFS-3634:
---------------------------------

       Resolution: Fixed
    Fix Version/s: 2.2.0-alpha
     Hadoop Flags: Reviewed
           Status: Resolved  (was: Patch Available)

I've just committed this to trunk and branch-2.

Thanks a lot for the contribution, Colin, and thanks a lot to Andy for the very thorough reviews.
                
> Add self-contained, mavenized fuse_dfs test
> -------------------------------------------
>
>                 Key: HDFS-3634
>                 URL: https://issues.apache.org/jira/browse/HDFS-3634
>             Project: Hadoop HDFS
>          Issue Type: Test
>          Components: fuse-dfs
>    Affects Versions: 2.1.0-alpha
>            Reporter: Colin Patrick McCabe
>            Assignee: Colin Patrick McCabe
>            Priority: Minor
>             Fix For: 2.2.0-alpha
>
>         Attachments: HDFS-3634.002.patch, HDFS-3634.003.patch, HDFS-3634.004.patch, HDFS-3634.005.patch, HDFS-3634.006.patch
>
>
> We should have a self-contained, mavenized FUSE unit test which runs as part of the normal build and can detect problems.  Of course, because FUSE is an optional build component, the unit test won't run unless the user has FUSE installed.  However, it would be very useful in improving the quality of fuse_dfs and detecting regressions.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (HDFS-3634) Add self-contained, mavenized fuse_dfs test

Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HDFS-3634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13430895#comment-13430895 ] 

Hadoop QA commented on HDFS-3634:
---------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12539771/HDFS-3634.006.patch
  against trunk revision .

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 3 new or modified test files.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 eclipse:eclipse.  The patch built with eclipse:eclipse.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    -1 core tests.  The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs:

                  org.apache.hadoop.hdfs.TestFileConcurrentReader

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2969//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2969//console

This message is automatically generated.
                
> Add self-contained, mavenized fuse_dfs test
> -------------------------------------------
>
>                 Key: HDFS-3634
>                 URL: https://issues.apache.org/jira/browse/HDFS-3634
>             Project: Hadoop HDFS
>          Issue Type: Test
>          Components: fuse-dfs
>    Affects Versions: 2.1.0-alpha
>            Reporter: Colin Patrick McCabe
>            Assignee: Colin Patrick McCabe
>            Priority: Minor
>         Attachments: HDFS-3634.002.patch, HDFS-3634.003.patch, HDFS-3634.004.patch, HDFS-3634.005.patch, HDFS-3634.006.patch
>
>
> We should have a self-contained, mavenized FUSE unit test which runs as part of the normal build and can detect problems.  Of course, because FUSE is an optional build component, the unit test won't run unless the user has FUSE installed.  However, it would be very useful in improving the quality of fuse_dfs and detecting regressions.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira