You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Amareshwari Sri Ramadasu (JIRA)" <ji...@apache.org> on 2008/01/31 11:11:34 UTC

[jira] Created: (HADOOP-2754) Path filter for Local file system list .crc files

Path filter for Local file system list .crc files
-------------------------------------------------

                 Key: HADOOP-2754
                 URL: https://issues.apache.org/jira/browse/HADOOP-2754
             Project: Hadoop Core
          Issue Type: Bug
            Reporter: Amareshwari Sri Ramadasu
             Fix For: 0.16.1



If we write a path filter for local file system, it  lists .crc files also.
If dont pass any filter, it lists paths properly without any .crc files. But we write a filter, it does list .crc files also.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (HADOOP-2754) Path filter for Local file system list .crc files

Posted by "Konstantin Shvachko (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HADOOP-2754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Konstantin Shvachko updated HADOOP-2754:
----------------------------------------

       Resolution: Fixed
    Fix Version/s:     (was: 0.16.1)
                   0.17.0
           Status: Resolved  (was: Patch Available)

I just committed this. Thank you Hairong.

> Path filter for Local file system list .crc files
> -------------------------------------------------
>
>                 Key: HADOOP-2754
>                 URL: https://issues.apache.org/jira/browse/HADOOP-2754
>             Project: Hadoop Core
>          Issue Type: Bug
>          Components: dfs
>            Reporter: Amareshwari Sri Ramadasu
>            Assignee: Hairong Kuang
>             Fix For: 0.17.0
>
>         Attachments: crc.patch
>
>
> If we write a path filter for local file system, it  lists .crc files also.
> If dont pass any filter, it lists paths properly without any .crc files. But we write a filter, it does list .crc files also.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HADOOP-2754) Path filter for Local file system list .crc files

Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HADOOP-2754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12564535#action_12564535 ] 

Hairong Kuang commented on HADOOP-2754:
---------------------------------------

Composing user-provided filter with the default filter is a good idea. Doug, do you mind if I open a new jira on this idea?

> Path filter for Local file system list .crc files
> -------------------------------------------------
>
>                 Key: HADOOP-2754
>                 URL: https://issues.apache.org/jira/browse/HADOOP-2754
>             Project: Hadoop Core
>          Issue Type: Bug
>          Components: dfs
>            Reporter: Amareshwari Sri Ramadasu
>            Assignee: Hairong Kuang
>             Fix For: 0.16.1
>
>
> If we write a path filter for local file system, it  lists .crc files also.
> If dont pass any filter, it lists paths properly without any .crc files. But we write a filter, it does list .crc files also.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (HADOOP-2754) Path filter for Local file system list .crc files

Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HADOOP-2754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hairong Kuang updated HADOOP-2754:
----------------------------------

    Attachment: crc.patch

> Currently fs either uses a user-provided filter or the default filter if no filter is provided by the user. 
Sorry that I did not look at this issue very closely. This how globStatus is implemented but not for listStatus.

Here is a patch that fixed the problem.

> Path filter for Local file system list .crc files
> -------------------------------------------------
>
>                 Key: HADOOP-2754
>                 URL: https://issues.apache.org/jira/browse/HADOOP-2754
>             Project: Hadoop Core
>          Issue Type: Bug
>          Components: dfs
>            Reporter: Amareshwari Sri Ramadasu
>            Assignee: Hairong Kuang
>             Fix For: 0.16.1
>
>         Attachments: crc.patch
>
>
> If we write a path filter for local file system, it  lists .crc files also.
> If dont pass any filter, it lists paths properly without any .crc files. But we write a filter, it does list .crc files also.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HADOOP-2754) Path filter for Local file system list .crc files

Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HADOOP-2754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12564872#action_12564872 ] 

Hairong Kuang commented on HADOOP-2754:
---------------------------------------

But composing filters is not the current semantics. Currently fs either uses a user-provided filter or the default filter if no filter is provided by the user. Composing filters is an incompatible change.

> Path filter for Local file system list .crc files
> -------------------------------------------------
>
>                 Key: HADOOP-2754
>                 URL: https://issues.apache.org/jira/browse/HADOOP-2754
>             Project: Hadoop Core
>          Issue Type: Bug
>          Components: dfs
>            Reporter: Amareshwari Sri Ramadasu
>            Assignee: Hairong Kuang
>             Fix For: 0.16.1
>
>
> If we write a path filter for local file system, it  lists .crc files also.
> If dont pass any filter, it lists paths properly without any .crc files. But we write a filter, it does list .crc files also.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HADOOP-2754) Path filter for Local file system list .crc files

Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HADOOP-2754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12564478#action_12564478 ] 

Doug Cutting commented on HADOOP-2754:
--------------------------------------

> The ChecksumFileSystem's default filter should still filter .crc files.

Yes, and when another filter is passed, the two must be composed.

> Path filter for Local file system list .crc files
> -------------------------------------------------
>
>                 Key: HADOOP-2754
>                 URL: https://issues.apache.org/jira/browse/HADOOP-2754
>             Project: Hadoop Core
>          Issue Type: Bug
>            Reporter: Amareshwari Sri Ramadasu
>            Assignee: Hairong Kuang
>             Fix For: 0.16.1
>
>
> If we write a path filter for local file system, it  lists .crc files also.
> If dont pass any filter, it lists paths properly without any .crc files. But we write a filter, it does list .crc files also.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HADOOP-2754) Path filter for Local file system list .crc files

Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HADOOP-2754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12564938#action_12564938 ] 

Doug Cutting commented on HADOOP-2754:
--------------------------------------

> It calls listStatus(Path), which already applies the default filter [ ... ]

Good point!  I missed that.

And maybe it's not worth removing the listPaths() implementations now.

Okay, I'm +1 on this patch now.

> Path filter for Local file system list .crc files
> -------------------------------------------------
>
>                 Key: HADOOP-2754
>                 URL: https://issues.apache.org/jira/browse/HADOOP-2754
>             Project: Hadoop Core
>          Issue Type: Bug
>          Components: dfs
>            Reporter: Amareshwari Sri Ramadasu
>            Assignee: Hairong Kuang
>             Fix For: 0.16.1
>
>         Attachments: crc.patch
>
>
> If we write a path filter for local file system, it  lists .crc files also.
> If dont pass any filter, it lists paths properly without any .crc files. But we write a filter, it does list .crc files also.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HADOOP-2754) Path filter for Local file system list .crc files

Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HADOOP-2754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12564442#action_12564442 ] 

Doug Cutting commented on HADOOP-2754:
--------------------------------------

I think perhaps the bug is that ChecksumFileSystem (on which LocalFileSystem is based) does not implement listStatus().  It has a listPaths() implementation that filters checksum files.  Probably this listPaths() implementation should be replaced with a listStatus() implementation that filters checksum files.  The default implementation of listPaths() is in terms of listStatus(), so that will still be filtered.  Right now, listStatus() is implemented by the FilterFileSystem base class, which just forwards to the RawLocalFileSystem, and thus includes the checksum files.

> Path filter for Local file system list .crc files
> -------------------------------------------------
>
>                 Key: HADOOP-2754
>                 URL: https://issues.apache.org/jira/browse/HADOOP-2754
>             Project: Hadoop Core
>          Issue Type: Bug
>            Reporter: Amareshwari Sri Ramadasu
>             Fix For: 0.16.1
>
>
> If we write a path filter for local file system, it  lists .crc files also.
> If dont pass any filter, it lists paths properly without any .crc files. But we write a filter, it does list .crc files also.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (HADOOP-2754) Path filter for Local file system list .crc files

Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HADOOP-2754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hairong Kuang updated HADOOP-2754:
----------------------------------

    Status: Patch Available  (was: Open)

Thanks Doug for your quick review.

> Path filter for Local file system list .crc files
> -------------------------------------------------
>
>                 Key: HADOOP-2754
>                 URL: https://issues.apache.org/jira/browse/HADOOP-2754
>             Project: Hadoop Core
>          Issue Type: Bug
>          Components: dfs
>            Reporter: Amareshwari Sri Ramadasu
>            Assignee: Hairong Kuang
>             Fix For: 0.16.1
>
>         Attachments: crc.patch
>
>
> If we write a path filter for local file system, it  lists .crc files also.
> If dont pass any filter, it lists paths properly without any .crc files. But we write a filter, it does list .crc files also.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (HADOOP-2754) Path filter for Local file system list .crc files

Posted by "Robert Chansler (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HADOOP-2754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Robert Chansler updated HADOOP-2754:
------------------------------------

    Component/s: dfs

> Path filter for Local file system list .crc files
> -------------------------------------------------
>
>                 Key: HADOOP-2754
>                 URL: https://issues.apache.org/jira/browse/HADOOP-2754
>             Project: Hadoop Core
>          Issue Type: Bug
>          Components: dfs
>            Reporter: Amareshwari Sri Ramadasu
>            Assignee: Hairong Kuang
>             Fix For: 0.16.1
>
>
> If we write a path filter for local file system, it  lists .crc files also.
> If dont pass any filter, it lists paths properly without any .crc files. But we write a filter, it does list .crc files also.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HADOOP-2754) Path filter for Local file system list .crc files

Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HADOOP-2754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12564473#action_12564473 ] 

Hairong Kuang commented on HADOOP-2754:
---------------------------------------

I believe that this is a regression from release 0.14 when eliminating .crc files in dfs. The default path filter in FileSystem was changed from filtering .crc files to be accepting any files. The ChecksumFileSystem's default filter should still filter .crc files.

> Path filter for Local file system list .crc files
> -------------------------------------------------
>
>                 Key: HADOOP-2754
>                 URL: https://issues.apache.org/jira/browse/HADOOP-2754
>             Project: Hadoop Core
>          Issue Type: Bug
>            Reporter: Amareshwari Sri Ramadasu
>            Assignee: Hairong Kuang
>             Fix For: 0.16.1
>
>
> If we write a path filter for local file system, it  lists .crc files also.
> If dont pass any filter, it lists paths properly without any .crc files. But we write a filter, it does list .crc files also.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HADOOP-2754) Path filter for Local file system list .crc files

Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HADOOP-2754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12564991#action_12564991 ] 

Hadoop QA commented on HADOOP-2754:
-----------------------------------

+1 overall.  Here are the results of testing the latest attachment 
http://issues.apache.org/jira/secure/attachment/12374577/crc.patch
against trunk revision 616796.

    @author +1.  The patch does not contain any @author tags.

    javadoc +1.  The javadoc tool did not generate any warning messages.

    javac +1.  The applied patch does not generate any new javac compiler warnings.

    release audit +1.  The applied patch does not generate any new release audit warnings.

    findbugs +1.  The patch does not introduce any new Findbugs warnings.

    core tests +1.  The patch passed core unit tests.

    contrib tests +1.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1724/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1724/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1724/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1724/console

This message is automatically generated.

> Path filter for Local file system list .crc files
> -------------------------------------------------
>
>                 Key: HADOOP-2754
>                 URL: https://issues.apache.org/jira/browse/HADOOP-2754
>             Project: Hadoop Core
>          Issue Type: Bug
>          Components: dfs
>            Reporter: Amareshwari Sri Ramadasu
>            Assignee: Hairong Kuang
>             Fix For: 0.16.1
>
>         Attachments: crc.patch
>
>
> If we write a path filter for local file system, it  lists .crc files also.
> If dont pass any filter, it lists paths properly without any .crc files. But we write a filter, it does list .crc files also.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HADOOP-2754) Path filter for Local file system list .crc files

Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HADOOP-2754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12564915#action_12564915 ] 

Doug Cutting commented on HADOOP-2754:
--------------------------------------

The attached patch fixes listStatus(Path), but not listStatus(Path, PathFilter).

I also think we should remove all the listPaths() implementations from ChecksumFileSystem, since the filtering added there is redundant.  Once listStatus() does the right thing, then the default implementations of listPaths() in FileSystem should work correctly for ChecksumFileSystem.


> Path filter for Local file system list .crc files
> -------------------------------------------------
>
>                 Key: HADOOP-2754
>                 URL: https://issues.apache.org/jira/browse/HADOOP-2754
>             Project: Hadoop Core
>          Issue Type: Bug
>          Components: dfs
>            Reporter: Amareshwari Sri Ramadasu
>            Assignee: Hairong Kuang
>             Fix For: 0.16.1
>
>         Attachments: crc.patch
>
>
> If we write a path filter for local file system, it  lists .crc files also.
> If dont pass any filter, it lists paths properly without any .crc files. But we write a filter, it does list .crc files also.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HADOOP-2754) Path filter for Local file system list .crc files

Posted by "Hudson (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HADOOP-2754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12568072#action_12568072 ] 

Hudson commented on HADOOP-2754:
--------------------------------

Integrated in Hadoop-trunk #398 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/398/])

> Path filter for Local file system list .crc files
> -------------------------------------------------
>
>                 Key: HADOOP-2754
>                 URL: https://issues.apache.org/jira/browse/HADOOP-2754
>             Project: Hadoop Core
>          Issue Type: Bug
>          Components: dfs
>            Reporter: Amareshwari Sri Ramadasu
>            Assignee: Hairong Kuang
>             Fix For: 0.16.1
>
>         Attachments: crc.patch
>
>
> If we write a path filter for local file system, it  lists .crc files also.
> If dont pass any filter, it lists paths properly without any .crc files. But we write a filter, it does list .crc files also.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HADOOP-2754) Path filter for Local file system list .crc files

Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HADOOP-2754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12564562#action_12564562 ] 

Doug Cutting commented on HADOOP-2754:
--------------------------------------

> Doug, do you mind if I open a new jira on this idea?

I think composition is required to fix this issue.  listStatus(Path, PathFilter) when applied to a CheckSumFileSystem should filter both for .crc files and for the user-specified PathFilter.  That means something like:

{noformat}
public FileStatus[] CheckSumFileSystem#listStatus(Path p, final PathFilter filter) throws IOException {
  fs.listStatus(p, new PathFilter() {
      public boolean accept(Path file) {
        return (!isChecksumFile(file) && filter.accept(file));
      }
    };
}
{noformat}
i.e., composing the filters.


> Path filter for Local file system list .crc files
> -------------------------------------------------
>
>                 Key: HADOOP-2754
>                 URL: https://issues.apache.org/jira/browse/HADOOP-2754
>             Project: Hadoop Core
>          Issue Type: Bug
>          Components: dfs
>            Reporter: Amareshwari Sri Ramadasu
>            Assignee: Hairong Kuang
>             Fix For: 0.16.1
>
>
> If we write a path filter for local file system, it  lists .crc files also.
> If dont pass any filter, it lists paths properly without any .crc files. But we write a filter, it does list .crc files also.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (HADOOP-2754) Path filter for Local file system list .crc files

Posted by "Sameer Paranjpye (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HADOOP-2754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Sameer Paranjpye updated HADOOP-2754:
-------------------------------------

       Assignee: Hairong Kuang
    Description: 
If we write a path filter for local file system, it  lists .crc files also.
If dont pass any filter, it lists paths properly without any .crc files. But we write a filter, it does list .crc files also.

  was:

If we write a path filter for local file system, it  lists .crc files also.
If dont pass any filter, it lists paths properly without any .crc files. But we write a filter, it does list .crc files also.


> Path filter for Local file system list .crc files
> -------------------------------------------------
>
>                 Key: HADOOP-2754
>                 URL: https://issues.apache.org/jira/browse/HADOOP-2754
>             Project: Hadoop Core
>          Issue Type: Bug
>            Reporter: Amareshwari Sri Ramadasu
>            Assignee: Hairong Kuang
>             Fix For: 0.16.1
>
>
> If we write a path filter for local file system, it  lists .crc files also.
> If dont pass any filter, it lists paths properly without any .crc files. But we write a filter, it does list .crc files also.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HADOOP-2754) Path filter for Local file system list .crc files

Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HADOOP-2754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12564934#action_12564934 ] 

Hairong Kuang commented on HADOOP-2754:
---------------------------------------

I thought listStatus(Path, PathFilter) did not do the composing either. But actually listStatus(Path, PathFilter) works right. It calls listStatus(Path), which already applies the default filter, then applies the user-provided filter.

> Path filter for Local file system list .crc files
> -------------------------------------------------
>
>                 Key: HADOOP-2754
>                 URL: https://issues.apache.org/jira/browse/HADOOP-2754
>             Project: Hadoop Core
>          Issue Type: Bug
>          Components: dfs
>            Reporter: Amareshwari Sri Ramadasu
>            Assignee: Hairong Kuang
>             Fix For: 0.16.1
>
>         Attachments: crc.patch
>
>
> If we write a path filter for local file system, it  lists .crc files also.
> If dont pass any filter, it lists paths properly without any .crc files. But we write a filter, it does list .crc files also.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.