[jira] [Commented] (FLUME-2958) Add ignorePattern for TaildirSource

["Stephen O'Donnell (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLUME-2958?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15836743#comment-15836743 ] 

Stephen O'Donnell commented on FLUME-2958:
------------------------------------------

Is the idea here that for each file group, you can specify an ignore pattern eg:

{code}
a1.sources.r1.filegroups = f1 f2
a1.sources.r1.filegroups.f1 = /var/log/test1/.*
a1.source.r1.ignorePattern.f1 = .*backup
a1.sources.r1.filegroups.f2 = /var/log/test2/.*log.*
a1.sources.r1.ignorePattern.f2 = /var/log/test2/.*notAlog.*
{code}

If no ignore pattern is present, then the behaviour is as today, but if there is an ignore pattern, then matched files will be further filtered down by the ignorePattern, eg:

{code}
    final PathMatcher matcher = FS.getPathMatcher("regex:" + regex);
    final PathMatcher ignoreMatcher = FS.getPathMatcher("regex:" + ignoreRegex);
    this.fileFilter = new DirectoryStream.Filter<Path>() {
      @Override
      public boolean accept(Path entry) throws IOException {
        return matcher.matches(entry.getFileName()) && !Files.isDirectory(entry) && !ignoreMatcher.maches(entry.getFileName());
      }
    };
{code}

> Add ignorePattern for TaildirSource
> -----------------------------------
>
>                 Key: FLUME-2958
>                 URL: https://issues.apache.org/jira/browse/FLUME-2958
>             Project: Flume
>          Issue Type: Improvement
>          Components: Sinks+Sources
>    Affects Versions: v1.7.0
>            Reporter: Hu Liu,
>            Assignee: Hu Liu,
>             Fix For: v1.8.0
>
>
> we have tried the TaildirSource and found that it's lack of ignorePattern specifying which files to ignore. I'm glad to work on it if anyone could assign it to me



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)