You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nutch.apache.org by thammegowda <gi...@git.apache.org> on 2016/02/11 01:24:54 UTC

[GitHub] nutch pull request: NUTCH-2144 : override db.ignore.external to ex...

GitHub user thammegowda opened a pull request:

    https://github.com/apache/nutch/pull/89

    NUTCH-2144 : override db.ignore.external to exempt interesting external domain URLs

     + Add extension point
          org.apache.nutch.net.URLExemptionFilter
     + Modify FetcherThread and ParseOutputFormat to
          integrate new extension point
     + Add extension urlfilter-ignoreexempt
     + build configs modified to include new extension
    
    
    Resolves https://issues.apache.org/jira/browse/NUTCH-2144

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/thammegowda/nutch NUTCH-2144-ignore-exempt

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/nutch/pull/89.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #89
    
----
commit 29c7ae4ec088f0f428ab95992e10af9d87a231ad
Author: Thamme Gowda <tg...@gmail.com>
Date:   2015-10-19T14:31:04Z

    Add an extension point and an extension to
     override 'db.ignore.external.links'.
    
     + Add extension point
          org.apache.nutch.net.URLExemptionFilter
     + Modify FetcherThread and ParseOutputFormat to
          integrate new extension point
     + Add extension urlfilter-ignoreexempt
     + build configs modified to include new extension

commit 43583f7af19c21dd7553c5e41c026cb852f0cfa1
Author: Thamme Gowda <tg...@gmail.com>
Date:   2016-02-11T00:03:03Z

    Added ignore exemption extension point and an extension

commit 559eb4d905a081e34095f0e9d5ee3e805363ccc3
Author: Thamme Gowda <tg...@gmail.com>
Date:   2016-02-11T00:21:08Z

    README updated

commit 3cf887befc56c3cf4127f45385e83c47248dd6e9
Author: Thamme Gowda <tg...@gmail.com>
Date:   2016-02-11T00:23:12Z

    Add an example rule fiile

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nutch pull request: NUTCH-2144 : override db.ignore.external to ex...

Posted by sebastian-nagel <gi...@git.apache.org>.
Github user sebastian-nagel commented on a diff in the pull request:

    https://github.com/apache/nutch/pull/89#discussion_r52833038
  
    --- Diff: src/plugin/urlfilter-ignoreexempt/src/java/org/apache/nutch/urlfilter/ignoreexempt/ExemptionUrlFilter.java ---
    @@ -0,0 +1,144 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *     http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +package org.apache.nutch.urlfilter.ignoreexempt;
    +
    +import org.apache.commons.io.IOUtils;
    +import org.apache.hadoop.conf.Configuration;
    +import org.apache.nutch.net.URLExemptionFilter;
    +import org.apache.nutch.util.NutchConfiguration;
    +import org.slf4j.Logger;
    +import org.slf4j.LoggerFactory;
    +
    +import java.io.IOException;
    +import java.io.InputStream;
    +import java.util.Arrays;
    +import java.util.regex.Pattern;
    +import java.util.List;
    +import java.util.ArrayList;
    +
    +
    +/**
    + * This implementation of {@link org.apache.nutch.net.URLExemptionFilter} uses regex configuration
    + * to check if URL is eligible for exemption from 'db.ignore.external'.
    + * When this filter is enabled, urls will be checked against configured sequence of regex rules.
    + *<p>
    + * The exemption rule file defaults to db-ignore-external-exemptions.txt in the classpath but can be
    + * overridden using the property  <code>"db.ignore.external.exemptions.file" in ./conf/nutch-*.xml</code>
    + *</p>
    + *
    + * The exemption rules are specified in plain text file where each line is a rule.
    + *
    + * When the url matches regex it is exempted from 'db.ignore.external...'<br/>
    + * <h3>Examples:</h3>
    + * <ol>
    + *   <li>
    + *     <b>Exempt urls ending with .jpg or .png or gif</b>
    + *      <br/><code>.*\.(jpg|JPG|png$|PNG|gif|GIF)$</code><br/><br/>
    + *   </li>
    + * </ol>
    + </pre>
    + *
    + * @since Feb 10, 2016
    + * @version 1
    + * @see URLExemptionFilter
    + */
    +public class ExemptionUrlFilter implements URLExemptionFilter {
    +
    +  public static final String DB_IGNORE_EXTERNAL_EXEMPTIONS_FILE = "db.ignore.external.exemptions.file";
    +  private static final Logger LOG = LoggerFactory.getLogger(ExemptionUrlFilter.class);
    +  private static ExemptionUrlFilter INSTANCE;
    +
    +  private List<Pattern> exemptions;
    +  private Configuration conf;
    +  private boolean enabled;
    +
    +  public static ExemptionUrlFilter getInstance() {
    --- End diff --
    
    Nutch automatically caches a single instance of each plugin class. The main() method could also call the constructor and then setConf().


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nutch pull request: NUTCH-2144 : override db.ignore.external to ex...

Posted by thammegowda <gi...@git.apache.org>.
Github user thammegowda closed the pull request at:

    https://github.com/apache/nutch/pull/89


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nutch pull request: NUTCH-2144 : override db.ignore.external to ex...

Posted by thammegowda <gi...@git.apache.org>.
Github user thammegowda commented on a diff in the pull request:

    https://github.com/apache/nutch/pull/89#discussion_r52835493
  
    --- Diff: src/plugin/urlfilter-ignoreexempt/src/java/org/apache/nutch/urlfilter/ignoreexempt/ExemptionUrlFilter.java ---
    @@ -0,0 +1,144 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *     http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +package org.apache.nutch.urlfilter.ignoreexempt;
    +
    +import org.apache.commons.io.IOUtils;
    +import org.apache.hadoop.conf.Configuration;
    +import org.apache.nutch.net.URLExemptionFilter;
    +import org.apache.nutch.util.NutchConfiguration;
    +import org.slf4j.Logger;
    +import org.slf4j.LoggerFactory;
    +
    +import java.io.IOException;
    +import java.io.InputStream;
    +import java.util.Arrays;
    +import java.util.regex.Pattern;
    +import java.util.List;
    +import java.util.ArrayList;
    +
    +
    +/**
    + * This implementation of {@link org.apache.nutch.net.URLExemptionFilter} uses regex configuration
    + * to check if URL is eligible for exemption from 'db.ignore.external'.
    + * When this filter is enabled, urls will be checked against configured sequence of regex rules.
    + *<p>
    + * The exemption rule file defaults to db-ignore-external-exemptions.txt in the classpath but can be
    + * overridden using the property  <code>"db.ignore.external.exemptions.file" in ./conf/nutch-*.xml</code>
    + *</p>
    + *
    + * The exemption rules are specified in plain text file where each line is a rule.
    + *
    + * When the url matches regex it is exempted from 'db.ignore.external...'<br/>
    + * <h3>Examples:</h3>
    + * <ol>
    + *   <li>
    + *     <b>Exempt urls ending with .jpg or .png or gif</b>
    + *      <br/><code>.*\.(jpg|JPG|png$|PNG|gif|GIF)$</code><br/><br/>
    + *   </li>
    + * </ol>
    + </pre>
    + *
    + * @since Feb 10, 2016
    + * @version 1
    + * @see URLExemptionFilter
    + */
    +public class ExemptionUrlFilter implements URLExemptionFilter {
    +
    +  public static final String DB_IGNORE_EXTERNAL_EXEMPTIONS_FILE = "db.ignore.external.exemptions.file";
    +  private static final Logger LOG = LoggerFactory.getLogger(ExemptionUrlFilter.class);
    +  private static ExemptionUrlFilter INSTANCE;
    +
    +  private List<Pattern> exemptions;
    +  private Configuration conf;
    +  private boolean enabled;
    +
    +  public static ExemptionUrlFilter getInstance() {
    --- End diff --
    
    :+1: 
    
    Agreed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nutch pull request: NUTCH-2144 : override db.ignore.external to ex...

Posted by thammegowda <gi...@git.apache.org>.
GitHub user thammegowda reopened a pull request:

    https://github.com/apache/nutch/pull/89

    NUTCH-2144 : override db.ignore.external to exempt interesting external domain URLs

     + Add extension point
          org.apache.nutch.net.URLExemptionFilter
     + Modify FetcherThread and ParseOutputFormat to
          integrate new extension point
     + Add extension urlfilter-ignoreexempt
     + build configs modified to include new extension
    
    
    Resolves https://issues.apache.org/jira/browse/NUTCH-2144

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/thammegowda/nutch NUTCH-2144-ignore-exempt

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/nutch/pull/89.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #89
    
----
commit 29c7ae4ec088f0f428ab95992e10af9d87a231ad
Author: Thamme Gowda <tg...@gmail.com>
Date:   2015-10-19T14:31:04Z

    Add an extension point and an extension to
     override 'db.ignore.external.links'.
    
     + Add extension point
          org.apache.nutch.net.URLExemptionFilter
     + Modify FetcherThread and ParseOutputFormat to
          integrate new extension point
     + Add extension urlfilter-ignoreexempt
     + build configs modified to include new extension

commit 43583f7af19c21dd7553c5e41c026cb852f0cfa1
Author: Thamme Gowda <tg...@gmail.com>
Date:   2016-02-11T00:03:03Z

    Added ignore exemption extension point and an extension

commit 559eb4d905a081e34095f0e9d5ee3e805363ccc3
Author: Thamme Gowda <tg...@gmail.com>
Date:   2016-02-11T00:21:08Z

    README updated

commit 3cf887befc56c3cf4127f45385e83c47248dd6e9
Author: Thamme Gowda <tg...@gmail.com>
Date:   2016-02-11T00:23:12Z

    Add an example rule fiile

commit b5cf404bf451fa80186ebb4120cfd39aa2c0f00b
Author: Thamme Gowda <th...@users.noreply.github.com>
Date:   2016-02-12T01:01:02Z

    Added License header

commit 3a555b106a4cef9bf0c0e0699f79aedd14ef9fa1
Author: Thamme Gowda <tg...@gmail.com>
Date:   2016-02-14T02:06:51Z

    Code reviewers suggestion incorporated
    
    + Reusing the rules and format from urlfilter-regex

commit 6bd026c8482f98b14a56b9b9bff78307f6998189
Author: Thamme Gowda <tg...@gmail.com>
Date:   2016-02-25T02:16:17Z

    merge upstream changed and Resolve all conflicts

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nutch pull request: NUTCH-2144 : override db.ignore.external to ex...

Posted by thammegowda <gi...@git.apache.org>.
Github user thammegowda commented on a diff in the pull request:

    https://github.com/apache/nutch/pull/89#discussion_r52835489
  
    --- Diff: src/plugin/urlfilter-ignoreexempt/src/java/org/apache/nutch/urlfilter/ignoreexempt/ExemptionUrlFilter.java ---
    @@ -0,0 +1,144 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *     http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +package org.apache.nutch.urlfilter.ignoreexempt;
    +
    +import org.apache.commons.io.IOUtils;
    +import org.apache.hadoop.conf.Configuration;
    +import org.apache.nutch.net.URLExemptionFilter;
    +import org.apache.nutch.util.NutchConfiguration;
    +import org.slf4j.Logger;
    +import org.slf4j.LoggerFactory;
    +
    +import java.io.IOException;
    +import java.io.InputStream;
    +import java.util.Arrays;
    +import java.util.regex.Pattern;
    +import java.util.List;
    +import java.util.ArrayList;
    +
    +
    +/**
    + * This implementation of {@link org.apache.nutch.net.URLExemptionFilter} uses regex configuration
    + * to check if URL is eligible for exemption from 'db.ignore.external'.
    + * When this filter is enabled, urls will be checked against configured sequence of regex rules.
    + *<p>
    + * The exemption rule file defaults to db-ignore-external-exemptions.txt in the classpath but can be
    + * overridden using the property  <code>"db.ignore.external.exemptions.file" in ./conf/nutch-*.xml</code>
    + *</p>
    + *
    + * The exemption rules are specified in plain text file where each line is a rule.
    + *
    + * When the url matches regex it is exempted from 'db.ignore.external...'<br/>
    + * <h3>Examples:</h3>
    + * <ol>
    + *   <li>
    + *     <b>Exempt urls ending with .jpg or .png or gif</b>
    + *      <br/><code>.*\.(jpg|JPG|png$|PNG|gif|GIF)$</code><br/><br/>
    + *   </li>
    + * </ol>
    + </pre>
    + *
    + * @since Feb 10, 2016
    + * @version 1
    + * @see URLExemptionFilter
    + */
    +public class ExemptionUrlFilter implements URLExemptionFilter {
    +
    +  public static final String DB_IGNORE_EXTERNAL_EXEMPTIONS_FILE = "db.ignore.external.exemptions.file";
    +  private static final Logger LOG = LoggerFactory.getLogger(ExemptionUrlFilter.class);
    +  private static ExemptionUrlFilter INSTANCE;
    +
    +  private List<Pattern> exemptions;
    +  private Configuration conf;
    +  private boolean enabled;
    +
    +  public static ExemptionUrlFilter getInstance() {
    +    if(INSTANCE == null) {
    +      synchronized (ExemptionUrlFilter.class) {
    +        if (INSTANCE == null) {
    +          INSTANCE = new ExemptionUrlFilter();
    +          INSTANCE.setConf(NutchConfiguration.create());
    +        }
    +      }
    +    }
    +    return INSTANCE;
    +  }
    +
    +  public boolean isEnabled() {
    +    return enabled;
    +  }
    +
    +  public List<Pattern> getExemptions() {
    +    return exemptions;
    +  }
    +
    +  @Override
    +  public boolean filter(String fromUrl, String toUrl) {
    +    //this implementation doesnt do anything with fromUrl
    +    if (exemptions != null) {
    +      for (Pattern pattern : exemptions) {
    +        if (pattern.matcher(toUrl).matches()) {
    +          return true; //If a regex matches, then exempted
    +        }
    +      }
    +    }
    +    //not exempted
    +    return false;
    +  }
    +
    +  @Override
    +  public void setConf(Configuration conf) {
    +    this.conf = conf;
    +    LOG.info("Ignore exemptions enabled");
    +    String fileName = this.conf.get(DB_IGNORE_EXTERNAL_EXEMPTIONS_FILE);
    +    InputStream stream = this.conf.getConfResourceAsInputStream(fileName);
    +    if (stream == null) {
    +      throw new RuntimeException("Couldn't find config file :" + fileName);
    +    }
    +    try {
    +      this.exemptions = new ArrayList<Pattern>();
    +      List<String> lines = IOUtils.readLines(stream);
    +      for (String line : lines) {
    +        line = line.trim();
    +        if (line.startsWith("#") || line.isEmpty()) {
    +          continue; //Skip : comment line or empty line
    +        }
    +        Pattern compiled = Pattern.compile(line);
    +        LOG.info("Regex :: {} ", line);
    +        exemptions.add(compiled);
    +      }
    +      LOG.info("Read {} rules from {}", exemptions.size(), fileName);
    +    } catch (IOException e) {
    +      throw new IllegalStateException(e);
    --- End diff --
    
    I suppose `InvalidConfiguration` leads to `IllegalState`. 
    Thanks for pointing out. It will be replaced with `RuntimeException`  


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nutch pull request: NUTCH-2144 : override db.ignore.external to ex...

Posted by sebastian-nagel <gi...@git.apache.org>.
Github user sebastian-nagel commented on a diff in the pull request:

    https://github.com/apache/nutch/pull/89#discussion_r52833199
  
    --- Diff: src/plugin/urlfilter-ignoreexempt/src/java/org/apache/nutch/urlfilter/ignoreexempt/ExemptionUrlFilter.java ---
    @@ -0,0 +1,144 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *     http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +package org.apache.nutch.urlfilter.ignoreexempt;
    +
    +import org.apache.commons.io.IOUtils;
    +import org.apache.hadoop.conf.Configuration;
    +import org.apache.nutch.net.URLExemptionFilter;
    +import org.apache.nutch.util.NutchConfiguration;
    +import org.slf4j.Logger;
    +import org.slf4j.LoggerFactory;
    +
    +import java.io.IOException;
    +import java.io.InputStream;
    +import java.util.Arrays;
    +import java.util.regex.Pattern;
    +import java.util.List;
    +import java.util.ArrayList;
    +
    +
    +/**
    + * This implementation of {@link org.apache.nutch.net.URLExemptionFilter} uses regex configuration
    + * to check if URL is eligible for exemption from 'db.ignore.external'.
    + * When this filter is enabled, urls will be checked against configured sequence of regex rules.
    + *<p>
    + * The exemption rule file defaults to db-ignore-external-exemptions.txt in the classpath but can be
    + * overridden using the property  <code>"db.ignore.external.exemptions.file" in ./conf/nutch-*.xml</code>
    + *</p>
    + *
    + * The exemption rules are specified in plain text file where each line is a rule.
    + *
    + * When the url matches regex it is exempted from 'db.ignore.external...'<br/>
    + * <h3>Examples:</h3>
    + * <ol>
    + *   <li>
    + *     <b>Exempt urls ending with .jpg or .png or gif</b>
    + *      <br/><code>.*\.(jpg|JPG|png$|PNG|gif|GIF)$</code><br/><br/>
    + *   </li>
    + * </ol>
    + </pre>
    + *
    + * @since Feb 10, 2016
    + * @version 1
    + * @see URLExemptionFilter
    + */
    +public class ExemptionUrlFilter implements URLExemptionFilter {
    +
    +  public static final String DB_IGNORE_EXTERNAL_EXEMPTIONS_FILE = "db.ignore.external.exemptions.file";
    +  private static final Logger LOG = LoggerFactory.getLogger(ExemptionUrlFilter.class);
    +  private static ExemptionUrlFilter INSTANCE;
    +
    +  private List<Pattern> exemptions;
    +  private Configuration conf;
    +  private boolean enabled;
    --- End diff --
    
    Is this variable necessary?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nutch pull request: NUTCH-2144 : override db.ignore.external to ex...

Posted by lewismc <gi...@git.apache.org>.
Github user lewismc commented on a diff in the pull request:

    https://github.com/apache/nutch/pull/89#discussion_r52692152
  
    --- Diff: conf/db-ignore-external-exemptions.txt ---
    @@ -0,0 +1,21 @@
    +# Exemption rules to db.ignore.external.links
    --- End diff --
    
    License header please


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nutch pull request: NUTCH-2144 : override db.ignore.external to ex...

Posted by sebastian-nagel <gi...@git.apache.org>.
Github user sebastian-nagel commented on a diff in the pull request:

    https://github.com/apache/nutch/pull/89#discussion_r52833653
  
    --- Diff: src/java/org/apache/nutch/parse/ParseOutputFormat.java ---
    @@ -338,6 +340,9 @@ public static String filterNormalize(String fromUrl, String toUrl,
           } catch (MalformedURLException e1) {
             return null; // skip it
           }
    +
    +
    +
           if ("bydomain".equalsIgnoreCase(ignoreExternalLinksMode)) {
             String toDomain = URLUtil.getDomainName(targetURL).toLowerCase();
             if (toDomain == null || !toDomain.equals(origin)) {
    --- End diff --
    
    Shouldn't this case also be covered (db.ignore.external.links == true and db.ignore.external.links.mode == byDomain)?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nutch pull request: NUTCH-2144 : override db.ignore.external to ex...

Posted by thammegowda <gi...@git.apache.org>.
Github user thammegowda commented on a diff in the pull request:

    https://github.com/apache/nutch/pull/89#discussion_r52835517
  
    --- Diff: src/plugin/urlfilter-ignoreexempt/src/java/org/apache/nutch/urlfilter/ignoreexempt/ExemptionUrlFilter.java ---
    @@ -0,0 +1,144 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *     http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +package org.apache.nutch.urlfilter.ignoreexempt;
    +
    +import org.apache.commons.io.IOUtils;
    +import org.apache.hadoop.conf.Configuration;
    +import org.apache.nutch.net.URLExemptionFilter;
    +import org.apache.nutch.util.NutchConfiguration;
    +import org.slf4j.Logger;
    +import org.slf4j.LoggerFactory;
    +
    +import java.io.IOException;
    +import java.io.InputStream;
    +import java.util.Arrays;
    +import java.util.regex.Pattern;
    +import java.util.List;
    +import java.util.ArrayList;
    +
    +
    +/**
    + * This implementation of {@link org.apache.nutch.net.URLExemptionFilter} uses regex configuration
    + * to check if URL is eligible for exemption from 'db.ignore.external'.
    + * When this filter is enabled, urls will be checked against configured sequence of regex rules.
    + *<p>
    + * The exemption rule file defaults to db-ignore-external-exemptions.txt in the classpath but can be
    + * overridden using the property  <code>"db.ignore.external.exemptions.file" in ./conf/nutch-*.xml</code>
    + *</p>
    + *
    + * The exemption rules are specified in plain text file where each line is a rule.
    + *
    + * When the url matches regex it is exempted from 'db.ignore.external...'<br/>
    + * <h3>Examples:</h3>
    + * <ol>
    + *   <li>
    + *     <b>Exempt urls ending with .jpg or .png or gif</b>
    + *      <br/><code>.*\.(jpg|JPG|png$|PNG|gif|GIF)$</code><br/><br/>
    + *   </li>
    + * </ol>
    + </pre>
    + *
    + * @since Feb 10, 2016
    + * @version 1
    + * @see URLExemptionFilter
    + */
    +public class ExemptionUrlFilter implements URLExemptionFilter {
    +
    +  public static final String DB_IGNORE_EXTERNAL_EXEMPTIONS_FILE = "db.ignore.external.exemptions.file";
    +  private static final Logger LOG = LoggerFactory.getLogger(ExemptionUrlFilter.class);
    +  private static ExemptionUrlFilter INSTANCE;
    +
    +  private List<Pattern> exemptions;
    +  private Configuration conf;
    +  private boolean enabled;
    --- End diff --
    
    Will be removed.
    Thanks for pointing out.
     


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nutch pull request: NUTCH-2144 : override db.ignore.external to ex...

Posted by sebastian-nagel <gi...@git.apache.org>.
Github user sebastian-nagel commented on a diff in the pull request:

    https://github.com/apache/nutch/pull/89#discussion_r52833909
  
    --- Diff: conf/db-ignore-external-exemptions.txt ---
    @@ -0,0 +1,37 @@
    +# Licensed to the Apache Software Foundation (ASF) under one or more
    +# contributor license agreements.  See the NOTICE file distributed with
    +# this work for additional information regarding copyright ownership.
    +# The ASF licenses this file to You under the Apache License, Version 2.0
    +# (the "License"); you may not use this file except in compliance with
    +# the License.  You may obtain a copy of the License at
    +#
    +#     http://www.apache.org/licenses/LICENSE-2.0
    +#
    +# Unless required by applicable law or agreed to in writing, software
    +# distributed under the License is distributed on an "AS IS" BASIS,
    +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    +# See the License for the specific language governing permissions and
    +# limitations under the License.
    +#
    +#
    +# Exemption rules to db.ignore.external.links
    +#
    +# Format :
    +#--------
    +# UrlRegex1
    +# UrlRegex2
    +# UrlRegex3
    +
    +
    +# NOTE ::
    +# 1. When the url matches any of the regex then that url is exempted.
    +# 2. # in the beginning makes it a comment line
    +# 3. To Test the regex, update this file and use the below command
    +#   bin/nutch plugin urlfilter-ignoreexempt org.apache.nutch.urlfilter.ignoreexempt.ExemptionUrlFilter <URL>
    +# 4. Dont forget to enable this plugin in nutch-site.xml
    +
    +
    +# Example 1:
    +#----------
    +# To exempt urls ending with image extensions, uncomment the below line
    +#.*\.(jpg|JPG|png$|PNG|gif|GIF)$
    --- End diff --
    
    Regex could be simplified to `#(?i).*\.(?:jpg|png|gif)` (or `#(?i)\.(?:jpg|png|gif)$` if Pattern.find() is used). `(?i)` makes the pattern case insensitive, cf. [NUTCH-2035](https://issues.apache.org/jira/browse/NUTCH-2035)



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nutch pull request: NUTCH-2144 : override db.ignore.external to ex...

Posted by thammegowda <gi...@git.apache.org>.
Github user thammegowda commented on a diff in the pull request:

    https://github.com/apache/nutch/pull/89#discussion_r52835613
  
    --- Diff: src/java/org/apache/nutch/parse/ParseOutputFormat.java ---
    @@ -338,6 +340,9 @@ public static String filterNormalize(String fromUrl, String toUrl,
           } catch (MalformedURLException e1) {
             return null; // skip it
           }
    +
    +
    +
           if ("bydomain".equalsIgnoreCase(ignoreExternalLinksMode)) {
             String toDomain = URLUtil.getDomainName(targetURL).toLowerCase();
             if (toDomain == null || !toDomain.equals(origin)) {
    --- End diff --
    
    I did not have a chance to test `db.ignore.external.links.mode : byDomain` feature.
    
    However, if this new feature should also be covered for overriding, then I am ready to do so.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nutch pull request: NUTCH-2144 : override db.ignore.external to ex...

Posted by sebastian-nagel <gi...@git.apache.org>.
Github user sebastian-nagel commented on a diff in the pull request:

    https://github.com/apache/nutch/pull/89#discussion_r52833853
  
    --- Diff: src/plugin/urlfilter-ignoreexempt/src/java/org/apache/nutch/urlfilter/ignoreexempt/ExemptionUrlFilter.java ---
    @@ -0,0 +1,144 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *     http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +package org.apache.nutch.urlfilter.ignoreexempt;
    +
    +import org.apache.commons.io.IOUtils;
    +import org.apache.hadoop.conf.Configuration;
    +import org.apache.nutch.net.URLExemptionFilter;
    +import org.apache.nutch.util.NutchConfiguration;
    +import org.slf4j.Logger;
    +import org.slf4j.LoggerFactory;
    +
    +import java.io.IOException;
    +import java.io.InputStream;
    +import java.util.Arrays;
    +import java.util.regex.Pattern;
    +import java.util.List;
    +import java.util.ArrayList;
    +
    +
    +/**
    + * This implementation of {@link org.apache.nutch.net.URLExemptionFilter} uses regex configuration
    + * to check if URL is eligible for exemption from 'db.ignore.external'.
    + * When this filter is enabled, urls will be checked against configured sequence of regex rules.
    + *<p>
    + * The exemption rule file defaults to db-ignore-external-exemptions.txt in the classpath but can be
    + * overridden using the property  <code>"db.ignore.external.exemptions.file" in ./conf/nutch-*.xml</code>
    + *</p>
    + *
    + * The exemption rules are specified in plain text file where each line is a rule.
    + *
    + * When the url matches regex it is exempted from 'db.ignore.external...'<br/>
    + * <h3>Examples:</h3>
    + * <ol>
    + *   <li>
    + *     <b>Exempt urls ending with .jpg or .png or gif</b>
    + *      <br/><code>.*\.(jpg|JPG|png$|PNG|gif|GIF)$</code><br/><br/>
    + *   </li>
    + * </ol>
    + </pre>
    + *
    + * @since Feb 10, 2016
    + * @version 1
    + * @see URLExemptionFilter
    + */
    +public class ExemptionUrlFilter implements URLExemptionFilter {
    +
    +  public static final String DB_IGNORE_EXTERNAL_EXEMPTIONS_FILE = "db.ignore.external.exemptions.file";
    +  private static final Logger LOG = LoggerFactory.getLogger(ExemptionUrlFilter.class);
    +  private static ExemptionUrlFilter INSTANCE;
    +
    +  private List<Pattern> exemptions;
    +  private Configuration conf;
    +  private boolean enabled;
    +
    +  public static ExemptionUrlFilter getInstance() {
    +    if(INSTANCE == null) {
    +      synchronized (ExemptionUrlFilter.class) {
    +        if (INSTANCE == null) {
    +          INSTANCE = new ExemptionUrlFilter();
    +          INSTANCE.setConf(NutchConfiguration.create());
    +        }
    +      }
    +    }
    +    return INSTANCE;
    +  }
    +
    +  public boolean isEnabled() {
    +    return enabled;
    +  }
    +
    +  public List<Pattern> getExemptions() {
    +    return exemptions;
    +  }
    +
    +  @Override
    +  public boolean filter(String fromUrl, String toUrl) {
    +    //this implementation doesnt do anything with fromUrl
    +    if (exemptions != null) {
    +      for (Pattern pattern : exemptions) {
    +        if (pattern.matcher(toUrl).matches()) {
    --- End diff --
    
    Could possibly use Pattern.find() instead of Pattern.matches():
    - would make patterns often shorter: `\.jpg$` instead of `.*\.jpg` (or `.*\.jpg$`)
    - (more important) the syntax would be the same as for urlfilter-regex rules


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nutch pull request: NUTCH-2144 : override db.ignore.external to ex...

Posted by thammegowda <gi...@git.apache.org>.
Github user thammegowda commented on a diff in the pull request:

    https://github.com/apache/nutch/pull/89#discussion_r52835676
  
    --- Diff: src/plugin/urlfilter-ignoreexempt/src/java/org/apache/nutch/urlfilter/ignoreexempt/ExemptionUrlFilter.java ---
    @@ -0,0 +1,144 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *     http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +package org.apache.nutch.urlfilter.ignoreexempt;
    +
    +import org.apache.commons.io.IOUtils;
    +import org.apache.hadoop.conf.Configuration;
    +import org.apache.nutch.net.URLExemptionFilter;
    +import org.apache.nutch.util.NutchConfiguration;
    +import org.slf4j.Logger;
    +import org.slf4j.LoggerFactory;
    +
    +import java.io.IOException;
    +import java.io.InputStream;
    +import java.util.Arrays;
    +import java.util.regex.Pattern;
    +import java.util.List;
    +import java.util.ArrayList;
    +
    +
    +/**
    + * This implementation of {@link org.apache.nutch.net.URLExemptionFilter} uses regex configuration
    + * to check if URL is eligible for exemption from 'db.ignore.external'.
    + * When this filter is enabled, urls will be checked against configured sequence of regex rules.
    + *<p>
    + * The exemption rule file defaults to db-ignore-external-exemptions.txt in the classpath but can be
    + * overridden using the property  <code>"db.ignore.external.exemptions.file" in ./conf/nutch-*.xml</code>
    + *</p>
    + *
    + * The exemption rules are specified in plain text file where each line is a rule.
    + *
    + * When the url matches regex it is exempted from 'db.ignore.external...'<br/>
    + * <h3>Examples:</h3>
    + * <ol>
    + *   <li>
    + *     <b>Exempt urls ending with .jpg or .png or gif</b>
    + *      <br/><code>.*\.(jpg|JPG|png$|PNG|gif|GIF)$</code><br/><br/>
    + *   </li>
    + * </ol>
    + </pre>
    + *
    + * @since Feb 10, 2016
    + * @version 1
    + * @see URLExemptionFilter
    + */
    +public class ExemptionUrlFilter implements URLExemptionFilter {
    +
    +  public static final String DB_IGNORE_EXTERNAL_EXEMPTIONS_FILE = "db.ignore.external.exemptions.file";
    +  private static final Logger LOG = LoggerFactory.getLogger(ExemptionUrlFilter.class);
    +  private static ExemptionUrlFilter INSTANCE;
    +
    +  private List<Pattern> exemptions;
    +  private Configuration conf;
    +  private boolean enabled;
    +
    +  public static ExemptionUrlFilter getInstance() {
    +    if(INSTANCE == null) {
    +      synchronized (ExemptionUrlFilter.class) {
    +        if (INSTANCE == null) {
    +          INSTANCE = new ExemptionUrlFilter();
    +          INSTANCE.setConf(NutchConfiguration.create());
    +        }
    +      }
    +    }
    +    return INSTANCE;
    +  }
    +
    +  public boolean isEnabled() {
    +    return enabled;
    +  }
    +
    +  public List<Pattern> getExemptions() {
    +    return exemptions;
    +  }
    +
    +  @Override
    +  public boolean filter(String fromUrl, String toUrl) {
    +    //this implementation doesnt do anything with fromUrl
    +    if (exemptions != null) {
    +      for (Pattern pattern : exemptions) {
    +        if (pattern.matcher(toUrl).matches()) {
    --- End diff --
    
    Totally agreed :+1: 
    
    My previous attempt to extend `org.apache.nutch.urlfilter.api.RegexURLFilterBase` ,
    failed due to my limited knowledge of ant build.
    
    I will give it another try.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nutch pull request: NUTCH-2144 : override db.ignore.external to ex...

Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:

    https://github.com/apache/nutch/pull/89


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nutch pull request: NUTCH-2144 : override db.ignore.external to ex...

Posted by thammegowda <gi...@git.apache.org>.
Github user thammegowda commented on a diff in the pull request:

    https://github.com/apache/nutch/pull/89#discussion_r52693529
  
    --- Diff: conf/db-ignore-external-exemptions.txt ---
    @@ -0,0 +1,21 @@
    +# Exemption rules to db.ignore.external.links
    --- End diff --
    
    +1,
    
    Updated the file with AL2.0 header


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nutch pull request: NUTCH-2144 : override db.ignore.external to ex...

Posted by sebastian-nagel <gi...@git.apache.org>.
Github user sebastian-nagel commented on a diff in the pull request:

    https://github.com/apache/nutch/pull/89#discussion_r52833002
  
    --- Diff: src/plugin/urlfilter-ignoreexempt/src/java/org/apache/nutch/urlfilter/ignoreexempt/ExemptionUrlFilter.java ---
    @@ -0,0 +1,144 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *     http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +package org.apache.nutch.urlfilter.ignoreexempt;
    +
    +import org.apache.commons.io.IOUtils;
    +import org.apache.hadoop.conf.Configuration;
    +import org.apache.nutch.net.URLExemptionFilter;
    +import org.apache.nutch.util.NutchConfiguration;
    +import org.slf4j.Logger;
    +import org.slf4j.LoggerFactory;
    +
    +import java.io.IOException;
    +import java.io.InputStream;
    +import java.util.Arrays;
    +import java.util.regex.Pattern;
    +import java.util.List;
    +import java.util.ArrayList;
    +
    +
    +/**
    + * This implementation of {@link org.apache.nutch.net.URLExemptionFilter} uses regex configuration
    + * to check if URL is eligible for exemption from 'db.ignore.external'.
    + * When this filter is enabled, urls will be checked against configured sequence of regex rules.
    + *<p>
    + * The exemption rule file defaults to db-ignore-external-exemptions.txt in the classpath but can be
    + * overridden using the property  <code>"db.ignore.external.exemptions.file" in ./conf/nutch-*.xml</code>
    + *</p>
    + *
    + * The exemption rules are specified in plain text file where each line is a rule.
    + *
    + * When the url matches regex it is exempted from 'db.ignore.external...'<br/>
    + * <h3>Examples:</h3>
    + * <ol>
    + *   <li>
    + *     <b>Exempt urls ending with .jpg or .png or gif</b>
    + *      <br/><code>.*\.(jpg|JPG|png$|PNG|gif|GIF)$</code><br/><br/>
    + *   </li>
    + * </ol>
    + </pre>
    + *
    + * @since Feb 10, 2016
    + * @version 1
    + * @see URLExemptionFilter
    + */
    +public class ExemptionUrlFilter implements URLExemptionFilter {
    +
    +  public static final String DB_IGNORE_EXTERNAL_EXEMPTIONS_FILE = "db.ignore.external.exemptions.file";
    +  private static final Logger LOG = LoggerFactory.getLogger(ExemptionUrlFilter.class);
    +  private static ExemptionUrlFilter INSTANCE;
    +
    +  private List<Pattern> exemptions;
    +  private Configuration conf;
    +  private boolean enabled;
    +
    +  public static ExemptionUrlFilter getInstance() {
    +    if(INSTANCE == null) {
    +      synchronized (ExemptionUrlFilter.class) {
    +        if (INSTANCE == null) {
    +          INSTANCE = new ExemptionUrlFilter();
    +          INSTANCE.setConf(NutchConfiguration.create());
    +        }
    +      }
    +    }
    +    return INSTANCE;
    +  }
    +
    +  public boolean isEnabled() {
    +    return enabled;
    +  }
    +
    +  public List<Pattern> getExemptions() {
    +    return exemptions;
    +  }
    +
    +  @Override
    +  public boolean filter(String fromUrl, String toUrl) {
    +    //this implementation doesnt do anything with fromUrl
    +    if (exemptions != null) {
    +      for (Pattern pattern : exemptions) {
    +        if (pattern.matcher(toUrl).matches()) {
    +          return true; //If a regex matches, then exempted
    +        }
    +      }
    +    }
    +    //not exempted
    +    return false;
    +  }
    +
    +  @Override
    +  public void setConf(Configuration conf) {
    +    this.conf = conf;
    +    LOG.info("Ignore exemptions enabled");
    +    String fileName = this.conf.get(DB_IGNORE_EXTERNAL_EXEMPTIONS_FILE);
    +    InputStream stream = this.conf.getConfResourceAsInputStream(fileName);
    +    if (stream == null) {
    +      throw new RuntimeException("Couldn't find config file :" + fileName);
    +    }
    +    try {
    +      this.exemptions = new ArrayList<Pattern>();
    +      List<String> lines = IOUtils.readLines(stream);
    +      for (String line : lines) {
    +        line = line.trim();
    +        if (line.startsWith("#") || line.isEmpty()) {
    +          continue; //Skip : comment line or empty line
    +        }
    +        Pattern compiled = Pattern.compile(line);
    +        LOG.info("Regex :: {} ", line);
    +        exemptions.add(compiled);
    +      }
    +      LOG.info("Read {} rules from {}", exemptions.size(), fileName);
    +    } catch (IOException e) {
    +      throw new IllegalStateException(e);
    --- End diff --
    
     Does the definition of an [IllegalStateException](https://docs.oracle.com/javase/7/docs/api/java/lang/IllegalStateException.html) really fit this situation? Could be a also a RuntimeException.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nutch pull request: NUTCH-2144 : override db.ignore.external to ex...

Posted by sebastian-nagel <gi...@git.apache.org>.
Github user sebastian-nagel commented on a diff in the pull request:

    https://github.com/apache/nutch/pull/89#discussion_r52833922
  
    --- Diff: src/plugin/urlfilter-ignoreexempt/README.md ---
    @@ -0,0 +1,52 @@
    +urlfilter-ignoreexempt
    +======================
    +  This plugin allows certain urls to be exempted when the external links are configured to be ignored.
    +  This is useful when focused crawl is setup but some resources like static files are linked from CDNs (external domains).
    +
    +How to enable ?
    +==============
    +Add `urlfilter-ignoreexempt` value to `plugin.includes` property
    +```xml
    +<property>
    +  <name>plugin.includes</name>
    +  <value>protocol-http|urlfilter-(regex|ignoreexempt)...</value>
    +</property>
    +```
    +
    +How to configure rules?
    +================
    +
    +open `conf/db-ignore-external-exemptions.txt` and add rules
    +
    +#### Format :
    +
    +```
    +UrlRegex1
    +UrlRegex2
    +UrlRegex3
    +```
    +
    +
    +#### NOTE ::
    + 1. If an url matches any of the given regexps then that url is exempted.
    + 2. \# in the beginning makes it a comment line
    + 3. To Test the regex, update this file and use the below command
    +    bin/nutch plugin urlfilter-ignoreexempt org.apache.nutch.urlfilter.ignoreexempt.ExemptionUrlFilter <URL>
    +
    +
    +#### Example :
    +
    + To exempt urls ending with image extensions, use this rule
    +
    +`.*\.(jpg|JPG|png$|PNG|gif|GIF)$# Testing`
    --- End diff --
    
    dito


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---