You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nutch.apache.org by sebastian-nagel <gi...@git.apache.org> on 2016/08/19 13:47:25 UTC

[GitHub] nutch pull request #141: NUTCH-2300 Fetcher to optionally save robots.txt

GitHub user sebastian-nagel opened a pull request:

    https://github.com/apache/nutch/pull/141

    NUTCH-2300 Fetcher to optionally save robots.txt

    If the property fetcher.store.robotstxt is set to true, Fetcher saves the robots.txt
    response (URL and Content including HTTP protocol status and metadata) in the
    segment (subfolder content/). It does not add a fetch datum, simply because this
    avoids that the robots.txt URL slips into CrawlDb or gets indexed. The robots.txt
    can then be retrieved from the segment, e.g., by
    ```
    # inject http://nutch.apache.org/
    # generate
    # and fetch with
    bin/nutch fetch -Dfetcher.store.robotstxt=true -Dfetcher.store.content=true ...path_to_segment
    
    # dump segment (without -nocontent)
    bin/nutch readseg -dump ...path_to_segment ...path_to_dump
    cat ...path_to_dump/dump
    ...
    URL:: http://nutch.apache.org/robots.txt
    
    Content::
    Version: -1
    url: http://nutch.apache.org/robots.txt
    base: http://nutch.apache.org/robots.txt
    contentType: text/html
    metadata: nutch.fetch.time=1471612087645 Server=Apache/2.4.7 (Ubuntu) Connection=close Content-Length=208 Date=Fri, 19 Aug 2016 13:08:07 GMT Content-Type=text/html; charset=iso-8859-1 
    Content:
    <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
    <html><head>
    <title>404 Not Found</title>
    </head><body>
    <h1>Not Found</h1>
    <p>The requested URL /robots.txt was not found on this server.</p>
    </body></html>
    ...
    ```

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/sebastian-nagel/nutch SaveRobotsTxt

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/nutch/pull/141.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #141
    
----
commit 6c9cca5e55e43458cbc5e59b8591e4d27ac425a2
Author: Sebastian Nagel <sn...@apache.org>
Date:   2016-05-25T12:24:11Z

    Allow Fetcher to optionally store robots.txt content (if property fetcher.store.robotstxt == true).
    Improved RobotRulesParser command-line tool.

commit 264eea01a4d868578dcf641d6ce405444d276929
Author: Sebastian Nagel <sn...@apache.org>
Date:   2016-08-19T13:06:14Z

    Ignore robots.txt when parsing segment, refactored storing of robots.txt in FetcherThread

commit 33cdca76ac91a63445d4e761081e8124a23413af
Author: Sebastian Nagel <sn...@apache.org>
Date:   2016-08-19T13:32:34Z

    add hint and log warning that fetcher.store.robotstxt works only in combination with fetcher.store.content

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nutch pull request #141: NUTCH-2300 Fetcher to optionally save robots.txt

Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:

    https://github.com/apache/nutch/pull/141


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nutch pull request #141: NUTCH-2300 Fetcher to optionally save robots.txt

Posted by lewismc <gi...@git.apache.org>.
Github user lewismc commented on a diff in the pull request:

    https://github.com/apache/nutch/pull/141#discussion_r75551347
  
    --- Diff: src/plugin/lib-http/src/java/org/apache/nutch/protocol/http/api/HttpBase.java ---
    @@ -563,23 +558,25 @@ protected static void main(HttpBase http, String[] args) throws Exception {
       protected abstract Response getResponse(URL url, CrawlDatum datum,
           boolean followRedirects) throws ProtocolException, IOException;
     
    -  public BaseRobotRules getRobotRules(Text url, CrawlDatum datum) {
    -    return robots.getRobotRulesSet(this, url);
    +  @Override
    +  public BaseRobotRules getRobotRules(Text url, CrawlDatum datum,
    +      List<Content> robotsTxtContent) {
    +    return robots.getRobotRulesSet(this, url, robotsTxtContent);
       }
       
       /**
        * Transforming a String[] into a HashMap for faster searching
        * @param input String[]
        * @return a new HashMap
        */
    -  private HashMap arrayToMap(String[]input){
    -    if (input==null ||input.length==0) {
    -      return new HashMap();
    +  private HashMap<String, String> arrayToMap(String[] input) {
    +    if (input == null || input.length == 0) {
    +      return new HashMap<String, String>();
         }
    -    HashMap hm=new HashMap();
    -    for (int i=0;i<input.length;i++){
    -      if (!"".equals(input[i].trim())){
    -        hm.put(input[i],input[i]);
    +    HashMap<String, String> hm = new HashMap<String, String>();
    --- End diff --
    
    Can just use diamond operator on RHS


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---