You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nutch.apache.org by sebastian-nagel <gi...@git.apache.org> on 2016/08/19 13:47:25 UTC
[GitHub] nutch pull request #141: NUTCH-2300 Fetcher to optionally save robots.txt
GitHub user sebastian-nagel opened a pull request:
https://github.com/apache/nutch/pull/141
NUTCH-2300 Fetcher to optionally save robots.txt
If the property fetcher.store.robotstxt is set to true, Fetcher saves the robots.txt
response (URL and Content including HTTP protocol status and metadata) in the
segment (subfolder content/). It does not add a fetch datum, simply because this
avoids that the robots.txt URL slips into CrawlDb or gets indexed. The robots.txt
can then be retrieved from the segment, e.g., by
```
# inject http://nutch.apache.org/
# generate
# and fetch with
bin/nutch fetch -Dfetcher.store.robotstxt=true -Dfetcher.store.content=true ...path_to_segment
# dump segment (without -nocontent)
bin/nutch readseg -dump ...path_to_segment ...path_to_dump
cat ...path_to_dump/dump
...
URL:: http://nutch.apache.org/robots.txt
Content::
Version: -1
url: http://nutch.apache.org/robots.txt
base: http://nutch.apache.org/robots.txt
contentType: text/html
metadata: nutch.fetch.time=1471612087645 Server=Apache/2.4.7 (Ubuntu) Connection=close Content-Length=208 Date=Fri, 19 Aug 2016 13:08:07 GMT Content-Type=text/html; charset=iso-8859-1
Content:
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>404 Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL /robots.txt was not found on this server.</p>
</body></html>
...
```
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/sebastian-nagel/nutch SaveRobotsTxt
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/nutch/pull/141.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #141
----
commit 6c9cca5e55e43458cbc5e59b8591e4d27ac425a2
Author: Sebastian Nagel <sn...@apache.org>
Date: 2016-05-25T12:24:11Z
Allow Fetcher to optionally store robots.txt content (if property fetcher.store.robotstxt == true).
Improved RobotRulesParser command-line tool.
commit 264eea01a4d868578dcf641d6ce405444d276929
Author: Sebastian Nagel <sn...@apache.org>
Date: 2016-08-19T13:06:14Z
Ignore robots.txt when parsing segment, refactored storing of robots.txt in FetcherThread
commit 33cdca76ac91a63445d4e761081e8124a23413af
Author: Sebastian Nagel <sn...@apache.org>
Date: 2016-08-19T13:32:34Z
add hint and log warning that fetcher.store.robotstxt works only in combination with fetcher.store.content
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nutch pull request #141: NUTCH-2300 Fetcher to optionally save robots.txt
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/nutch/pull/141
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nutch pull request #141: NUTCH-2300 Fetcher to optionally save robots.txt
Posted by lewismc <gi...@git.apache.org>.
Github user lewismc commented on a diff in the pull request:
https://github.com/apache/nutch/pull/141#discussion_r75551347
--- Diff: src/plugin/lib-http/src/java/org/apache/nutch/protocol/http/api/HttpBase.java ---
@@ -563,23 +558,25 @@ protected static void main(HttpBase http, String[] args) throws Exception {
protected abstract Response getResponse(URL url, CrawlDatum datum,
boolean followRedirects) throws ProtocolException, IOException;
- public BaseRobotRules getRobotRules(Text url, CrawlDatum datum) {
- return robots.getRobotRulesSet(this, url);
+ @Override
+ public BaseRobotRules getRobotRules(Text url, CrawlDatum datum,
+ List<Content> robotsTxtContent) {
+ return robots.getRobotRulesSet(this, url, robotsTxtContent);
}
/**
* Transforming a String[] into a HashMap for faster searching
* @param input String[]
* @return a new HashMap
*/
- private HashMap arrayToMap(String[]input){
- if (input==null ||input.length==0) {
- return new HashMap();
+ private HashMap<String, String> arrayToMap(String[] input) {
+ if (input == null || input.length == 0) {
+ return new HashMap<String, String>();
}
- HashMap hm=new HashMap();
- for (int i=0;i<input.length;i++){
- if (!"".equals(input[i].trim())){
- hm.put(input[i],input[i]);
+ HashMap<String, String> hm = new HashMap<String, String>();
--- End diff --
Can just use diamond operator on RHS
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---