You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nutch.apache.org by "Xue Yong Zhi (JIRA)" <ji...@apache.org> on 2008/05/26 09:27:55 UTC
[jira] Created: (NUTCH-633) ParseSegment no longer allow reparsing
ParseSegment no longer allow reparsing
--------------------------------------
Key: NUTCH-633
URL: https://issues.apache.org/jira/browse/NUTCH-633
Project: Nutch
Issue Type: Bug
Affects Versions: 1.0.0
Environment: any
Reporter: Xue Yong Zhi
Priority: Minor
ParseSegment used to allow reparsing even if parsing has been enabled in Fetcher. But now it throws a NumberFormatException as 'content.getMetadata().get(Nutch.FETCH_STATUS_KEY)' is null.
This patch will fix the problem:
--- a/src/java/org/apache/nutch/parse/ParseSegment.java
+++ b/src/java/org/apache/nutch/parse/ParseSegment.java
@@ -70,8 +70,10 @@ public class ParseSegment extends Configured implements Tool, Mapper<WritableCom
key = newKey;
}
+ //status_key is only available when parsing is not done in fetcher
+ String status_key = content.getMetadata().get(Nutch.FETCH_STATUS_KEY);
int status =
- Integer.parseInt(content.getMetadata().get(Nutch.FETCH_STATUS_KEY));
+ (null == status_key) ? CrawlDatum.STATUS_FETCH_SUCCESS : Integer.parseInt(status_key);
if (status != CrawlDatum.STATUS_FETCH_SUCCESS) {
// content not fetched successfully, skip document
LOG.debug("Skipping " + key + " as content is not fetched successfully");
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
Re: [jira] Created: (NUTCH-633) ParseSegment no longer allow
reparsing
Posted by Grease <gi...@aplopio.com>.
Hi -
I'm getting the same problem with Nutch 0.9 - wondering if the patch applies
for that too.
- Grease
JIRA jira@apache.org wrote:
>
> ParseSegment no longer allow reparsing
> --------------------------------------
>
> Key: NUTCH-633
> URL: https://issues.apache.org/jira/browse/NUTCH-633
> Project: Nutch
> Issue Type: Bug
> Affects Versions: 1.0.0
> Environment: any
> Reporter: Xue Yong Zhi
> Priority: Minor
>
>
> ParseSegment used to allow reparsing even if parsing has been enabled in
> Fetcher. But now it throws a NumberFormatException as
> 'content.getMetadata().get(Nutch.FETCH_STATUS_KEY)' is null.
>
> This patch will fix the problem:
>
> --- a/src/java/org/apache/nutch/parse/ParseSegment.java
> +++ b/src/java/org/apache/nutch/parse/ParseSegment.java
> @@ -70,8 +70,10 @@ public class ParseSegment extends Configured implements
> Tool, Mapper<WritableCom
> key = newKey;
> }
>
> + //status_key is only available when parsing is not done in fetcher
> + String status_key =
> content.getMetadata().get(Nutch.FETCH_STATUS_KEY);
> int status =
> -
> Integer.parseInt(content.getMetadata().get(Nutch.FETCH_STATUS_KEY));
> + (null == status_key) ? CrawlDatum.STATUS_FETCH_SUCCESS :
> Integer.parseInt(status_key);
> if (status != CrawlDatum.STATUS_FETCH_SUCCESS) {
> // content not fetched successfully, skip document
> LOG.debug("Skipping " + key + " as content is not fetched
> successfully");
>
>
> --
> This message is automatically generated by JIRA.
> -
> You can reply to this email to add a comment to the issue online.
>
>
>
--
View this message in context: http://www.nabble.com/-jira--Created%3A-%28NUTCH-633%29-ParseSegment-no-longer-allow-reparsing-tp17467079p21760251.html
Sent from the Nutch - Dev mailing list archive at Nabble.com.
[jira] Commented: (NUTCH-633) ParseSegment no longer allow
reparsing
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12633615#action_12633615 ]
Hudson commented on NUTCH-633:
------------------------------
Integrated in Nutch-trunk #580 (See [http://hudson.zones.apache.org/hudson/job/Nutch-trunk/580/])
> ParseSegment no longer allow reparsing
> --------------------------------------
>
> Key: NUTCH-633
> URL: https://issues.apache.org/jira/browse/NUTCH-633
> Project: Nutch
> Issue Type: Bug
> Affects Versions: 1.0.0
> Environment: any
> Reporter: Xue Yong Zhi
> Assignee: Doğacan Güney
> Priority: Minor
> Fix For: 1.0.0
>
> Attachments: NUTCH_633.patch
>
>
> ParseSegment used to allow reparsing even if parsing has been enabled in Fetcher. But now it throws a NumberFormatException as 'content.getMetadata().get(Nutch.FETCH_STATUS_KEY)' is null.
> This patch will fix the problem:
> --- a/src/java/org/apache/nutch/parse/ParseSegment.java
> +++ b/src/java/org/apache/nutch/parse/ParseSegment.java
> @@ -70,8 +70,10 @@ public class ParseSegment extends Configured implements Tool, Mapper<WritableCom
> key = newKey;
> }
>
> + //status_key is only available when parsing is not done in fetcher
> + String status_key = content.getMetadata().get(Nutch.FETCH_STATUS_KEY);
> int status =
> - Integer.parseInt(content.getMetadata().get(Nutch.FETCH_STATUS_KEY));
> + (null == status_key) ? CrawlDatum.STATUS_FETCH_SUCCESS : Integer.parseInt(status_key);
> if (status != CrawlDatum.STATUS_FETCH_SUCCESS) {
> // content not fetched successfully, skip document
> LOG.debug("Skipping " + key + " as content is not fetched successfully");
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Closed: (NUTCH-633) ParseSegment no longer allow reparsing
Posted by "Doğacan Güney (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-633?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Doğacan Güney closed NUTCH-633.
-------------------------------
Resolution: Fixed
Fix Version/s: 1.0.0
Assignee: Doğacan Güney
Patch committed as of rev. 697896.
> ParseSegment no longer allow reparsing
> --------------------------------------
>
> Key: NUTCH-633
> URL: https://issues.apache.org/jira/browse/NUTCH-633
> Project: Nutch
> Issue Type: Bug
> Affects Versions: 1.0.0
> Environment: any
> Reporter: Xue Yong Zhi
> Assignee: Doğacan Güney
> Priority: Minor
> Fix For: 1.0.0
>
> Attachments: NUTCH_633.patch
>
>
> ParseSegment used to allow reparsing even if parsing has been enabled in Fetcher. But now it throws a NumberFormatException as 'content.getMetadata().get(Nutch.FETCH_STATUS_KEY)' is null.
> This patch will fix the problem:
> --- a/src/java/org/apache/nutch/parse/ParseSegment.java
> +++ b/src/java/org/apache/nutch/parse/ParseSegment.java
> @@ -70,8 +70,10 @@ public class ParseSegment extends Configured implements Tool, Mapper<WritableCom
> key = newKey;
> }
>
> + //status_key is only available when parsing is not done in fetcher
> + String status_key = content.getMetadata().get(Nutch.FETCH_STATUS_KEY);
> int status =
> - Integer.parseInt(content.getMetadata().get(Nutch.FETCH_STATUS_KEY));
> + (null == status_key) ? CrawlDatum.STATUS_FETCH_SUCCESS : Integer.parseInt(status_key);
> if (status != CrawlDatum.STATUS_FETCH_SUCCESS) {
> // content not fetched successfully, skip document
> LOG.debug("Skipping " + key + " as content is not fetched successfully");
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (NUTCH-633) ParseSegment no longer allow reparsing
Posted by "Doğacan Güney (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-633?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Doğacan Güney updated NUTCH-633:
--------------------------------
Attachment: NUTCH_633.patch
OK, I shouldn't have missed this one :)
Anyway, I think it is better to modify the fetchers so that they always store FETCH_STATUS_KEY instead of modifying parser.
And, here is a patch which does exactly that :D
> ParseSegment no longer allow reparsing
> --------------------------------------
>
> Key: NUTCH-633
> URL: https://issues.apache.org/jira/browse/NUTCH-633
> Project: Nutch
> Issue Type: Bug
> Affects Versions: 1.0.0
> Environment: any
> Reporter: Xue Yong Zhi
> Priority: Minor
> Attachments: NUTCH_633.patch
>
>
> ParseSegment used to allow reparsing even if parsing has been enabled in Fetcher. But now it throws a NumberFormatException as 'content.getMetadata().get(Nutch.FETCH_STATUS_KEY)' is null.
> This patch will fix the problem:
> --- a/src/java/org/apache/nutch/parse/ParseSegment.java
> +++ b/src/java/org/apache/nutch/parse/ParseSegment.java
> @@ -70,8 +70,10 @@ public class ParseSegment extends Configured implements Tool, Mapper<WritableCom
> key = newKey;
> }
>
> + //status_key is only available when parsing is not done in fetcher
> + String status_key = content.getMetadata().get(Nutch.FETCH_STATUS_KEY);
> int status =
> - Integer.parseInt(content.getMetadata().get(Nutch.FETCH_STATUS_KEY));
> + (null == status_key) ? CrawlDatum.STATUS_FETCH_SUCCESS : Integer.parseInt(status_key);
> if (status != CrawlDatum.STATUS_FETCH_SUCCESS) {
> // content not fetched successfully, skip document
> LOG.debug("Skipping " + key + " as content is not fetched successfully");
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.