You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nutch.apache.org by "Markus Jelsma (Created) (JIRA)" <ji...@apache.org> on 2012/01/04 08:59:38 UTC
[jira] [Created] (NUTCH-1241) CrawlDBScanner should also be able to
find records
CrawlDBScanner should also be able to find records
--------------------------------------------------
Key: NUTCH-1241
URL: https://issues.apache.org/jira/browse/NUTCH-1241
Project: Nutch
Issue Type: Improvement
Reporter: Markus Jelsma
Assignee: Markus Jelsma
Priority: Minor
Fix For: 1.5
The CrawlDBScanner cannot find partial matches because it uses String.match(); Instead, it should be able to use the Matcher.find() to find partial matches. Right now regex "http" will never match any records. It can then also reuse a compiled pattern.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (NUTCH-1241) CrawlDBScanner should also be able
to find records
Posted by "Markus Jelsma (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-1241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13179386#comment-13179386 ]
Markus Jelsma commented on NUTCH-1241:
--------------------------------------
hmm yes, if we add a -regex option that goes with the -dump option in the reader we can also have csv output! However, due to NUTCH-1029 i cannot test it properly in a production environment. Care to have a look?
> CrawlDBScanner should also be able to find records
> --------------------------------------------------
>
> Key: NUTCH-1241
> URL: https://issues.apache.org/jira/browse/NUTCH-1241
> Project: Nutch
> Issue Type: Improvement
> Reporter: Markus Jelsma
> Assignee: Markus Jelsma
> Priority: Minor
> Fix For: 1.5
>
>
> The CrawlDBScanner cannot find partial matches because it uses String.match(); Instead, it should be able to use the Matcher.find() to find partial matches. Right now regex "http" will never match any records. It can then also reuse a compiled pattern.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (NUTCH-1241) CrawlDBScanner should also be able
to find records
Posted by "Julien Nioche (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-1241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13179382#comment-13179382 ]
Julien Nioche commented on NUTCH-1241:
--------------------------------------
Entering '.+/product/.*' is not that user-unfriendly. As for performance I don't think the actual regex takes much of the time here. Feel free to change it to find if you feel strongly about it though :-) What we should probably do is incorporate this functionality wihtin the crawldb reader instead of having a separate class for it.
> CrawlDBScanner should also be able to find records
> --------------------------------------------------
>
> Key: NUTCH-1241
> URL: https://issues.apache.org/jira/browse/NUTCH-1241
> Project: Nutch
> Issue Type: Improvement
> Reporter: Markus Jelsma
> Assignee: Markus Jelsma
> Priority: Minor
> Fix For: 1.5
>
>
> The CrawlDBScanner cannot find partial matches because it uses String.match(); Instead, it should be able to use the Matcher.find() to find partial matches. Right now regex "http" will never match any records. It can then also reuse a compiled pattern.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (NUTCH-1241) CrawlDBScanner should also be able
to find records
Posted by "Markus Jelsma (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-1241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13179517#comment-13179517 ]
Markus Jelsma commented on NUTCH-1241:
--------------------------------------
Ugh, that's NUTCH-1084 instead. I don't need that ticket to test the regex dump because dumping the DB still works, reading a record doesn't.
> CrawlDBScanner should also be able to find records
> --------------------------------------------------
>
> Key: NUTCH-1241
> URL: https://issues.apache.org/jira/browse/NUTCH-1241
> Project: Nutch
> Issue Type: Improvement
> Reporter: Markus Jelsma
> Assignee: Markus Jelsma
> Priority: Minor
> Fix For: 1.5
>
>
> The CrawlDBScanner cannot find partial matches because it uses String.match(); Instead, it should be able to use the Matcher.find() to find partial matches. Right now regex "http" will never match any records. It can then also reuse a compiled pattern.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (NUTCH-1241) CrawlDBScanner should also be able
to find records
Posted by "Julien Nioche (Resolved) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-1241?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Julien Nioche resolved NUTCH-1241.
----------------------------------
Resolution: Not A Problem
just add a more complete regex e.g. 'http.+'
> CrawlDBScanner should also be able to find records
> --------------------------------------------------
>
> Key: NUTCH-1241
> URL: https://issues.apache.org/jira/browse/NUTCH-1241
> Project: Nutch
> Issue Type: Improvement
> Reporter: Markus Jelsma
> Assignee: Markus Jelsma
> Priority: Minor
> Fix For: 1.5
>
>
> The CrawlDBScanner cannot find partial matches because it uses String.match(); Instead, it should be able to use the Matcher.find() to find partial matches. Right now regex "http" will never match any records. It can then also reuse a compiled pattern.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (NUTCH-1241) CrawlDBScanner should also be able
to find records
Posted by "Markus Jelsma (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-1241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13179374#comment-13179374 ]
Markus Jelsma commented on NUTCH-1241:
--------------------------------------
yes of course but it is not user friendly. You cannot search for product in http://host/product/123 in a user friendly manner. Also, using a Matcher would slighly boost performance.
> CrawlDBScanner should also be able to find records
> --------------------------------------------------
>
> Key: NUTCH-1241
> URL: https://issues.apache.org/jira/browse/NUTCH-1241
> Project: Nutch
> Issue Type: Improvement
> Reporter: Markus Jelsma
> Assignee: Markus Jelsma
> Priority: Minor
> Fix For: 1.5
>
>
> The CrawlDBScanner cannot find partial matches because it uses String.match(); Instead, it should be able to use the Matcher.find() to find partial matches. Right now regex "http" will never match any records. It can then also reuse a compiled pattern.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (NUTCH-1241) CrawlDBScanner should also be able
to find records
Posted by "Julien Nioche (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-1241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13179501#comment-13179501 ]
Julien Nioche commented on NUTCH-1241:
--------------------------------------
bq. However, due to NUTCH-1029 i cannot test it properly in a production environment. Care to have a look?
Hasn't 1029 been fixed? What is it that you want to test?
> CrawlDBScanner should also be able to find records
> --------------------------------------------------
>
> Key: NUTCH-1241
> URL: https://issues.apache.org/jira/browse/NUTCH-1241
> Project: Nutch
> Issue Type: Improvement
> Reporter: Markus Jelsma
> Assignee: Markus Jelsma
> Priority: Minor
> Fix For: 1.5
>
>
> The CrawlDBScanner cannot find partial matches because it uses String.match(); Instead, it should be able to use the Matcher.find() to find partial matches. Right now regex "http" will never match any records. It can then also reuse a compiled pattern.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira