You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nutch.apache.org by "Nathan Gass (JIRA)" <ji...@apache.org> on 2012/11/09 17:44:12 UTC
[jira] [Created] (NUTCH-1495) -normalize and -filter for updatedb
command in nutch 2.x
Nathan Gass created NUTCH-1495:
----------------------------------
Summary: -normalize and -filter for updatedb command in nutch 2.x
Key: NUTCH-1495
URL: https://issues.apache.org/jira/browse/NUTCH-1495
Project: Nutch
Issue Type: Improvement
Affects Versions: 2.2
Reporter: Nathan Gass
AFAIS in nutch 1.x you could change your url filters and normalizers during the crawl, and update the db using crawldb -normalize -filter. There does not seem to be a away to achieve the same in nutch 2.x?
Anyway, I went ahead and tried to implement -normalize and -filter for the nutch 2.x updatedb command. I have no experience with any of the used technologies including java, so please check the attached code carefully before using it. I'm very interested to hear if this is the right approach or any other comments.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (NUTCH-1495) -normalize and -filter for updatedb
command in nutch 2.x
Posted by "Nathan Gass (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-1495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13494113#comment-13494113 ]
Nathan Gass commented on NUTCH-1495:
------------------------------------
P.S. I only tested the filter part of the patch, new normalizations are not yet tested and I noticed at least one bug in the patch for normalizations. I'll add a new patch as soon as I got around to test normalizations.
> -normalize and -filter for updatedb command in nutch 2.x
> --------------------------------------------------------
>
> Key: NUTCH-1495
> URL: https://issues.apache.org/jira/browse/NUTCH-1495
> Project: Nutch
> Issue Type: Improvement
> Affects Versions: 2.2
> Reporter: Nathan Gass
> Attachments: patch-updatedb-normalize-filter-2012-11-09.txt
>
>
> AFAIS in nutch 1.x you could change your url filters and normalizers during the crawl, and update the db using crawldb -normalize -filter. There does not seem to be a away to achieve the same in nutch 2.x?
> Anyway, I went ahead and tried to implement -normalize and -filter for the nutch 2.x updatedb command. I have no experience with any of the used technologies including java, so please check the attached code carefully before using it. I'm very interested to hear if this is the right approach or any other comments.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (NUTCH-1495) -normalize and -filter for updatedb
command in nutch 2.x
Posted by "Ferdy Galema (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-1495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13500974#comment-13500974 ]
Ferdy Galema commented on NUTCH-1495:
-------------------------------------
Fair enough.
I understand the reasoning of deleting the normalized rows. I don't think we need to implement the normalizeKeep. We just need to realize and document the deleting behaviour thoroughly.
About the inlinks/outlinks issue: My bad, I made a mistake thinking that the inlinks are not cleared prior to determining the new inlinks. As I've just checked it seems that they *are* cleared (see inlinkedScoreData.clear() in DbUpdateReducer). It depends on the DataStore how the clearing of entire maps are implemented. In any case it's not relevant to this issue.
> -normalize and -filter for updatedb command in nutch 2.x
> --------------------------------------------------------
>
> Key: NUTCH-1495
> URL: https://issues.apache.org/jira/browse/NUTCH-1495
> Project: Nutch
> Issue Type: Improvement
> Affects Versions: 2.2
> Reporter: Nathan Gass
> Attachments: patch-updatedb-normalize-filter-2012-11-09.txt, patch-updatedb-normalize-filter-2012-11-13.txt
>
>
> AFAIS in nutch 1.x you could change your url filters and normalizers during the crawl, and update the db using crawldb -normalize -filter. There does not seem to be a away to achieve the same in nutch 2.x?
> Anyway, I went ahead and tried to implement -normalize and -filter for the nutch 2.x updatedb command. I have no experience with any of the used technologies including java, so please check the attached code carefully before using it. I'm very interested to hear if this is the right approach or any other comments.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (NUTCH-1495) -normalize and -filter for updatedb
command in nutch 2.x
Posted by "Nathan Gass (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-1495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Nathan Gass updated NUTCH-1495:
-------------------------------
Attachment: patch-updatedb-normalize-filter-2012-11-13.txt
The attached patch shows where I'm currently standing.
normalize basically works and possible duplicate entries are handled similar to nutch 1.x (by taking the newest one).
I'm not at all sure if this is enough/the best approach. Currently fields like baseUrl are not changed. Should DbUpdater try to adapt them to the new url (by doing the same normalizations)? What about the fetched content? Another approach could be to add a new empty entry, so updatedb -normalize would actually throw away already fetched and/or parsed content of urls with new normalizations.
More testing is also necessary, but I'm waiting for comments if this approach is at all feasible before I continue working on this.
> -normalize and -filter for updatedb command in nutch 2.x
> --------------------------------------------------------
>
> Key: NUTCH-1495
> URL: https://issues.apache.org/jira/browse/NUTCH-1495
> Project: Nutch
> Issue Type: Improvement
> Affects Versions: 2.2
> Reporter: Nathan Gass
> Attachments: patch-updatedb-normalize-filter-2012-11-09.txt, patch-updatedb-normalize-filter-2012-11-13.txt
>
>
> AFAIS in nutch 1.x you could change your url filters and normalizers during the crawl, and update the db using crawldb -normalize -filter. There does not seem to be a away to achieve the same in nutch 2.x?
> Anyway, I went ahead and tried to implement -normalize and -filter for the nutch 2.x updatedb command. I have no experience with any of the used technologies including java, so please check the attached code carefully before using it. I'm very interested to hear if this is the right approach or any other comments.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (NUTCH-1495) -normalize and -filter for updatedb
command in nutch 2.x
Posted by "Nathan Gass (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-1495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13500943#comment-13500943 ]
Nathan Gass commented on NUTCH-1495:
------------------------------------
I remember running in an Exception when directly adding the new normalized links to the outlinks, thats why I used a newNormalizations map. Removing did, until now at least, not throw any exception, perhaps to removeFromOutlinks method or the getOutlinks method is helping here (as said I'm no java programmer)?
About testing, my current setup is actually not distributed (so all my tests where in local mode) and I did not yet look into nutch 1.x tests. If they have anything about crawldb -normalize -filter I could reuse that. I assume this two are the minimum to get the patch in. If anything else is missing, please let me know.
I'm currently of the opinion, that just removing keys which were normalized is the best default approach. The newly normalized outlinks will add a new entry if necessary and we avoid any possible inconsistencies at the cost of some refetches. Moreover this avoids the additional costs of having to read and write all webpage fields when -normalize is enabled.
My own use-case is to remove dupes because of previously unknown session ids so I'll have most normalized urls already in the db anyway.
We could add an additional option -normalizeKeep or similar for the dangerous and costly variant which tries to avoid the refetches. But given that we avoid a lot of the complexity of the second patch if we just not support this, I'm compelled to leave this feature out.
I don't understand why inlinks and outlinks could get out of sync. I will have to think more about it when I have time.
> -normalize and -filter for updatedb command in nutch 2.x
> --------------------------------------------------------
>
> Key: NUTCH-1495
> URL: https://issues.apache.org/jira/browse/NUTCH-1495
> Project: Nutch
> Issue Type: Improvement
> Affects Versions: 2.2
> Reporter: Nathan Gass
> Attachments: patch-updatedb-normalize-filter-2012-11-09.txt, patch-updatedb-normalize-filter-2012-11-13.txt
>
>
> AFAIS in nutch 1.x you could change your url filters and normalizers during the crawl, and update the db using crawldb -normalize -filter. There does not seem to be a away to achieve the same in nutch 2.x?
> Anyway, I went ahead and tried to implement -normalize and -filter for the nutch 2.x updatedb command. I have no experience with any of the used technologies including java, so please check the attached code carefully before using it. I'm very interested to hear if this is the right approach or any other comments.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (NUTCH-1495) -normalize and -filter for updatedb
command in nutch 2.x
Posted by "Nathan Gass (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-1495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Nathan Gass updated NUTCH-1495:
-------------------------------
Attachment: patch-updatedb-normalize-filter-2012-11-09.txt
> -normalize and -filter for updatedb command in nutch 2.x
> --------------------------------------------------------
>
> Key: NUTCH-1495
> URL: https://issues.apache.org/jira/browse/NUTCH-1495
> Project: Nutch
> Issue Type: Improvement
> Affects Versions: 2.2
> Reporter: Nathan Gass
> Attachments: patch-updatedb-normalize-filter-2012-11-09.txt
>
>
> AFAIS in nutch 1.x you could change your url filters and normalizers during the crawl, and update the db using crawldb -normalize -filter. There does not seem to be a away to achieve the same in nutch 2.x?
> Anyway, I went ahead and tried to implement -normalize and -filter for the nutch 2.x updatedb command. I have no experience with any of the used technologies including java, so please check the attached code carefully before using it. I'm very interested to hear if this is the right approach or any other comments.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (NUTCH-1495) -normalize and -filter for updatedb
command in nutch 2.x
Posted by "Ferdy Galema (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-1495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13500895#comment-13500895 ]
Ferdy Galema commented on NUTCH-1495:
-------------------------------------
Hi,
Nice one! I took a glance at your patch and it certainly looks all right. Have you tested it some more?
Normalization will always be a hard problem to solve, especially when rules are changes over time. I wouldn't worry to much about baseUrl or content. Those are "derived" fields. The most important thing is the handling of the keys (reversed urls) and it seems you got that right. Also note that inlinks might not always be in sync with outlinks, because outlinks might be removed after normalization. (But in a previous iteration they could already been added to inlinks during a db update). There might be other consequences of changing normalization rules during a fetch, that I can't come up with now.
There might be a small Java iterating issue in the way you normalize outlinks. Namely if you iterate over a collection without a proper iterator you might get a ConcurrentModificationException.
http://stackoverflow.com/questions/1884889/iterating-over-and-removing-from-a-map
This is just a minor thing that is easily fixed. Either use an iterator (although I'm not sure how well this is supported within Gora) or simply save the to be deleted urls in a separate set/list and delete them afterwards.
> -normalize and -filter for updatedb command in nutch 2.x
> --------------------------------------------------------
>
> Key: NUTCH-1495
> URL: https://issues.apache.org/jira/browse/NUTCH-1495
> Project: Nutch
> Issue Type: Improvement
> Affects Versions: 2.2
> Reporter: Nathan Gass
> Attachments: patch-updatedb-normalize-filter-2012-11-09.txt, patch-updatedb-normalize-filter-2012-11-13.txt
>
>
> AFAIS in nutch 1.x you could change your url filters and normalizers during the crawl, and update the db using crawldb -normalize -filter. There does not seem to be a away to achieve the same in nutch 2.x?
> Anyway, I went ahead and tried to implement -normalize and -filter for the nutch 2.x updatedb command. I have no experience with any of the used technologies including java, so please check the attached code carefully before using it. I'm very interested to hear if this is the right approach or any other comments.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira