You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nutch.apache.org by "Markus Jelsma (Created) (JIRA)" <ji...@apache.org> on 2012/01/24 18:26:42 UTC
[jira] [Created] (NUTCH-1256) WebGraph to dump host + score
WebGraph to dump host + score
-----------------------------
Key: NUTCH-1256
URL: https://issues.apache.org/jira/browse/NUTCH-1256
Project: Nutch
Issue Type: Improvement
Affects Versions: 1.4
Reporter: Markus Jelsma
Assignee: Markus Jelsma
Fix For: 1.5
WebGraph's NodeDumper tool can dump url,score information but a host|domain,score output can also be put to good use. This is likely to require a new MapReduce job as the NodeDumper's atonomy is not suited to return max or or summed scores. Code could also be merged with the tool.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (NUTCH-1256) WebGraph to dump host + score
Posted by "Markus Jelsma (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-1256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13196033#comment-13196033 ]
Markus Jelsma commented on NUTCH-1256:
--------------------------------------
comments?
> WebGraph to dump host + score
> -----------------------------
>
> Key: NUTCH-1256
> URL: https://issues.apache.org/jira/browse/NUTCH-1256
> Project: Nutch
> Issue Type: Improvement
> Affects Versions: 1.4
> Reporter: Markus Jelsma
> Assignee: Markus Jelsma
> Fix For: 1.5
>
> Attachments: NUTCH-1256-1.5-1.patch
>
>
> WebGraph's NodeDumper tool can dump url,score information but a host|domain,score output can also be put to good use. This is likely to require a new MapReduce job as the NodeDumper's atonomy is not suited to return max or or summed scores. Code could also be merged with the tool.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (NUTCH-1256) WebGraph to dump host + score
Posted by "Markus Jelsma (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-1256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13196830#comment-13196830 ]
Markus Jelsma commented on NUTCH-1256:
--------------------------------------
I'll commit this one today unless there are objections.
> WebGraph to dump host + score
> -----------------------------
>
> Key: NUTCH-1256
> URL: https://issues.apache.org/jira/browse/NUTCH-1256
> Project: Nutch
> Issue Type: Improvement
> Affects Versions: 1.4
> Reporter: Markus Jelsma
> Assignee: Markus Jelsma
> Fix For: 1.5
>
> Attachments: NUTCH-1256-1.5-1.patch
>
>
> WebGraph's NodeDumper tool can dump url,score information but a host|domain,score output can also be put to good use. This is likely to require a new MapReduce job as the NodeDumper's atonomy is not suited to return max or or summed scores. Code could also be merged with the tool.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (NUTCH-1256) WebGraph to dump host + score
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-1256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13196965#comment-13196965 ]
Hudson commented on NUTCH-1256:
-------------------------------
Integrated in nutch-trunk-maven #129 (See [https://builds.apache.org/job/nutch-trunk-maven/129/])
NUTCH-1256 WebGraph to dump host + score. Most if not all WebGraph options have been added to nutch-default as well.
markus : http://svn.apache.org/viewvc/nutch/trunk/viewvc/?view=rev&root=&revision=1238590
Files :
* /nutch/trunk/CHANGES.txt
* /nutch/trunk/conf/nutch-default.xml
* /nutch/trunk/src/java/org/apache/nutch/scoring/webgraph/NodeDumper.java
> WebGraph to dump host + score
> -----------------------------
>
> Key: NUTCH-1256
> URL: https://issues.apache.org/jira/browse/NUTCH-1256
> Project: Nutch
> Issue Type: Improvement
> Affects Versions: 1.4
> Reporter: Markus Jelsma
> Assignee: Markus Jelsma
> Fix For: 1.5
>
> Attachments: NUTCH-1256-1.5-1.patch
>
>
> WebGraph's NodeDumper tool can dump url,score information but a host|domain,score output can also be put to good use. This is likely to require a new MapReduce job as the NodeDumper's atonomy is not suited to return max or or summed scores. Code could also be merged with the tool.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (NUTCH-1256) WebGraph to dump host + score
Posted by "Markus Jelsma (Resolved) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-1256?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Markus Jelsma resolved NUTCH-1256.
----------------------------------
Resolution: Fixed
Thanks. Added some global inline comments for the methods. I also added most, if not all, WebGraph configuration options with their default settings to nutch-default.
Committed for 1.5 in rev. 1238590.
> WebGraph to dump host + score
> -----------------------------
>
> Key: NUTCH-1256
> URL: https://issues.apache.org/jira/browse/NUTCH-1256
> Project: Nutch
> Issue Type: Improvement
> Affects Versions: 1.4
> Reporter: Markus Jelsma
> Assignee: Markus Jelsma
> Fix For: 1.5
>
> Attachments: NUTCH-1256-1.5-1.patch
>
>
> WebGraph's NodeDumper tool can dump url,score information but a host|domain,score output can also be put to good use. This is likely to require a new MapReduce job as the NodeDumper's atonomy is not suited to return max or or summed scores. Code could also be merged with the tool.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (NUTCH-1256) WebGraph to dump host + score
Posted by "Markus Jelsma (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-1256?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Markus Jelsma updated NUTCH-1256:
---------------------------------
Attachment: NUTCH-1256-1.5-1.patch
Patch introduces new parameter with two mandatory arguments.
-group <host|domain> <sum|max> groups the scores by host or domain. The second argument determines whether scores are summed up or if the highest value prevails.
> WebGraph to dump host + score
> -----------------------------
>
> Key: NUTCH-1256
> URL: https://issues.apache.org/jira/browse/NUTCH-1256
> Project: Nutch
> Issue Type: Improvement
> Affects Versions: 1.4
> Reporter: Markus Jelsma
> Assignee: Markus Jelsma
> Fix For: 1.5
>
> Attachments: NUTCH-1256-1.5-1.patch
>
>
> WebGraph's NodeDumper tool can dump url,score information but a host|domain,score output can also be put to good use. This is likely to require a new MapReduce job as the NodeDumper's atonomy is not suited to return max or or summed scores. Code could also be merged with the tool.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (NUTCH-1256) WebGraph to dump host + score
Posted by "Lewis John McGibbney (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-1256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13196894#comment-13196894 ]
Lewis John McGibbney commented on NUTCH-1256:
---------------------------------------------
I like this Markus. I wonder if it is possible for you to add some in-line documentation? Or Javadoc, depends on what you wish. Also if you get time, is it possible for this to be added here
http://wiki.apache.org/nutch/bin/nutch%20nodedumper
> WebGraph to dump host + score
> -----------------------------
>
> Key: NUTCH-1256
> URL: https://issues.apache.org/jira/browse/NUTCH-1256
> Project: Nutch
> Issue Type: Improvement
> Affects Versions: 1.4
> Reporter: Markus Jelsma
> Assignee: Markus Jelsma
> Fix For: 1.5
>
> Attachments: NUTCH-1256-1.5-1.patch
>
>
> WebGraph's NodeDumper tool can dump url,score information but a host|domain,score output can also be put to good use. This is likely to require a new MapReduce job as the NodeDumper's atonomy is not suited to return max or or summed scores. Code could also be merged with the tool.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (NUTCH-1256) WebGraph to dump host + score
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-1256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13197571#comment-13197571 ]
Hudson commented on NUTCH-1256:
-------------------------------
Integrated in Nutch-trunk #1745 (See [https://builds.apache.org/job/Nutch-trunk/1745/])
NUTCH-1256 WebGraph to dump host + score. Most if not all WebGraph options have been added to nutch-default as well.
markus : http://svn.apache.org/viewvc/nutch/trunk/viewvc/?view=rev&root=.&revision=1238590
Files :
* /nutch/trunk/CHANGES.txt
* /nutch/trunk/conf/nutch-default.xml
* /nutch/trunk/src/java/org/apache/nutch/scoring/webgraph/NodeDumper.java
> WebGraph to dump host + score
> -----------------------------
>
> Key: NUTCH-1256
> URL: https://issues.apache.org/jira/browse/NUTCH-1256
> Project: Nutch
> Issue Type: Improvement
> Affects Versions: 1.4
> Reporter: Markus Jelsma
> Assignee: Markus Jelsma
> Fix For: 1.5
>
> Attachments: NUTCH-1256-1.5-1.patch
>
>
> WebGraph's NodeDumper tool can dump url,score information but a host|domain,score output can also be put to good use. This is likely to require a new MapReduce job as the NodeDumper's atonomy is not suited to return max or or summed scores. Code could also be merged with the tool.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira