[jira] [Commented] (NUTCH-1749) Optionally exclude title from content field

["ASF GitHub Bot (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/NUTCH-1749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16922386#comment-16922386 ] 

ASF GitHub Bot commented on NUTCH-1749:
---------------------------------------

jorgelbg commented on pull request #285: fix for NUTCH-1749 contributed by steeveb972
URL: https://github.com/apache/nutch/pull/285#discussion_r320669571
 
 

 ##########
 File path: src/plugin/parse-html/src/java/org/apache/nutch/parse/html/DOMContentUtils.java
 ##########
 @@ -116,36 +114,50 @@ public void setConf(Configuration conf) {
    * 
    * @return true if nested anchors were found
    */
-  public boolean getText(StringBuffer sb, Node node,
-      boolean abortOnNestedAnchors) {
-    if (getTextHelper(sb, node, abortOnNestedAnchors, 0)) {
+  private boolean getText(StringBuffer sb, Node node,
+                          boolean abortOnNestedAnchors, Set<String> excludedElementNames) {
 
 Review comment:
   Formatting
 
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


> Optionally exclude title from content field
> -------------------------------------------
>
>                 Key: NUTCH-1749
>                 URL: https://issues.apache.org/jira/browse/NUTCH-1749
>             Project: Nutch
>          Issue Type: Improvement
>          Components: parser
>    Affects Versions: 1.7
>            Reporter: Greg Padiasek
>            Priority: Major
>             Fix For: 1.16
>
>         Attachments: DOMContentUtils.patch
>
>
> The HTML parser plugin inserts document title into document content. Since the title alone can be retrieved via DOMContentUtils.getTitle() and content is retrieved via DOMContentUtils.getText(), there is no need to duplicate title in the content. When title is included in the content it becomes difficult/impossible to extract document body without title. A need to extract document body without title is visible when user wants to index or display body and title separately.
> Attached is a patch which prevents including title in document content in the HTML parser plugin.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)