You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@nutch.apache.org by Doug Baber <do...@yahoo.com.INVALID> on 2014/07/10 22:06:23 UTC

Excluding parts of the HTML from the content field

I am crawling a set of web pages that are generated by ASP.  ASP uses a large VIEWSTATE variable to track information about what the user is doing.  I'm including the "content" field from nutch in my solr index, but because the viewstate variable is very large, I hit my content size limit with very little of the real page content put in the index.  

Is there a good way to exclude sections of the html from the "content" field with the existing code?  I've seen some comments in the archive, but nothing clear.  

If I write my own plug-in for it, would it be better to write a parser plugin or an index plugin to exclude it?  It seems that the index-basic plugin is the one that cuts the size down, so the exclusion would need to happen for that one runs.  

The pattern for the viewstate variable in the html is "<input type="hidden" name="__VIEWSTATE" id="__VIEWSTATE" value=".....

Any input on how to do this would be appreciated.