You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@nutch.apache.org by Vikas Hazrati <vi...@knoldus.com> on 2012/04/17 10:10:43 UTC

Getting the parsed HTML content back

Hi,

I have an HTML filter

class WorkDetailParserFilter extends HtmlParseFilter {

which gives me the HTML text back to me, however since we get

//Scala code
 def filter(content: Content, parseResult: ParseResult, metaTags:
HTMLMetaTags, doc: DocumentFragment): ParseResult = {
    parseResult.filter()
    val parsedContent = parseResult.get(content.getUrl).getText
    parseResult
  }

Now I get back raw text content back, which is fine but then in order to
retrieve the specific portions i am interested in, i have to string parse
the result.

Question

1) Is there a way (another filter) in which I get the dom structure back
which I can then parse with Xpath?
2) Any other approach to get back specific portions of the page that i am
interested in?

Regards | Vikas
www.knoldus.com

Re: Getting the parsed HTML content back

Posted by Vikas Hazrati <vi...@knoldus.com>.

Thanks Sebastian,

I already started working on the DOM approach and it seems to be giving me
what i need for now. Thanks for mentioning the plugins, I would look at
them soon.

Regards | Vikas
www.knoldus.com

On Fri, Apr 20, 2012 at 1:45 AM, Sebastian Nagel <wastl.nagel@googlemail.com
> wrote:

> Hi Vikas,
>
>
> > 1) Is there a way (another filter) in which I get the dom structure back
> > which I can then parse with Xpath?
>
> The filter method takes the DOM as one of it's arguments (DocumentFragment
> doc).
> You can use it as input of XPathExpression.evaluate(doc, ...)
> CAVEAT: XPath is case-sensitive while casing of element and attribute names
> in the DOM depends on the choosen parser.
>
>
> > 2) Any other approach to get back specific portions of the page that i am
> > interested in?
>
> Have a look at the following plugins: microformats-reltag, parse-metatags,
> or headings.
>
> Sebastian
>
>
> On 04/17/2012 10:10 AM, Vikas Hazrati wrote:
>
>> Hi,
>>
>> I have an HTML filter
>>
>> class WorkDetailParserFilter extends HtmlParseFilter {
>>
>> which gives me the HTML text back to me, however since we get
>>
>> //Scala code
>>  def filter(content: Content, parseResult: ParseResult, metaTags:
>> HTMLMetaTags, doc: DocumentFragment): ParseResult = {
>>     parseResult.filter()
>>     val parsedContent = parseResult.get(content.**getUrl).getText
>>     parseResult
>>   }
>>
>> Now I get back raw text content back, which is fine but then in order to
>> retrieve the specific portions i am interested in, i have to string parse
>> the result.
>>
>> Question
>>
>> 1) Is there a way (another filter) in which I get the dom structure back
>> which I can then parse with Xpath?
>> 2) Any other approach to get back specific portions of the page that i am
>> interested in?
>>
>> Regards | Vikas
>> www.knoldus.com
>>
>>
>

Re: Getting the parsed HTML content back

Posted by Sebastian Nagel <wa...@googlemail.com>.

Hi Vikas,

 > 1) Is there a way (another filter) in which I get the dom structure back
 > which I can then parse with Xpath?

The filter method takes the DOM as one of it's arguments (DocumentFragment doc).
You can use it as input of XPathExpression.evaluate(doc, ...)
CAVEAT: XPath is case-sensitive while casing of element and attribute names
in the DOM depends on the choosen parser.

 > 2) Any other approach to get back specific portions of the page that i am
 > interested in?

Have a look at the following plugins: microformats-reltag, parse-metatags, or headings.

Sebastian

On 04/17/2012 10:10 AM, Vikas Hazrati wrote:
> Hi,
>
> I have an HTML filter
>
> class WorkDetailParserFilter extends HtmlParseFilter {
>
> which gives me the HTML text back to me, however since we get
>
> //Scala code
>   def filter(content: Content, parseResult: ParseResult, metaTags:
> HTMLMetaTags, doc: DocumentFragment): ParseResult = {
>      parseResult.filter()
>      val parsedContent = parseResult.get(content.getUrl).getText
>      parseResult
>    }
>
> Now I get back raw text content back, which is fine but then in order to
> retrieve the specific portions i am interested in, i have to string parse
> the result.
>
> Question
>
> 1) Is there a way (another filter) in which I get the dom structure back
> which I can then parse with Xpath?
> 2) Any other approach to get back specific portions of the page that i am
> interested in?
>
> Regards | Vikas
> www.knoldus.com
>