Java Script Crawling using nutch

[Dayal <li...@gmail.com>]

Can nutch be used to crawl pages with javascripts in them.Will the nutch
crawl efficiently executes the js script and fetch the html file after java
script execution 

Also like in html unit is there a way to use our own custom JSExecution
engine to Nutch

Thanks in advance

--
View this message in context: http://lucene.472066.n3.nabble.com/Java-Script-Crawling-using-nutch-tp3804266p3804266.html
Sent from the Nutch - User mailing list archive at Nabble.com.

Re: Java Script Crawling using nutch

[Ferdy Galema <fe...@kalooga.com>]

Hi,

Nutch includes a parser for JS but it does not execute it or anything like
that. It simply tries to detect outlinks (so that they can be crawled in a
following crawl cycle). The plugin is default not enabled. This can be done
using the plugin.includes property.

>Also like in html unit is there a way to use our own custom JSExecution
>engine to Nutch
Not sure what mean exactly, but if you are referring to ParseFilter
framework that is only provided when parsing HTML, then the answer is no.
For JS (or any other mimetypes besides HTML for that matter) it is
currently not possible to implement additional parsing of metadata using
separate filters. You implement a single parser for those mimetypes.

Ferdy.

On Tue, Mar 6, 2012 at 6:10 PM, Dayal <li...@gmail.com> wrote:

> Can nutch be used to crawl pages with javascripts in them.Will the nutch
> crawl efficiently executes the js script and fetch the html file after java
> script execution
>
> Also like in html unit is there a way to use our own custom JSExecution
> engine to Nutch
>
> Thanks in advance
>
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/Java-Script-Crawling-using-nutch-tp3804266p3804266.html
> Sent from the Nutch - User mailing list archive at Nabble.com.
>