Crawling e-commerce website

[jyoti aditya <jy...@gmail.com>]

Hi team,

Is it possible to crawl e-commerce website like amazon, flipkart,ebay using
nutch.
Can we crawl and extract product specific data?

And if yes, How much scalability we can achieve? Like, How much time nutch
will take to crawl  entire product under a specific category?


With Regards
Jyoti Aditya

Re: Crawling e-commerce website

[Tom Chiverton <tc...@extravision.com>]

Yes, it's possible. Nutch is a generic web crawler.

You will probably need to write and maintain custom extractors to parse 
the meta data out of each site. This may or may not be legal in your 
country.

Nutch will scale as far as you like. Time taken depends on many factors, 
like the size and shape of your servers and their cluster, as well as 
which backend you are storing too.

Tom


On 07/12/16 12:12, jyoti aditya wrote:
> Hi team,
>
> Is it possible to crawl e-commerce website like amazon, flipkart,ebay using
> nutch.
> Can we crawl and extract product specific data?
>
> And if yes, How much scalability we can achieve? Like, How much time nutch
> will take to crawl  entire product under a specific category?
>
>
> With Regards
> Jyoti Aditya
>
>
> ______________________________________________________________________
> This email has been scanned by the Symantec Email Security.cloud service.
> For more information please visit http://www.symanteccloud.com
> ______________________________________________________________________