You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@nutch.apache.org by Nguyen Manh Tien <ti...@gmail.com> on 2015/12/04 08:18:24 UTC

Chosing AWS instance for Nutch 1.X

Hello,

I'm setting the AWS cluster for Nutch 1.10 to crawl about 100M+ pages from
www.

Could some one please advice about choosing aws instance, storage:
- We don't use EMR
- Which aws instance type is best for us?
- Should we use EBS for storage?
- Should we use a dedicated DNS or install local DNS on each crawl machines
is good enough?
- Which is best for Nutch 1.10: hadoop 1.x or hadoop 2.x?

Thanks
TIen