You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by dito subandono <di...@gmail.com> on 2008/07/22 19:31:42 UTC

Smallest file size limit for hadoop to work faster than other application

i had a test with a log file analysis that was written with java and ran on
Hadoop.
i ran my log file analysis on a Intel Quad Core processor with 2 GB of
memory.
i set the map task to 40 and reduce task to 8.

the size of the log files i had test are 1GB to 4GB because i ran out of
storage resource.
i compare it with webalizer on a computer w/ Celeron processor and 256MB of
memory.

webalizer ran about 10x more faster.

what i did was just a small experiment and maybe still have to config things
more.
can anyone share the experience about the smallest file size limit for
hadoop to run faster than other application?

thanks..