Running A Paralle Job On HDFS

[Ra...@Dell.com]

Hi,
    I've MPI-BLAST application to run on HDFS and evaluate Parallel I/O. Can I submit the job by using "mpirun" command(MPICH1, which is installed on my system)....? Or do I need to convert this to MapReduce program....?
   Any reply is highly appreciated.

Thanks & Regards,
Ranga

Re: Running A Paralle Job On HDFS

[Brian Bockelman <bb...@cse.unl.edu>]

Hi Ranga,

HDFS is a high-throughput file system, designed to maximize the number of gigabytes of data served to clients, done reliably via commodity hardware.  In high-throughput computing, you often don't worry about maximizing the single-stream bandwidth (i.e., 10Gbps to a single client), but the aggregate throughput over a long time period (i.e., delivered 10 PB over a year).

It is a special-purpose file system, so you need to either use a special API to access it (native Java or libhdfs) or mount it through FUSE (which has its own pros and cons).

In our experience, MPI-BLAST is very sensitive to latency, so HDFS is likely not idea for your application.  However, don't take my word for it, feel free to explore for yourself.

Brian

On Nov 8, 2010, at 6:29 AM, <Ra...@Dell.com> wrote:

> Hi,
>    I've MPI-BLAST application to run on HDFS and evaluate Parallel I/O. Can I submit the job by using "mpirun" command(MPICH1, which is installed on my system)....? Or do I need to convert this to MapReduce program....?
>   Any reply is highly appreciated.
> 
> Thanks & Regards,
> Ranga
> 
>