Running multiple processes on a single machine

[da...@gmail.com]

Hi,

If I want to run several parsers on a single quad-core machine  
simultaneously, would I still need to have Hadoop setup as a single-node  
cluster? Can several updatedbs be run simultaneously? I believe not, since  
the db seems to be locked when it's being updated.

Thanks.

Michael

Re: Running multiple processes on a single machine

[Michael Chan <da...@gmail.com>]

On Wed, Mar 11, 2009 at 4:41 PM, Sami Siren <ss...@gmail.com> wrote:

> dayzman@gmail.com wrote:
>
>> Hi,
>>
>> If I want to run several parsers on a single quad-core machine
>> simultaneously, would I still need to have Hadoop setup as a single-node
>> cluster?
>>
> I think that the fetcher is currently the only component that can take
> advantage of multiple cores when running in "local" mode. We should perhaps
> address that at some point since it is not that hard to parallelize at least
> some of the processing inside individual tools so single machine users could
> benefit from multiple cores.


But for parsing on a single machine with quad core, I can setup 2 map and 2
reduce jobs and get about 80-90% utilisation of 4 CPUs. When only the reduce
jobs are left, I get about 50% utilisation, as expected (2/4). Do I actually
not get a performance gain even though more CPUs are being utilised?

Michael


>
> I am not sure but I think that the only way to do it properly is run
> jobtracker and tasktracker on that machine and configure proper block sizes
> & number of map and reduce tasks.
>
>> Can several updatedbs be run simultaneously? I believe not, since the db
>> seems to be locked when it's being updated.
>>
> Locking prevents multiple applications of accessing crawl db simultaneously
> (also linkdb).
>
> --
> Sami Siren
>
>

Re: Running multiple processes on a single machine

[Sami Siren <ss...@gmail.com>]

dayzman@gmail.com wrote:
> Hi,
>
> If I want to run several parsers on a single quad-core machine 
> simultaneously, would I still need to have Hadoop setup as a 
> single-node cluster? 
I think that the fetcher is currently the only component that can take 
advantage of multiple cores when running in "local" mode. We should 
perhaps address that at some point since it is not that hard to 
parallelize at least some of the processing inside individual tools so 
single machine users could benefit from multiple cores.

I am not sure but I think that the only way to do it properly is run 
jobtracker and tasktracker on that machine and configure proper block 
sizes & number of map and reduce tasks.
> Can several updatedbs be run simultaneously? I believe not, since the 
> db seems to be locked when it's being updated.
Locking prevents multiple applications of accessing crawl db 
simultaneously (also linkdb).

--
 Sami Siren