You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Maximilian Schöfmann <sc...@googlemail.com> on 2007/03/19 11:10:56 UTC
Grep example won't terminate on "virtual" cluster (using xen)
Hi *,
I was playing around with the latest hadoop (0.12.1) on a virtual "cluster"
(4 nodes, one as namenode and jobtracker) using Xen. Each node runs the
current Xen kernel (2.6.16.33-xen) with 192 MB Ram and JDK 1.5.0_11.
Running the grep example in standalone mode works fine, but it won't
terminate in distributed mode:
~/hadoop$ ./bin/hadoop jar hadoop-0.12.1-examples.jar grep input output
'dfs[a-z.]+'
07/03/19 09:55:46 INFO mapred.InputFormatBase: Total input paths to process
: 3
07/03/19 09:55:47 INFO mapred.JobClient: Running job: job_0001
07/03/19 09:55:48 INFO mapred.JobClient: map 0% reduce 0%
07/03/19 09:56:02 INFO mapred.JobClient : map 3% reduce 0%
07/03/19 09:56:10 INFO mapred.JobClient: map 6% reduce 0%
07/03/19 09:56:11 INFO mapred.JobClient: map 9% reduce 0%
07/03/19 09:56:16 INFO mapred.JobClient: map 12% reduce 0%
(...)
07/03/19 09:58:03 INFO mapred.JobClient: map 90% reduce 5%
07/03/19 09:58:05 INFO mapred.JobClient: map 93% reduce 5%
07/03/19 09:58:06 INFO mapred.JobClient: map 96% reduce 5%
(...)
07/03/19 09:58:24 INFO mapred.JobClient: map 96% reduce 4%
07/03/19 09:58:26 INFO mapred.JobClient: map 96% reduce 5%
07/03/19 09:58:33 INFO mapred.JobClient: map 96% reduce 4%
07/03/19 09:58:36 INFO mapred.JobClient: map 96% reduce 5%
07/03/19 09:58:43 INFO mapred.JobClient : map 96% reduce 4%
07/03/19 09:58:46 INFO mapred.JobClient: map 96% reduce 5%
(... and goes on and on ...)
Is there something wrong with my setup?
Thanks,
Max
RE: Grep example won't terminate on "virtual" cluster (using xen)
Posted by Richard Yang <ri...@richardyang.net>.
Just to share experiences, I tried to use VMWare setting up a 2-node
cluster. The errors showing up were java.io.* related errors. There was
still a 20% chance finishing the job, such as sample Grep, randomwriter,
sort.
Best Regards
Richard Yang
richardyang@richardyang.net
kusanagiyang@gmail.com
-----Original Message-----
From: Maximilian Schöfmann [mailto:schoefmann@googlemail.com]
Sent: Wednesday, March 21, 2007 8:30 AM
To: hadoop-user@lucene.apache.org
Subject: Re: Grep example won't terminate on "virtual" cluster (using xen)
> > I was playing around with the latest hadoop (0.12.1) on a virtual
> > "cluster"
> > (4 nodes, one as namenode and jobtracker) using Xen. Each node runs the
> > current Xen kernel (2.6.16.33-xen) with 192 MB Ram and JDK 1.5.0_11.
> > Running the grep example in standalone mode works fine, but it won't
> > terminate in distributed mode:
> >
>
> Maybe 192MB of RAM for each nodes is not enough ?
Memory usage is well below 110 MB for each node. I've also tried using a
completely virtually switched network -- with the same results..
Re: Grep example won't terminate on "virtual" cluster (using xen)
Posted by Maximilian Schöfmann <sc...@googlemail.com>.
> > I was playing around with the latest hadoop (0.12.1) on a virtual
> > "cluster"
> > (4 nodes, one as namenode and jobtracker) using Xen. Each node runs the
> > current Xen kernel (2.6.16.33-xen) with 192 MB Ram and JDK 1.5.0_11.
> > Running the grep example in standalone mode works fine, but it won't
> > terminate in distributed mode:
> >
>
> Maybe 192MB of RAM for each nodes is not enough ?
Memory usage is well below 110 MB for each node. I've also tried using a
completely virtually switched network -- with the same results..
Re: Grep example won't terminate on "virtual" cluster (using xen)
Posted by Philippe Gassmann <ph...@anyware-tech.com>.
Maximilian Schöfmann a écrit :
> Hi *,
>
> I was playing around with the latest hadoop (0.12.1) on a virtual
> "cluster"
> (4 nodes, one as namenode and jobtracker) using Xen. Each node runs the
> current Xen kernel (2.6.16.33-xen) with 192 MB Ram and JDK 1.5.0_11.
> Running the grep example in standalone mode works fine, but it won't
> terminate in distributed mode:
>
Maybe 192MB of RAM for each nodes is not enough ?
> ~/hadoop$ ./bin/hadoop jar hadoop-0.12.1-examples.jar grep input output
> 'dfs[a-z.]+'
> 07/03/19 09:55:46 INFO mapred.InputFormatBase: Total input paths to
> process
> : 3
> 07/03/19 09:55:47 INFO mapred.JobClient: Running job: job_0001
> 07/03/19 09:55:48 INFO mapred.JobClient: map 0% reduce 0%
> 07/03/19 09:56:02 INFO mapred.JobClient : map 3% reduce 0%
> 07/03/19 09:56:10 INFO mapred.JobClient: map 6% reduce 0%
> 07/03/19 09:56:11 INFO mapred.JobClient: map 9% reduce 0%
> 07/03/19 09:56:16 INFO mapred.JobClient: map 12% reduce 0%
> (...)
> 07/03/19 09:58:03 INFO mapred.JobClient: map 90% reduce 5%
> 07/03/19 09:58:05 INFO mapred.JobClient: map 93% reduce 5%
> 07/03/19 09:58:06 INFO mapred.JobClient: map 96% reduce 5%
> (...)
> 07/03/19 09:58:24 INFO mapred.JobClient: map 96% reduce 4%
> 07/03/19 09:58:26 INFO mapred.JobClient: map 96% reduce 5%
> 07/03/19 09:58:33 INFO mapred.JobClient: map 96% reduce 4%
> 07/03/19 09:58:36 INFO mapred.JobClient: map 96% reduce 5%
> 07/03/19 09:58:43 INFO mapred.JobClient : map 96% reduce 4%
> 07/03/19 09:58:46 INFO mapred.JobClient: map 96% reduce 5%
> (... and goes on and on ...)
>
> Is there something wrong with my setup?
>
> Thanks,
> Max
>