You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Gaurav Bagga <gb...@gmail.com> on 2012/01/12 01:59:43 UTC
0 tasktrackers in jobtracker but all datanodes present
Hi
hadoop-0.19
I have a working hadoop cluster which has been running perfectly for months.
But today after restarting the cluster, at jobtracker UI its showing state
INITIALIZING for a long time and is staying on the same state.
The nodes in jobtracker are zero whereas all the nodes are present on the
dfs.
It says Safe mode is on.
grep'ed on slaves and I see the tasktrackers running.
In namenode logs i get the following error
2012-01-11 16:50:57,195 WARN ipc.Server - Out of Memory in server select
java.lang.OutOfMemoryError: Java heap space
at java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:39)
at java.nio.ByteBuffer.allocate(ByteBuffer.java:312)
at
org.apache.hadoop.ipc.Server$Connection.readAndProcess(Server.java:804)
at org.apache.hadoop.ipc.Server$Listener.doRead(Server.java:400)
at org.apache.hadoop.ipc.Server$Listener.run(Server.java:309)
Not sure why the cluster is not coming up
-G
Re: 0 tasktrackers in jobtracker but all datanodes present
Posted by Bejoy Ks <be...@gmail.com>.
Gaurav
NN memory might have hit its upper bound. As a bench mark, for every
1 million files/blocks/directories 1GB of memory is required on the NN. The
number of files in your cluster might have grown beyond this treshold. So
the options left for you would be
- If there are large number of small files, use HAR or Sequence File for
grouping the same
- Increase the NN heap
Regards
Bejoy KS
On Mon, Apr 2, 2012 at 12:08 PM, madhu phatak <ph...@gmail.com> wrote:
> Hi,
> 1. Stop the job tracker and task trackers. - bin/stop-mapred.sh
>
> 2. Disable namenode safemode - bin/hadoop dfsadmin -safemode leave
>
> 3. Start the job tracker and tasktrackers again - bin/start-mapred.sh
>
> On Fri, Jan 13, 2012 at 5:20 AM, Ravi Prakash <ra...@gmail.com>
> wrote:
>
> > Courtesy Kihwal and Bobby
> >
> > Have you tried increasing the max heap size with -Xmx? and make sure that
> > you have swap enabled.
> >
> > On Wed, Jan 11, 2012 at 6:59 PM, Gaurav Bagga <gb...@gmail.com>
> wrote:
> >
> > > Hi
> > >
> > > hadoop-0.19
> > > I have a working hadoop cluster which has been running perfectly for
> > > months.
> > > But today after restarting the cluster, at jobtracker UI its showing
> > state
> > > INITIALIZING for a long time and is staying on the same state.
> > > The nodes in jobtracker are zero whereas all the nodes are present on
> the
> > > dfs.
> > > It says Safe mode is on.
> > > grep'ed on slaves and I see the tasktrackers running.
> > >
> > > In namenode logs i get the following error
> > >
> > >
> > > 2012-01-11 16:50:57,195 WARN ipc.Server - Out of Memory in server
> select
> > > java.lang.OutOfMemoryError: Java heap space
> > > at java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:39)
> > > at java.nio.ByteBuffer.allocate(ByteBuffer.java:312)
> > > at
> > > org.apache.hadoop.ipc.Server$Connection.readAndProcess(Server.java:804)
> > > at org.apache.hadoop.ipc.Server$Listener.doRead(Server.java:400)
> > > at org.apache.hadoop.ipc.Server$Listener.run(Server.java:309)
> > >
> > > Not sure why the cluster is not coming up
> > > -G
> > >
> >
>
>
>
> --
> https://github.com/zinnia-phatak-dev/Nectar
>
Re: 0 tasktrackers in jobtracker but all datanodes present
Posted by madhu phatak <ph...@gmail.com>.
Hi,
1. Stop the job tracker and task trackers. - bin/stop-mapred.sh
2. Disable namenode safemode - bin/hadoop dfsadmin -safemode leave
3. Start the job tracker and tasktrackers again - bin/start-mapred.sh
On Fri, Jan 13, 2012 at 5:20 AM, Ravi Prakash <ra...@gmail.com> wrote:
> Courtesy Kihwal and Bobby
>
> Have you tried increasing the max heap size with -Xmx? and make sure that
> you have swap enabled.
>
> On Wed, Jan 11, 2012 at 6:59 PM, Gaurav Bagga <gb...@gmail.com> wrote:
>
> > Hi
> >
> > hadoop-0.19
> > I have a working hadoop cluster which has been running perfectly for
> > months.
> > But today after restarting the cluster, at jobtracker UI its showing
> state
> > INITIALIZING for a long time and is staying on the same state.
> > The nodes in jobtracker are zero whereas all the nodes are present on the
> > dfs.
> > It says Safe mode is on.
> > grep'ed on slaves and I see the tasktrackers running.
> >
> > In namenode logs i get the following error
> >
> >
> > 2012-01-11 16:50:57,195 WARN ipc.Server - Out of Memory in server select
> > java.lang.OutOfMemoryError: Java heap space
> > at java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:39)
> > at java.nio.ByteBuffer.allocate(ByteBuffer.java:312)
> > at
> > org.apache.hadoop.ipc.Server$Connection.readAndProcess(Server.java:804)
> > at org.apache.hadoop.ipc.Server$Listener.doRead(Server.java:400)
> > at org.apache.hadoop.ipc.Server$Listener.run(Server.java:309)
> >
> > Not sure why the cluster is not coming up
> > -G
> >
>
--
https://github.com/zinnia-phatak-dev/Nectar
Re: 0 tasktrackers in jobtracker but all datanodes present
Posted by Ravi Prakash <ra...@gmail.com>.
Courtesy Kihwal and Bobby
Have you tried increasing the max heap size with -Xmx? and make sure that
you have swap enabled.
On Wed, Jan 11, 2012 at 6:59 PM, Gaurav Bagga <gb...@gmail.com> wrote:
> Hi
>
> hadoop-0.19
> I have a working hadoop cluster which has been running perfectly for
> months.
> But today after restarting the cluster, at jobtracker UI its showing state
> INITIALIZING for a long time and is staying on the same state.
> The nodes in jobtracker are zero whereas all the nodes are present on the
> dfs.
> It says Safe mode is on.
> grep'ed on slaves and I see the tasktrackers running.
>
> In namenode logs i get the following error
>
>
> 2012-01-11 16:50:57,195 WARN ipc.Server - Out of Memory in server select
> java.lang.OutOfMemoryError: Java heap space
> at java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:39)
> at java.nio.ByteBuffer.allocate(ByteBuffer.java:312)
> at
> org.apache.hadoop.ipc.Server$Connection.readAndProcess(Server.java:804)
> at org.apache.hadoop.ipc.Server$Listener.doRead(Server.java:400)
> at org.apache.hadoop.ipc.Server$Listener.run(Server.java:309)
>
> Not sure why the cluster is not coming up
> -G
>