Weird error starting up pseudo-dist cluster.

[Keith Wiley <kw...@keithwiley.com>]

Here's an error I've never seen before.  I rebooted my machine sometime last week, so obviously when I tried to run a hadoop job this morning, the first thing I was quickly reminded of was that the pseudo-distributed cluster wasn't running.  I started it up only to watch the job tracker appear in the browser briefly and then go away (typical error complaining that the port was closed, as if the jobtracker is gone).  The namenode, interestingly, never came up during this time.  I tried stopping and starting "all" a few times but to no avail.

I inspected the logs and saw this:

java.io.IOException: Missing directory /tmp/hadoop-keithw/dfs/name

Sure enough, it isn't there.  I'm not familiar with this directory, so I can't say whether it was ever there before, but presumably it was.

Now, I assume I could get around this by formatting a new namenode, but then I would have to copy my data back into HDFS from scratch.

So, two questions:

(1) Any idea what the heck is going on here, how this happened, what it means?

(2) Is there any way to recover without starting over from scratch?

Thanks.

________________________________________________________________________________
Keith Wiley     kwiley@keithwiley.com     keithwiley.com    music.keithwiley.com

"And what if we picked the wrong religion?  Every week, we're just making God
madder and madder!"
                                           --  Homer Simpson
________________________________________________________________________________

Re: Weird error starting up pseudo-dist cluster.

[Keith Wiley <kw...@keithwiley.com>]

On Apr 30, 2012, at 11:10 , Andrzej Bialecki wrote:

> On 30/04/2012 19:48, Keith Wiley wrote:
>> 
>> (1) Any idea what the heck is going on here, how this happened, what it means?
> 
> The default hdfs config puts the namenode data in /tmp. This may be ok for casual testing, but in all other situations it's the worst location imaginable - for example, linux cleans this directory on reboot, and I think that's what happened here. Your HDFS data is gone to a better world...
> 
>> 
>> (2) Is there any way to recover without starting over from scratch?
> 
> Regretfully, no. The lesson is: don't put precious files in /tmp.


Ah, okay, so, when setting up a single-machine, just a pseudo-dist cluster, what is a better way to do it?  Where would one put the temp directories in order to gain improved robustness of the hadoop system?  Is this the sort of thing to put in a home directory?  I never really conceptualized it that way; I always thought HDFS and hadoop in general were sort of system-level concepts.  This is a single-user machine, I have full root/admin control over it, so it's not a permissions issue, I'm just asking at a philosophical level how to set up a pseudo-dist cluster in the most effective way?

Thanks.


________________________________________________________________________________
Keith Wiley     kwiley@keithwiley.com     keithwiley.com    music.keithwiley.com

"I used to be with it, but then they changed what it was.  Now, what I'm with
isn't it, and what's it seems weird and scary to me."
                                           --  Abe (Grandpa) Simpson
________________________________________________________________________________

Re: Weird error starting up pseudo-dist cluster.

[Andrzej Bialecki <ab...@getopt.org>]

On 30/04/2012 19:48, Keith Wiley wrote:
> Here's an error I've never seen before.  I rebooted my machine sometime last week, so obviously when I tried to run a hadoop job this morning, the first thing I was quickly reminded of was that the pseudo-distributed cluster wasn't running.  I started it up only to watch the job tracker appear in the browser briefly and then go away (typical error complaining that the port was closed, as if the jobtracker is gone).  The namenode, interestingly, never came up during this time.  I tried stopping and starting "all" a few times but to no avail.
>
> I inspected the logs and saw this:
>
> java.io.IOException: Missing directory /tmp/hadoop-keithw/dfs/name
>
> Sure enough, it isn't there.  I'm not familiar with this directory, so I can't say whether it was ever there before, but presumably it was.
>
> Now, I assume I could get around this by formatting a new namenode, but then I would have to copy my data back into HDFS from scratch.
>
> So, two questions:
>
> (1) Any idea what the heck is going on here, how this happened, what it means?

The default hdfs config puts the namenode data in /tmp. This may be ok 
for casual testing, but in all other situations it's the worst location 
imaginable - for example, linux cleans this directory on reboot, and I 
think that's what happened here. Your HDFS data is gone to a better world...


>
> (2) Is there any way to recover without starting over from scratch?

Regretfully, no. The lesson is: don't put precious files in /tmp.


-- 
Best regards,
Andrzej Bialecki     <><
  ___. ___ ___ ___ _ _   __________________________________
[__ || __|__/|__||\/|  Information Retrieval, Semantic Web
___|||__||  \|  ||  |  Embedded Unix, System Integration
http://www.sigram.com  Contact: info at sigram dot com