You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-user@hadoop.apache.org by Jean-Marc Spaggiari <je...@spaggiari.org> on 2012/12/18 21:28:59 UTC
Misconfiguration of hdfs-site.xml
Hi,
For months now I'm using my hadoop cluster with absolutly nothing
related to the drive directory on my hdfs-site.xml file.
It seems that it's using the hadoop.tmp.dir directory to store data.
My hadoop.tmp.dir is pointing to
/home/hadoop/haddop_drive/${user.name} and on my
/home/hadoop/haddop_drive directory I can see hadoop and hbase.
Now, I want to configure that properly without loosing the data I have.
I'm using 1.0.3.
Based on the documentation
http://hadoop.apache.org/docs/r1.0.3/cluster_setup.html#Site+Configuration
it seems I need to setup dfs.data.dir to point to
/home/hadoop/haddop_drive. If I add this entry and restart my
datanode, will it "simply" continue to point to the same place and
find the data he needs?
Also, what about dfs.name.dir ? I guess I can simply point it to the
same place, right?
On /home/hadoop/haddop_drive/hadoop/dfs I have data and name
directories. So I'm not sure if I should point the properties above to
/home/hadoop/haddop_drive or to /home/hadoop/haddop_drive/dfs.
Thanks,
JM
Re: Misconfiguration of hdfs-site.xml
Posted by Bryan Beaudreault <bb...@hubspot.com>.
Updating the config and restarting will result in data loss. You're going
to want to migrate the data from your tmp directory to the appropriate
place.
The NameNode will use the dfs.name.dir property, and DataNodes will use the
dfs.data.dir. Assuming you aren't running a DN and NN on the same server
(don't do that), then they shouldn't collide/overlap. Even still its
probably best to use different directories. For instance mine is like this:
dfs.name.dir = /mnt/hdfs/name,/mnt2/hdfs/name
dfs.data.dir
= /mnt/hdfs/data,/mnt2/hdfs/data,/mnt3/hdfs/data,/mnt4/hdfs/data
Hope this helps.
On Tue, Dec 18, 2012 at 3:28 PM, Jean-Marc Spaggiari <
jean-marc@spaggiari.org> wrote:
> Hi,
>
> For months now I'm using my hadoop cluster with absolutly nothing
> related to the drive directory on my hdfs-site.xml file.
>
> It seems that it's using the hadoop.tmp.dir directory to store data.
>
> My hadoop.tmp.dir is pointing to
> /home/hadoop/haddop_drive/${user.name} and on my
> /home/hadoop/haddop_drive directory I can see hadoop and hbase.
>
> Now, I want to configure that properly without loosing the data I have.
>
> I'm using 1.0.3.
>
> Based on the documentation
> http://hadoop.apache.org/docs/r1.0.3/cluster_setup.html#Site+Configuration
> it seems I need to setup dfs.data.dir to point to
> /home/hadoop/haddop_drive. If I add this entry and restart my
> datanode, will it "simply" continue to point to the same place and
> find the data he needs?
>
> Also, what about dfs.name.dir ? I guess I can simply point it to the
> same place, right?
>
> On /home/hadoop/haddop_drive/hadoop/dfs I have data and name
> directories. So I'm not sure if I should point the properties above to
> /home/hadoop/haddop_drive or to /home/hadoop/haddop_drive/dfs.
>
> Thanks,
>
> JM
>
Re: Misconfiguration of hdfs-site.xml
Posted by Bryan Beaudreault <bb...@hubspot.com>.
Updating the config and restarting will result in data loss. You're going
to want to migrate the data from your tmp directory to the appropriate
place.
The NameNode will use the dfs.name.dir property, and DataNodes will use the
dfs.data.dir. Assuming you aren't running a DN and NN on the same server
(don't do that), then they shouldn't collide/overlap. Even still its
probably best to use different directories. For instance mine is like this:
dfs.name.dir = /mnt/hdfs/name,/mnt2/hdfs/name
dfs.data.dir
= /mnt/hdfs/data,/mnt2/hdfs/data,/mnt3/hdfs/data,/mnt4/hdfs/data
Hope this helps.
On Tue, Dec 18, 2012 at 3:28 PM, Jean-Marc Spaggiari <
jean-marc@spaggiari.org> wrote:
> Hi,
>
> For months now I'm using my hadoop cluster with absolutly nothing
> related to the drive directory on my hdfs-site.xml file.
>
> It seems that it's using the hadoop.tmp.dir directory to store data.
>
> My hadoop.tmp.dir is pointing to
> /home/hadoop/haddop_drive/${user.name} and on my
> /home/hadoop/haddop_drive directory I can see hadoop and hbase.
>
> Now, I want to configure that properly without loosing the data I have.
>
> I'm using 1.0.3.
>
> Based on the documentation
> http://hadoop.apache.org/docs/r1.0.3/cluster_setup.html#Site+Configuration
> it seems I need to setup dfs.data.dir to point to
> /home/hadoop/haddop_drive. If I add this entry and restart my
> datanode, will it "simply" continue to point to the same place and
> find the data he needs?
>
> Also, what about dfs.name.dir ? I guess I can simply point it to the
> same place, right?
>
> On /home/hadoop/haddop_drive/hadoop/dfs I have data and name
> directories. So I'm not sure if I should point the properties above to
> /home/hadoop/haddop_drive or to /home/hadoop/haddop_drive/dfs.
>
> Thanks,
>
> JM
>
Re: Misconfiguration of hdfs-site.xml
Posted by Bryan Beaudreault <bb...@hubspot.com>.
Updating the config and restarting will result in data loss. You're going
to want to migrate the data from your tmp directory to the appropriate
place.
The NameNode will use the dfs.name.dir property, and DataNodes will use the
dfs.data.dir. Assuming you aren't running a DN and NN on the same server
(don't do that), then they shouldn't collide/overlap. Even still its
probably best to use different directories. For instance mine is like this:
dfs.name.dir = /mnt/hdfs/name,/mnt2/hdfs/name
dfs.data.dir
= /mnt/hdfs/data,/mnt2/hdfs/data,/mnt3/hdfs/data,/mnt4/hdfs/data
Hope this helps.
On Tue, Dec 18, 2012 at 3:28 PM, Jean-Marc Spaggiari <
jean-marc@spaggiari.org> wrote:
> Hi,
>
> For months now I'm using my hadoop cluster with absolutly nothing
> related to the drive directory on my hdfs-site.xml file.
>
> It seems that it's using the hadoop.tmp.dir directory to store data.
>
> My hadoop.tmp.dir is pointing to
> /home/hadoop/haddop_drive/${user.name} and on my
> /home/hadoop/haddop_drive directory I can see hadoop and hbase.
>
> Now, I want to configure that properly without loosing the data I have.
>
> I'm using 1.0.3.
>
> Based on the documentation
> http://hadoop.apache.org/docs/r1.0.3/cluster_setup.html#Site+Configuration
> it seems I need to setup dfs.data.dir to point to
> /home/hadoop/haddop_drive. If I add this entry and restart my
> datanode, will it "simply" continue to point to the same place and
> find the data he needs?
>
> Also, what about dfs.name.dir ? I guess I can simply point it to the
> same place, right?
>
> On /home/hadoop/haddop_drive/hadoop/dfs I have data and name
> directories. So I'm not sure if I should point the properties above to
> /home/hadoop/haddop_drive or to /home/hadoop/haddop_drive/dfs.
>
> Thanks,
>
> JM
>
Re: Misconfiguration of hdfs-site.xml
Posted by Bryan Beaudreault <bb...@hubspot.com>.
Updating the config and restarting will result in data loss. You're going
to want to migrate the data from your tmp directory to the appropriate
place.
The NameNode will use the dfs.name.dir property, and DataNodes will use the
dfs.data.dir. Assuming you aren't running a DN and NN on the same server
(don't do that), then they shouldn't collide/overlap. Even still its
probably best to use different directories. For instance mine is like this:
dfs.name.dir = /mnt/hdfs/name,/mnt2/hdfs/name
dfs.data.dir
= /mnt/hdfs/data,/mnt2/hdfs/data,/mnt3/hdfs/data,/mnt4/hdfs/data
Hope this helps.
On Tue, Dec 18, 2012 at 3:28 PM, Jean-Marc Spaggiari <
jean-marc@spaggiari.org> wrote:
> Hi,
>
> For months now I'm using my hadoop cluster with absolutly nothing
> related to the drive directory on my hdfs-site.xml file.
>
> It seems that it's using the hadoop.tmp.dir directory to store data.
>
> My hadoop.tmp.dir is pointing to
> /home/hadoop/haddop_drive/${user.name} and on my
> /home/hadoop/haddop_drive directory I can see hadoop and hbase.
>
> Now, I want to configure that properly without loosing the data I have.
>
> I'm using 1.0.3.
>
> Based on the documentation
> http://hadoop.apache.org/docs/r1.0.3/cluster_setup.html#Site+Configuration
> it seems I need to setup dfs.data.dir to point to
> /home/hadoop/haddop_drive. If I add this entry and restart my
> datanode, will it "simply" continue to point to the same place and
> find the data he needs?
>
> Also, what about dfs.name.dir ? I guess I can simply point it to the
> same place, right?
>
> On /home/hadoop/haddop_drive/hadoop/dfs I have data and name
> directories. So I'm not sure if I should point the properties above to
> /home/hadoop/haddop_drive or to /home/hadoop/haddop_drive/dfs.
>
> Thanks,
>
> JM
>
Re: Misconfiguration of hdfs-site.xml
Posted by Jean-Marc Spaggiari <je...@spaggiari.org>.
> Updating the config and restarting will result in data loss.
Ok. If that's the case, then I can still re-configure the servers one
by one manually and restart them. And moving to the next server only
when the replication is completed.
> dfs.data.dir presently is /home/hadoop/haddop_drive/${user.name}/dfs/data
> dfs.name.dir presently is /home/hadoop/haddop_drive/${user.name}/dfs/name
Ok. That sound like what I was expecting. I don't really want to move
the data/name directories. My goal is just to fix the lack of
configuration.
I will try to update with the properties above on one of the servers
and restart it. And if it's working fine, I will apply that on all the
others.
Thanks,
JM
2012/12/18, Harsh J <ha...@cloudera.com>:
> Given your previous config is
> "/home/hadoop/haddop_drive/${user.name}", if the defaults are to go
> by, then:
>
> dfs.data.dir presently is /home/hadoop/haddop_drive/${user.name}/dfs/data
> dfs.name.dir presently is /home/hadoop/haddop_drive/${user.name}/dfs/name
>
> If you are specifying explicit paths, it will need to be the full one
> (preferably pre-evaluated for ${user.name}) as above, exactly.
>
> If you wish to move the location somewhere else, you will need to mv
> the {data,name} directories elsewhere and re-point down to that path
> component again.
>
> On Wed, Dec 19, 2012 at 1:58 AM, Jean-Marc Spaggiari
> <je...@spaggiari.org> wrote:
>> Hi,
>>
>> For months now I'm using my hadoop cluster with absolutly nothing
>> related to the drive directory on my hdfs-site.xml file.
>>
>> It seems that it's using the hadoop.tmp.dir directory to store data.
>>
>> My hadoop.tmp.dir is pointing to
>> /home/hadoop/haddop_drive/${user.name} and on my
>> /home/hadoop/haddop_drive directory I can see hadoop and hbase.
>>
>> Now, I want to configure that properly without loosing the data I have.
>>
>> I'm using 1.0.3.
>>
>> Based on the documentation
>> http://hadoop.apache.org/docs/r1.0.3/cluster_setup.html#Site+Configuration
>> it seems I need to setup dfs.data.dir to point to
>> /home/hadoop/haddop_drive. If I add this entry and restart my
>> datanode, will it "simply" continue to point to the same place and
>> find the data he needs?
>>
>> Also, what about dfs.name.dir ? I guess I can simply point it to the
>> same place, right?
>>
>> On /home/hadoop/haddop_drive/hadoop/dfs I have data and name
>> directories. So I'm not sure if I should point the properties above to
>> /home/hadoop/haddop_drive or to /home/hadoop/haddop_drive/dfs.
>>
>> Thanks,
>>
>> JM
>
>
>
> --
> Harsh J
>
Re: Misconfiguration of hdfs-site.xml
Posted by Jean-Marc Spaggiari <je...@spaggiari.org>.
> Updating the config and restarting will result in data loss.
Ok. If that's the case, then I can still re-configure the servers one
by one manually and restart them. And moving to the next server only
when the replication is completed.
> dfs.data.dir presently is /home/hadoop/haddop_drive/${user.name}/dfs/data
> dfs.name.dir presently is /home/hadoop/haddop_drive/${user.name}/dfs/name
Ok. That sound like what I was expecting. I don't really want to move
the data/name directories. My goal is just to fix the lack of
configuration.
I will try to update with the properties above on one of the servers
and restart it. And if it's working fine, I will apply that on all the
others.
Thanks,
JM
2012/12/18, Harsh J <ha...@cloudera.com>:
> Given your previous config is
> "/home/hadoop/haddop_drive/${user.name}", if the defaults are to go
> by, then:
>
> dfs.data.dir presently is /home/hadoop/haddop_drive/${user.name}/dfs/data
> dfs.name.dir presently is /home/hadoop/haddop_drive/${user.name}/dfs/name
>
> If you are specifying explicit paths, it will need to be the full one
> (preferably pre-evaluated for ${user.name}) as above, exactly.
>
> If you wish to move the location somewhere else, you will need to mv
> the {data,name} directories elsewhere and re-point down to that path
> component again.
>
> On Wed, Dec 19, 2012 at 1:58 AM, Jean-Marc Spaggiari
> <je...@spaggiari.org> wrote:
>> Hi,
>>
>> For months now I'm using my hadoop cluster with absolutly nothing
>> related to the drive directory on my hdfs-site.xml file.
>>
>> It seems that it's using the hadoop.tmp.dir directory to store data.
>>
>> My hadoop.tmp.dir is pointing to
>> /home/hadoop/haddop_drive/${user.name} and on my
>> /home/hadoop/haddop_drive directory I can see hadoop and hbase.
>>
>> Now, I want to configure that properly without loosing the data I have.
>>
>> I'm using 1.0.3.
>>
>> Based on the documentation
>> http://hadoop.apache.org/docs/r1.0.3/cluster_setup.html#Site+Configuration
>> it seems I need to setup dfs.data.dir to point to
>> /home/hadoop/haddop_drive. If I add this entry and restart my
>> datanode, will it "simply" continue to point to the same place and
>> find the data he needs?
>>
>> Also, what about dfs.name.dir ? I guess I can simply point it to the
>> same place, right?
>>
>> On /home/hadoop/haddop_drive/hadoop/dfs I have data and name
>> directories. So I'm not sure if I should point the properties above to
>> /home/hadoop/haddop_drive or to /home/hadoop/haddop_drive/dfs.
>>
>> Thanks,
>>
>> JM
>
>
>
> --
> Harsh J
>
Re: Misconfiguration of hdfs-site.xml
Posted by Jean-Marc Spaggiari <je...@spaggiari.org>.
> Updating the config and restarting will result in data loss.
Ok. If that's the case, then I can still re-configure the servers one
by one manually and restart them. And moving to the next server only
when the replication is completed.
> dfs.data.dir presently is /home/hadoop/haddop_drive/${user.name}/dfs/data
> dfs.name.dir presently is /home/hadoop/haddop_drive/${user.name}/dfs/name
Ok. That sound like what I was expecting. I don't really want to move
the data/name directories. My goal is just to fix the lack of
configuration.
I will try to update with the properties above on one of the servers
and restart it. And if it's working fine, I will apply that on all the
others.
Thanks,
JM
2012/12/18, Harsh J <ha...@cloudera.com>:
> Given your previous config is
> "/home/hadoop/haddop_drive/${user.name}", if the defaults are to go
> by, then:
>
> dfs.data.dir presently is /home/hadoop/haddop_drive/${user.name}/dfs/data
> dfs.name.dir presently is /home/hadoop/haddop_drive/${user.name}/dfs/name
>
> If you are specifying explicit paths, it will need to be the full one
> (preferably pre-evaluated for ${user.name}) as above, exactly.
>
> If you wish to move the location somewhere else, you will need to mv
> the {data,name} directories elsewhere and re-point down to that path
> component again.
>
> On Wed, Dec 19, 2012 at 1:58 AM, Jean-Marc Spaggiari
> <je...@spaggiari.org> wrote:
>> Hi,
>>
>> For months now I'm using my hadoop cluster with absolutly nothing
>> related to the drive directory on my hdfs-site.xml file.
>>
>> It seems that it's using the hadoop.tmp.dir directory to store data.
>>
>> My hadoop.tmp.dir is pointing to
>> /home/hadoop/haddop_drive/${user.name} and on my
>> /home/hadoop/haddop_drive directory I can see hadoop and hbase.
>>
>> Now, I want to configure that properly without loosing the data I have.
>>
>> I'm using 1.0.3.
>>
>> Based on the documentation
>> http://hadoop.apache.org/docs/r1.0.3/cluster_setup.html#Site+Configuration
>> it seems I need to setup dfs.data.dir to point to
>> /home/hadoop/haddop_drive. If I add this entry and restart my
>> datanode, will it "simply" continue to point to the same place and
>> find the data he needs?
>>
>> Also, what about dfs.name.dir ? I guess I can simply point it to the
>> same place, right?
>>
>> On /home/hadoop/haddop_drive/hadoop/dfs I have data and name
>> directories. So I'm not sure if I should point the properties above to
>> /home/hadoop/haddop_drive or to /home/hadoop/haddop_drive/dfs.
>>
>> Thanks,
>>
>> JM
>
>
>
> --
> Harsh J
>
Re: Misconfiguration of hdfs-site.xml
Posted by Jean-Marc Spaggiari <je...@spaggiari.org>.
> Updating the config and restarting will result in data loss.
Ok. If that's the case, then I can still re-configure the servers one
by one manually and restart them. And moving to the next server only
when the replication is completed.
> dfs.data.dir presently is /home/hadoop/haddop_drive/${user.name}/dfs/data
> dfs.name.dir presently is /home/hadoop/haddop_drive/${user.name}/dfs/name
Ok. That sound like what I was expecting. I don't really want to move
the data/name directories. My goal is just to fix the lack of
configuration.
I will try to update with the properties above on one of the servers
and restart it. And if it's working fine, I will apply that on all the
others.
Thanks,
JM
2012/12/18, Harsh J <ha...@cloudera.com>:
> Given your previous config is
> "/home/hadoop/haddop_drive/${user.name}", if the defaults are to go
> by, then:
>
> dfs.data.dir presently is /home/hadoop/haddop_drive/${user.name}/dfs/data
> dfs.name.dir presently is /home/hadoop/haddop_drive/${user.name}/dfs/name
>
> If you are specifying explicit paths, it will need to be the full one
> (preferably pre-evaluated for ${user.name}) as above, exactly.
>
> If you wish to move the location somewhere else, you will need to mv
> the {data,name} directories elsewhere and re-point down to that path
> component again.
>
> On Wed, Dec 19, 2012 at 1:58 AM, Jean-Marc Spaggiari
> <je...@spaggiari.org> wrote:
>> Hi,
>>
>> For months now I'm using my hadoop cluster with absolutly nothing
>> related to the drive directory on my hdfs-site.xml file.
>>
>> It seems that it's using the hadoop.tmp.dir directory to store data.
>>
>> My hadoop.tmp.dir is pointing to
>> /home/hadoop/haddop_drive/${user.name} and on my
>> /home/hadoop/haddop_drive directory I can see hadoop and hbase.
>>
>> Now, I want to configure that properly without loosing the data I have.
>>
>> I'm using 1.0.3.
>>
>> Based on the documentation
>> http://hadoop.apache.org/docs/r1.0.3/cluster_setup.html#Site+Configuration
>> it seems I need to setup dfs.data.dir to point to
>> /home/hadoop/haddop_drive. If I add this entry and restart my
>> datanode, will it "simply" continue to point to the same place and
>> find the data he needs?
>>
>> Also, what about dfs.name.dir ? I guess I can simply point it to the
>> same place, right?
>>
>> On /home/hadoop/haddop_drive/hadoop/dfs I have data and name
>> directories. So I'm not sure if I should point the properties above to
>> /home/hadoop/haddop_drive or to /home/hadoop/haddop_drive/dfs.
>>
>> Thanks,
>>
>> JM
>
>
>
> --
> Harsh J
>
Re: Misconfiguration of hdfs-site.xml
Posted by Harsh J <ha...@cloudera.com>.
Given your previous config is
"/home/hadoop/haddop_drive/${user.name}", if the defaults are to go
by, then:
dfs.data.dir presently is /home/hadoop/haddop_drive/${user.name}/dfs/data
dfs.name.dir presently is /home/hadoop/haddop_drive/${user.name}/dfs/name
If you are specifying explicit paths, it will need to be the full one
(preferably pre-evaluated for ${user.name}) as above, exactly.
If you wish to move the location somewhere else, you will need to mv
the {data,name} directories elsewhere and re-point down to that path
component again.
On Wed, Dec 19, 2012 at 1:58 AM, Jean-Marc Spaggiari
<je...@spaggiari.org> wrote:
> Hi,
>
> For months now I'm using my hadoop cluster with absolutly nothing
> related to the drive directory on my hdfs-site.xml file.
>
> It seems that it's using the hadoop.tmp.dir directory to store data.
>
> My hadoop.tmp.dir is pointing to
> /home/hadoop/haddop_drive/${user.name} and on my
> /home/hadoop/haddop_drive directory I can see hadoop and hbase.
>
> Now, I want to configure that properly without loosing the data I have.
>
> I'm using 1.0.3.
>
> Based on the documentation
> http://hadoop.apache.org/docs/r1.0.3/cluster_setup.html#Site+Configuration
> it seems I need to setup dfs.data.dir to point to
> /home/hadoop/haddop_drive. If I add this entry and restart my
> datanode, will it "simply" continue to point to the same place and
> find the data he needs?
>
> Also, what about dfs.name.dir ? I guess I can simply point it to the
> same place, right?
>
> On /home/hadoop/haddop_drive/hadoop/dfs I have data and name
> directories. So I'm not sure if I should point the properties above to
> /home/hadoop/haddop_drive or to /home/hadoop/haddop_drive/dfs.
>
> Thanks,
>
> JM
--
Harsh J
Re: Misconfiguration of hdfs-site.xml
Posted by Harsh J <ha...@cloudera.com>.
Given your previous config is
"/home/hadoop/haddop_drive/${user.name}", if the defaults are to go
by, then:
dfs.data.dir presently is /home/hadoop/haddop_drive/${user.name}/dfs/data
dfs.name.dir presently is /home/hadoop/haddop_drive/${user.name}/dfs/name
If you are specifying explicit paths, it will need to be the full one
(preferably pre-evaluated for ${user.name}) as above, exactly.
If you wish to move the location somewhere else, you will need to mv
the {data,name} directories elsewhere and re-point down to that path
component again.
On Wed, Dec 19, 2012 at 1:58 AM, Jean-Marc Spaggiari
<je...@spaggiari.org> wrote:
> Hi,
>
> For months now I'm using my hadoop cluster with absolutly nothing
> related to the drive directory on my hdfs-site.xml file.
>
> It seems that it's using the hadoop.tmp.dir directory to store data.
>
> My hadoop.tmp.dir is pointing to
> /home/hadoop/haddop_drive/${user.name} and on my
> /home/hadoop/haddop_drive directory I can see hadoop and hbase.
>
> Now, I want to configure that properly without loosing the data I have.
>
> I'm using 1.0.3.
>
> Based on the documentation
> http://hadoop.apache.org/docs/r1.0.3/cluster_setup.html#Site+Configuration
> it seems I need to setup dfs.data.dir to point to
> /home/hadoop/haddop_drive. If I add this entry and restart my
> datanode, will it "simply" continue to point to the same place and
> find the data he needs?
>
> Also, what about dfs.name.dir ? I guess I can simply point it to the
> same place, right?
>
> On /home/hadoop/haddop_drive/hadoop/dfs I have data and name
> directories. So I'm not sure if I should point the properties above to
> /home/hadoop/haddop_drive or to /home/hadoop/haddop_drive/dfs.
>
> Thanks,
>
> JM
--
Harsh J
Re: Misconfiguration of hdfs-site.xml
Posted by Harsh J <ha...@cloudera.com>.
Given your previous config is
"/home/hadoop/haddop_drive/${user.name}", if the defaults are to go
by, then:
dfs.data.dir presently is /home/hadoop/haddop_drive/${user.name}/dfs/data
dfs.name.dir presently is /home/hadoop/haddop_drive/${user.name}/dfs/name
If you are specifying explicit paths, it will need to be the full one
(preferably pre-evaluated for ${user.name}) as above, exactly.
If you wish to move the location somewhere else, you will need to mv
the {data,name} directories elsewhere and re-point down to that path
component again.
On Wed, Dec 19, 2012 at 1:58 AM, Jean-Marc Spaggiari
<je...@spaggiari.org> wrote:
> Hi,
>
> For months now I'm using my hadoop cluster with absolutly nothing
> related to the drive directory on my hdfs-site.xml file.
>
> It seems that it's using the hadoop.tmp.dir directory to store data.
>
> My hadoop.tmp.dir is pointing to
> /home/hadoop/haddop_drive/${user.name} and on my
> /home/hadoop/haddop_drive directory I can see hadoop and hbase.
>
> Now, I want to configure that properly without loosing the data I have.
>
> I'm using 1.0.3.
>
> Based on the documentation
> http://hadoop.apache.org/docs/r1.0.3/cluster_setup.html#Site+Configuration
> it seems I need to setup dfs.data.dir to point to
> /home/hadoop/haddop_drive. If I add this entry and restart my
> datanode, will it "simply" continue to point to the same place and
> find the data he needs?
>
> Also, what about dfs.name.dir ? I guess I can simply point it to the
> same place, right?
>
> On /home/hadoop/haddop_drive/hadoop/dfs I have data and name
> directories. So I'm not sure if I should point the properties above to
> /home/hadoop/haddop_drive or to /home/hadoop/haddop_drive/dfs.
>
> Thanks,
>
> JM
--
Harsh J
Re: Misconfiguration of hdfs-site.xml
Posted by Harsh J <ha...@cloudera.com>.
Given your previous config is
"/home/hadoop/haddop_drive/${user.name}", if the defaults are to go
by, then:
dfs.data.dir presently is /home/hadoop/haddop_drive/${user.name}/dfs/data
dfs.name.dir presently is /home/hadoop/haddop_drive/${user.name}/dfs/name
If you are specifying explicit paths, it will need to be the full one
(preferably pre-evaluated for ${user.name}) as above, exactly.
If you wish to move the location somewhere else, you will need to mv
the {data,name} directories elsewhere and re-point down to that path
component again.
On Wed, Dec 19, 2012 at 1:58 AM, Jean-Marc Spaggiari
<je...@spaggiari.org> wrote:
> Hi,
>
> For months now I'm using my hadoop cluster with absolutly nothing
> related to the drive directory on my hdfs-site.xml file.
>
> It seems that it's using the hadoop.tmp.dir directory to store data.
>
> My hadoop.tmp.dir is pointing to
> /home/hadoop/haddop_drive/${user.name} and on my
> /home/hadoop/haddop_drive directory I can see hadoop and hbase.
>
> Now, I want to configure that properly without loosing the data I have.
>
> I'm using 1.0.3.
>
> Based on the documentation
> http://hadoop.apache.org/docs/r1.0.3/cluster_setup.html#Site+Configuration
> it seems I need to setup dfs.data.dir to point to
> /home/hadoop/haddop_drive. If I add this entry and restart my
> datanode, will it "simply" continue to point to the same place and
> find the data he needs?
>
> Also, what about dfs.name.dir ? I guess I can simply point it to the
> same place, right?
>
> On /home/hadoop/haddop_drive/hadoop/dfs I have data and name
> directories. So I'm not sure if I should point the properties above to
> /home/hadoop/haddop_drive or to /home/hadoop/haddop_drive/dfs.
>
> Thanks,
>
> JM
--
Harsh J