You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-user@hadoop.apache.org by Charles Robertson <ch...@gmail.com> on 2014/09/16 14:28:09 UTC
DataNode not recognising available disk space
Hi all,
I've added a new slave node to my cluster with a (single) larger disk size
(100Gb) than on the other nodes. However, Amabri is reporting a total of
8.6 Gb disk space. lsblk correctly reports the disk size.
Does anyone why this might be? As I understand things you need to tell HDFS
how much space *not* to use, and it will use what it needs from the rest.
(BTW this is not the same data node in my other question on this list,
"Cannot start DataNode after adding new volume.")
Thanks,
Charles
Re: DataNode not recognising available disk space
Posted by Charles Robertson <ch...@gmail.com>.
Hi Yusaku,
Thank you for your reply, unfortunately that's not the problem - the new
slave node has a single drive, so is using the default data directory path.
I'll post this to the Ambari list.
Regards,
Charles
On 16 September 2014 23:12, Yusaku Sako <yu...@hortonworks.com> wrote:
> Charles,
>
> If the newly added slave node has an extra disk that the other nodes
> don't have, then you will have to do the following on the Ambari Web
> UI so that "dfs.data.dir" is reflected to include this extra drive for
> that node:
> * Go to Services > HDFS > Configs
> * Click on "Manage Config Groups" link
> * Create a new HDFS Configuration Group (name it appropriately), add
> the slave host in question to that group, and Save.
> * In the list of configs for HDFS, you will see "DataNode directories"
> with the current directories set up for DataNodes for all other hosts.
> * Click on the "+" next to "DataNode directories" and list out all the
> drives for this new DataNode.
> * Save the configuration changes.
> * Restart the DataNode.
> * Ambari (and HDFS) should start reflecting the added disk space.
>
> I hope this solves your problem.
>
> FYI, there's an Ambari-specific mailing list at user@ambari.apache.org
> that you can subscribe by sending an email to
> user-subscribe@ambari.apache.org.
>
> Thanks,
> Yusaku
>
>
> On Tue, Sep 16, 2014 at 5:28 AM, Charles Robertson
> <ch...@gmail.com> wrote:
> > Hi all,
> >
> > I've added a new slave node to my cluster with a (single) larger disk
> size
> > (100Gb) than on the other nodes. However, Amabri is reporting a total of
> 8.6
> > Gb disk space. lsblk correctly reports the disk size.
> >
> > Does anyone why this might be? As I understand things you need to tell
> HDFS
> > how much space *not* to use, and it will use what it needs from the rest.
> >
> > (BTW this is not the same data node in my other question on this list,
> > "Cannot start DataNode after adding new volume.")
> >
> > Thanks,
> > Charles
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>
Re: DataNode not recognising available disk space
Posted by Charles Robertson <ch...@gmail.com>.
Hi Yusaku,
Thank you for your reply, unfortunately that's not the problem - the new
slave node has a single drive, so is using the default data directory path.
I'll post this to the Ambari list.
Regards,
Charles
On 16 September 2014 23:12, Yusaku Sako <yu...@hortonworks.com> wrote:
> Charles,
>
> If the newly added slave node has an extra disk that the other nodes
> don't have, then you will have to do the following on the Ambari Web
> UI so that "dfs.data.dir" is reflected to include this extra drive for
> that node:
> * Go to Services > HDFS > Configs
> * Click on "Manage Config Groups" link
> * Create a new HDFS Configuration Group (name it appropriately), add
> the slave host in question to that group, and Save.
> * In the list of configs for HDFS, you will see "DataNode directories"
> with the current directories set up for DataNodes for all other hosts.
> * Click on the "+" next to "DataNode directories" and list out all the
> drives for this new DataNode.
> * Save the configuration changes.
> * Restart the DataNode.
> * Ambari (and HDFS) should start reflecting the added disk space.
>
> I hope this solves your problem.
>
> FYI, there's an Ambari-specific mailing list at user@ambari.apache.org
> that you can subscribe by sending an email to
> user-subscribe@ambari.apache.org.
>
> Thanks,
> Yusaku
>
>
> On Tue, Sep 16, 2014 at 5:28 AM, Charles Robertson
> <ch...@gmail.com> wrote:
> > Hi all,
> >
> > I've added a new slave node to my cluster with a (single) larger disk
> size
> > (100Gb) than on the other nodes. However, Amabri is reporting a total of
> 8.6
> > Gb disk space. lsblk correctly reports the disk size.
> >
> > Does anyone why this might be? As I understand things you need to tell
> HDFS
> > how much space *not* to use, and it will use what it needs from the rest.
> >
> > (BTW this is not the same data node in my other question on this list,
> > "Cannot start DataNode after adding new volume.")
> >
> > Thanks,
> > Charles
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>
Re: DataNode not recognising available disk space
Posted by Charles Robertson <ch...@gmail.com>.
Hi Yusaku,
Thank you for your reply, unfortunately that's not the problem - the new
slave node has a single drive, so is using the default data directory path.
I'll post this to the Ambari list.
Regards,
Charles
On 16 September 2014 23:12, Yusaku Sako <yu...@hortonworks.com> wrote:
> Charles,
>
> If the newly added slave node has an extra disk that the other nodes
> don't have, then you will have to do the following on the Ambari Web
> UI so that "dfs.data.dir" is reflected to include this extra drive for
> that node:
> * Go to Services > HDFS > Configs
> * Click on "Manage Config Groups" link
> * Create a new HDFS Configuration Group (name it appropriately), add
> the slave host in question to that group, and Save.
> * In the list of configs for HDFS, you will see "DataNode directories"
> with the current directories set up for DataNodes for all other hosts.
> * Click on the "+" next to "DataNode directories" and list out all the
> drives for this new DataNode.
> * Save the configuration changes.
> * Restart the DataNode.
> * Ambari (and HDFS) should start reflecting the added disk space.
>
> I hope this solves your problem.
>
> FYI, there's an Ambari-specific mailing list at user@ambari.apache.org
> that you can subscribe by sending an email to
> user-subscribe@ambari.apache.org.
>
> Thanks,
> Yusaku
>
>
> On Tue, Sep 16, 2014 at 5:28 AM, Charles Robertson
> <ch...@gmail.com> wrote:
> > Hi all,
> >
> > I've added a new slave node to my cluster with a (single) larger disk
> size
> > (100Gb) than on the other nodes. However, Amabri is reporting a total of
> 8.6
> > Gb disk space. lsblk correctly reports the disk size.
> >
> > Does anyone why this might be? As I understand things you need to tell
> HDFS
> > how much space *not* to use, and it will use what it needs from the rest.
> >
> > (BTW this is not the same data node in my other question on this list,
> > "Cannot start DataNode after adding new volume.")
> >
> > Thanks,
> > Charles
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>
Re: DataNode not recognising available disk space
Posted by Charles Robertson <ch...@gmail.com>.
Hi Yusaku,
Thank you for your reply, unfortunately that's not the problem - the new
slave node has a single drive, so is using the default data directory path.
I'll post this to the Ambari list.
Regards,
Charles
On 16 September 2014 23:12, Yusaku Sako <yu...@hortonworks.com> wrote:
> Charles,
>
> If the newly added slave node has an extra disk that the other nodes
> don't have, then you will have to do the following on the Ambari Web
> UI so that "dfs.data.dir" is reflected to include this extra drive for
> that node:
> * Go to Services > HDFS > Configs
> * Click on "Manage Config Groups" link
> * Create a new HDFS Configuration Group (name it appropriately), add
> the slave host in question to that group, and Save.
> * In the list of configs for HDFS, you will see "DataNode directories"
> with the current directories set up for DataNodes for all other hosts.
> * Click on the "+" next to "DataNode directories" and list out all the
> drives for this new DataNode.
> * Save the configuration changes.
> * Restart the DataNode.
> * Ambari (and HDFS) should start reflecting the added disk space.
>
> I hope this solves your problem.
>
> FYI, there's an Ambari-specific mailing list at user@ambari.apache.org
> that you can subscribe by sending an email to
> user-subscribe@ambari.apache.org.
>
> Thanks,
> Yusaku
>
>
> On Tue, Sep 16, 2014 at 5:28 AM, Charles Robertson
> <ch...@gmail.com> wrote:
> > Hi all,
> >
> > I've added a new slave node to my cluster with a (single) larger disk
> size
> > (100Gb) than on the other nodes. However, Amabri is reporting a total of
> 8.6
> > Gb disk space. lsblk correctly reports the disk size.
> >
> > Does anyone why this might be? As I understand things you need to tell
> HDFS
> > how much space *not* to use, and it will use what it needs from the rest.
> >
> > (BTW this is not the same data node in my other question on this list,
> > "Cannot start DataNode after adding new volume.")
> >
> > Thanks,
> > Charles
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>
Re: DataNode not recognising available disk space
Posted by Yusaku Sako <yu...@hortonworks.com>.
Charles,
If the newly added slave node has an extra disk that the other nodes
don't have, then you will have to do the following on the Ambari Web
UI so that "dfs.data.dir" is reflected to include this extra drive for
that node:
* Go to Services > HDFS > Configs
* Click on "Manage Config Groups" link
* Create a new HDFS Configuration Group (name it appropriately), add
the slave host in question to that group, and Save.
* In the list of configs for HDFS, you will see "DataNode directories"
with the current directories set up for DataNodes for all other hosts.
* Click on the "+" next to "DataNode directories" and list out all the
drives for this new DataNode.
* Save the configuration changes.
* Restart the DataNode.
* Ambari (and HDFS) should start reflecting the added disk space.
I hope this solves your problem.
FYI, there's an Ambari-specific mailing list at user@ambari.apache.org
that you can subscribe by sending an email to
user-subscribe@ambari.apache.org.
Thanks,
Yusaku
On Tue, Sep 16, 2014 at 5:28 AM, Charles Robertson
<ch...@gmail.com> wrote:
> Hi all,
>
> I've added a new slave node to my cluster with a (single) larger disk size
> (100Gb) than on the other nodes. However, Amabri is reporting a total of 8.6
> Gb disk space. lsblk correctly reports the disk size.
>
> Does anyone why this might be? As I understand things you need to tell HDFS
> how much space *not* to use, and it will use what it needs from the rest.
>
> (BTW this is not the same data node in my other question on this list,
> "Cannot start DataNode after adding new volume.")
>
> Thanks,
> Charles
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: DataNode not recognising available disk space
Posted by Yusaku Sako <yu...@hortonworks.com>.
Charles,
If the newly added slave node has an extra disk that the other nodes
don't have, then you will have to do the following on the Ambari Web
UI so that "dfs.data.dir" is reflected to include this extra drive for
that node:
* Go to Services > HDFS > Configs
* Click on "Manage Config Groups" link
* Create a new HDFS Configuration Group (name it appropriately), add
the slave host in question to that group, and Save.
* In the list of configs for HDFS, you will see "DataNode directories"
with the current directories set up for DataNodes for all other hosts.
* Click on the "+" next to "DataNode directories" and list out all the
drives for this new DataNode.
* Save the configuration changes.
* Restart the DataNode.
* Ambari (and HDFS) should start reflecting the added disk space.
I hope this solves your problem.
FYI, there's an Ambari-specific mailing list at user@ambari.apache.org
that you can subscribe by sending an email to
user-subscribe@ambari.apache.org.
Thanks,
Yusaku
On Tue, Sep 16, 2014 at 5:28 AM, Charles Robertson
<ch...@gmail.com> wrote:
> Hi all,
>
> I've added a new slave node to my cluster with a (single) larger disk size
> (100Gb) than on the other nodes. However, Amabri is reporting a total of 8.6
> Gb disk space. lsblk correctly reports the disk size.
>
> Does anyone why this might be? As I understand things you need to tell HDFS
> how much space *not* to use, and it will use what it needs from the rest.
>
> (BTW this is not the same data node in my other question on this list,
> "Cannot start DataNode after adding new volume.")
>
> Thanks,
> Charles
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: DataNode not recognising available disk space
Posted by Yusaku Sako <yu...@hortonworks.com>.
Charles,
If the newly added slave node has an extra disk that the other nodes
don't have, then you will have to do the following on the Ambari Web
UI so that "dfs.data.dir" is reflected to include this extra drive for
that node:
* Go to Services > HDFS > Configs
* Click on "Manage Config Groups" link
* Create a new HDFS Configuration Group (name it appropriately), add
the slave host in question to that group, and Save.
* In the list of configs for HDFS, you will see "DataNode directories"
with the current directories set up for DataNodes for all other hosts.
* Click on the "+" next to "DataNode directories" and list out all the
drives for this new DataNode.
* Save the configuration changes.
* Restart the DataNode.
* Ambari (and HDFS) should start reflecting the added disk space.
I hope this solves your problem.
FYI, there's an Ambari-specific mailing list at user@ambari.apache.org
that you can subscribe by sending an email to
user-subscribe@ambari.apache.org.
Thanks,
Yusaku
On Tue, Sep 16, 2014 at 5:28 AM, Charles Robertson
<ch...@gmail.com> wrote:
> Hi all,
>
> I've added a new slave node to my cluster with a (single) larger disk size
> (100Gb) than on the other nodes. However, Amabri is reporting a total of 8.6
> Gb disk space. lsblk correctly reports the disk size.
>
> Does anyone why this might be? As I understand things you need to tell HDFS
> how much space *not* to use, and it will use what it needs from the rest.
>
> (BTW this is not the same data node in my other question on this list,
> "Cannot start DataNode after adding new volume.")
>
> Thanks,
> Charles
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: DataNode not recognising available disk space
Posted by Yusaku Sako <yu...@hortonworks.com>.
Charles,
If the newly added slave node has an extra disk that the other nodes
don't have, then you will have to do the following on the Ambari Web
UI so that "dfs.data.dir" is reflected to include this extra drive for
that node:
* Go to Services > HDFS > Configs
* Click on "Manage Config Groups" link
* Create a new HDFS Configuration Group (name it appropriately), add
the slave host in question to that group, and Save.
* In the list of configs for HDFS, you will see "DataNode directories"
with the current directories set up for DataNodes for all other hosts.
* Click on the "+" next to "DataNode directories" and list out all the
drives for this new DataNode.
* Save the configuration changes.
* Restart the DataNode.
* Ambari (and HDFS) should start reflecting the added disk space.
I hope this solves your problem.
FYI, there's an Ambari-specific mailing list at user@ambari.apache.org
that you can subscribe by sending an email to
user-subscribe@ambari.apache.org.
Thanks,
Yusaku
On Tue, Sep 16, 2014 at 5:28 AM, Charles Robertson
<ch...@gmail.com> wrote:
> Hi all,
>
> I've added a new slave node to my cluster with a (single) larger disk size
> (100Gb) than on the other nodes. However, Amabri is reporting a total of 8.6
> Gb disk space. lsblk correctly reports the disk size.
>
> Does anyone why this might be? As I understand things you need to tell HDFS
> how much space *not* to use, and it will use what it needs from the rest.
>
> (BTW this is not the same data node in my other question on this list,
> "Cannot start DataNode after adding new volume.")
>
> Thanks,
> Charles
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.