You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-user@hadoop.apache.org by Brian Jeltema <br...@digitalenvoy.net> on 2014/05/29 16:52:37 UTC
full partition handling
Google wasn’t helpful for this question:
If a Hadoop cluster has multiple disk partitions assigned to HDFS and one of those partitions is full,
how does HDFS react?
TIA
Brian
Re: full partition handling
Posted by Rahul Chaudhari <ra...@gmail.com>.
There is no error reported as that specific partition will not be used for
further write requests. In the configurations, you can specify multiple
disk partitions for data node and even if any of that partition does not
exists, it is simply skipped by hadoop.
_Rahul
On Thu, May 29, 2014 at 8:39 PM, Aitor Perez Cedres <ap...@pragsis.com>wrote:
>
> Hi Brian,
>
> I'm not sure how it will react; I guess the datanode will report an error
> of no space left on device and fail to serve some requests trying to write
> on the partition that is full.
>
> To solve this issue, you have to manually move files from
> *dfs.datanode.data.dir*/current to a different partition. Also you have
> to stop the datanode before this "re-balancing".
>
>
> On 29/05/14 16:52, Brian Jeltema wrote:
>
> Google wasn’t helpful for this question:
>
> If a Hadoop cluster has multiple disk partitions assigned to HDFS and one of those partitions is full,
> how does HDFS react?
>
> TIA
>
> Brian
>
>
> --
> *Aitor Pérez*
> *Big Data System Engineer*
>
> Telf.: +34 917 680 490
> Fax: +34 913 833 301
> C/Manuel Tovar, 49-53 - 28034 Madrid - Spain
>
> *http://www.bidoop.es <http://www.bidoop.es>*
>
--
Regards,
Rahul Chaudhari
Re: full partition handling
Posted by Rahul Chaudhari <ra...@gmail.com>.
There is no error reported as that specific partition will not be used for
further write requests. In the configurations, you can specify multiple
disk partitions for data node and even if any of that partition does not
exists, it is simply skipped by hadoop.
_Rahul
On Thu, May 29, 2014 at 8:39 PM, Aitor Perez Cedres <ap...@pragsis.com>wrote:
>
> Hi Brian,
>
> I'm not sure how it will react; I guess the datanode will report an error
> of no space left on device and fail to serve some requests trying to write
> on the partition that is full.
>
> To solve this issue, you have to manually move files from
> *dfs.datanode.data.dir*/current to a different partition. Also you have
> to stop the datanode before this "re-balancing".
>
>
> On 29/05/14 16:52, Brian Jeltema wrote:
>
> Google wasn’t helpful for this question:
>
> If a Hadoop cluster has multiple disk partitions assigned to HDFS and one of those partitions is full,
> how does HDFS react?
>
> TIA
>
> Brian
>
>
> --
> *Aitor Pérez*
> *Big Data System Engineer*
>
> Telf.: +34 917 680 490
> Fax: +34 913 833 301
> C/Manuel Tovar, 49-53 - 28034 Madrid - Spain
>
> *http://www.bidoop.es <http://www.bidoop.es>*
>
--
Regards,
Rahul Chaudhari
Re: full partition handling
Posted by Rahul Chaudhari <ra...@gmail.com>.
There is no error reported as that specific partition will not be used for
further write requests. In the configurations, you can specify multiple
disk partitions for data node and even if any of that partition does not
exists, it is simply skipped by hadoop.
_Rahul
On Thu, May 29, 2014 at 8:39 PM, Aitor Perez Cedres <ap...@pragsis.com>wrote:
>
> Hi Brian,
>
> I'm not sure how it will react; I guess the datanode will report an error
> of no space left on device and fail to serve some requests trying to write
> on the partition that is full.
>
> To solve this issue, you have to manually move files from
> *dfs.datanode.data.dir*/current to a different partition. Also you have
> to stop the datanode before this "re-balancing".
>
>
> On 29/05/14 16:52, Brian Jeltema wrote:
>
> Google wasn’t helpful for this question:
>
> If a Hadoop cluster has multiple disk partitions assigned to HDFS and one of those partitions is full,
> how does HDFS react?
>
> TIA
>
> Brian
>
>
> --
> *Aitor Pérez*
> *Big Data System Engineer*
>
> Telf.: +34 917 680 490
> Fax: +34 913 833 301
> C/Manuel Tovar, 49-53 - 28034 Madrid - Spain
>
> *http://www.bidoop.es <http://www.bidoop.es>*
>
--
Regards,
Rahul Chaudhari
Re: full partition handling
Posted by Rahul Chaudhari <ra...@gmail.com>.
There is no error reported as that specific partition will not be used for
further write requests. In the configurations, you can specify multiple
disk partitions for data node and even if any of that partition does not
exists, it is simply skipped by hadoop.
_Rahul
On Thu, May 29, 2014 at 8:39 PM, Aitor Perez Cedres <ap...@pragsis.com>wrote:
>
> Hi Brian,
>
> I'm not sure how it will react; I guess the datanode will report an error
> of no space left on device and fail to serve some requests trying to write
> on the partition that is full.
>
> To solve this issue, you have to manually move files from
> *dfs.datanode.data.dir*/current to a different partition. Also you have
> to stop the datanode before this "re-balancing".
>
>
> On 29/05/14 16:52, Brian Jeltema wrote:
>
> Google wasn’t helpful for this question:
>
> If a Hadoop cluster has multiple disk partitions assigned to HDFS and one of those partitions is full,
> how does HDFS react?
>
> TIA
>
> Brian
>
>
> --
> *Aitor Pérez*
> *Big Data System Engineer*
>
> Telf.: +34 917 680 490
> Fax: +34 913 833 301
> C/Manuel Tovar, 49-53 - 28034 Madrid - Spain
>
> *http://www.bidoop.es <http://www.bidoop.es>*
>
--
Regards,
Rahul Chaudhari
Re: full partition handling
Posted by Aitor Perez Cedres <ap...@pragsis.com>.
Hi Brian,
I'm not sure how it will react; I guess the datanode will report an
error of no space left on device and fail to serve some requests trying
to write on the partition that is full.
To solve this issue, you have to manually move files from
/dfs.datanode.data.dir//current to a different partition. Also you have
to stop the datanode before this "re-balancing".
On 29/05/14 16:52, Brian Jeltema wrote:
> Google wasn’t helpful for this question:
>
> If a Hadoop cluster has multiple disk partitions assigned to HDFS and one of those partitions is full,
> how does HDFS react?
>
> TIA
>
> Brian
--
*Aitor Pérez*
/Big Data System Engineer/
Telf.: +34 917 680 490
Fax: +34 913 833 301
C/Manuel Tovar, 49-53 - 28034 Madrid - Spain
_http://www.bidoop.es_
Re: full partition handling
Posted by Aitor Perez Cedres <ap...@pragsis.com>.
Hi Brian,
I'm not sure how it will react; I guess the datanode will report an
error of no space left on device and fail to serve some requests trying
to write on the partition that is full.
To solve this issue, you have to manually move files from
/dfs.datanode.data.dir//current to a different partition. Also you have
to stop the datanode before this "re-balancing".
On 29/05/14 16:52, Brian Jeltema wrote:
> Google wasn’t helpful for this question:
>
> If a Hadoop cluster has multiple disk partitions assigned to HDFS and one of those partitions is full,
> how does HDFS react?
>
> TIA
>
> Brian
--
*Aitor Pérez*
/Big Data System Engineer/
Telf.: +34 917 680 490
Fax: +34 913 833 301
C/Manuel Tovar, 49-53 - 28034 Madrid - Spain
_http://www.bidoop.es_
Re: full partition handling
Posted by Aitor Perez Cedres <ap...@pragsis.com>.
Hi Brian,
I'm not sure how it will react; I guess the datanode will report an
error of no space left on device and fail to serve some requests trying
to write on the partition that is full.
To solve this issue, you have to manually move files from
/dfs.datanode.data.dir//current to a different partition. Also you have
to stop the datanode before this "re-balancing".
On 29/05/14 16:52, Brian Jeltema wrote:
> Google wasn’t helpful for this question:
>
> If a Hadoop cluster has multiple disk partitions assigned to HDFS and one of those partitions is full,
> how does HDFS react?
>
> TIA
>
> Brian
--
*Aitor Pérez*
/Big Data System Engineer/
Telf.: +34 917 680 490
Fax: +34 913 833 301
C/Manuel Tovar, 49-53 - 28034 Madrid - Spain
_http://www.bidoop.es_
Re: full partition handling
Posted by Aitor Perez Cedres <ap...@pragsis.com>.
Hi Brian,
I'm not sure how it will react; I guess the datanode will report an
error of no space left on device and fail to serve some requests trying
to write on the partition that is full.
To solve this issue, you have to manually move files from
/dfs.datanode.data.dir//current to a different partition. Also you have
to stop the datanode before this "re-balancing".
On 29/05/14 16:52, Brian Jeltema wrote:
> Google wasn’t helpful for this question:
>
> If a Hadoop cluster has multiple disk partitions assigned to HDFS and one of those partitions is full,
> how does HDFS react?
>
> TIA
>
> Brian
--
*Aitor Pérez*
/Big Data System Engineer/
Telf.: +34 917 680 490
Fax: +34 913 833 301
C/Manuel Tovar, 49-53 - 28034 Madrid - Spain
_http://www.bidoop.es_