You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-user@hadoop.apache.org by "Brian C. Huffman" <bh...@etinternational.com> on 2014/10/06 15:46:18 UTC
Datanode volume full, but not moving to free volume
All,
I have a small hadoop cluster (2.5.0) with 4 datanodes and 3 data disks
per node. Lately some of the volumes have been filling, but instead of
moving to other configured volumes that *have* free space, it's giving
errors in the datanode logs:
2014-10-03 11:52:44,989 ERROR
org.apache.hadoop.hdfs.server.datanode.DataNode:
thor2.xmen.eti:50010:DataXceiver error processing WRITE_BLOCK
operation src: /172.17.1.3:35412 dst: /172.17.1.2:50010
java.io.IOException: No space left on device
at java.io.FileOutputStream.writeBytes(Native Method)
at java.io.FileOutputStream.write(FileOutputStream.java:345)
at
org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:592)
at
org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:734)
at
org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:741)
at
org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:124)
at
org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:71)
at
org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:234)
at java.lang.Thread.run(Thread.java:745)
Unfortunately it's continuing to try to write and when it fails, it's
passing the exception to the client.
I did a restart and then it seemed to figure out that it should move to
the next volume.
Any suggestions to keep this from happening in the future?
Also - could it be an issue that I have a small amount of non-HDFS data
on those volumes?
Thanks,
Brian
Re: Datanode volume full, but not moving to free volume
Posted by Azuryy Yu <az...@gmail.com>.
Hi Brian,
Did you try set dfs.datanode.fsdataset.volume.choosing.policy to
org.apache.hadoop.hdfs.server.datanode.fsdataset.AvailableSpaceVolumeChoosingPolicy
then there are some other related options with this policy. you can google
it.
On Wed, Oct 8, 2014 at 4:44 PM, Aitor Cedres <ac...@pivotal.io> wrote:
>
> Hi Brian,
>
> Hadoop does not balance the disks within a DataNode. If you ran out of
> space and then add additional disks, you should shutdown the DataNode and
> move manually a few files to the new disk.
>
> Regards,
>
> Aitor Cedrés
>
>
> On 6 October 2014 14:46, Brian C. Huffman <bh...@etinternational.com>
> wrote:
>
>> All,
>>
>> I have a small hadoop cluster (2.5.0) with 4 datanodes and 3 data disks
>> per node. Lately some of the volumes have been filling, but instead of
>> moving to other configured volumes that *have* free space, it's giving
>> errors in the datanode logs:
>> 2014-10-03 11:52:44,989 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode:
>> thor2.xmen.eti:50010:DataXceiver error processing WRITE_BLOCK
>> operation src: /172.17.1.3:35412 dst: /172.17.1.2:50010
>> java.io.IOException: No space left on device
>> at java.io.FileOutputStream.writeBytes(Native Method)
>> at java.io.FileOutputStream.write(FileOutputStream.java:345)
>> at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.
>> receivePacket(BlockReceiver.java:592)
>> at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.
>> receiveBlock(BlockReceiver.java:734)
>> at org.apache.hadoop.hdfs.server.datanode.DataXceiver.
>> writeBlock(DataXceiver.java:741)
>> at org.apache.hadoop.hdfs.protocol.datatransfer.
>> Receiver.opWriteBlock(Receiver.java:124)
>> at org.apache.hadoop.hdfs.protocol.datatransfer.
>> Receiver.processOp(Receiver.java:71)
>> at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(
>> DataXceiver.java:234)
>> at java.lang.Thread.run(Thread.java:745)
>>
>> Unfortunately it's continuing to try to write and when it fails, it's
>> passing the exception to the client.
>>
>> I did a restart and then it seemed to figure out that it should move to
>> the next volume.
>>
>> Any suggestions to keep this from happening in the future?
>>
>> Also - could it be an issue that I have a small amount of non-HDFS data
>> on those volumes?
>>
>> Thanks,
>> Brian
>>
>>
>
Re: Datanode volume full, but not moving to free volume
Posted by Azuryy Yu <az...@gmail.com>.
yes, all blockpool directory names are all same is expected if you don't
configure HDFS federation.
On Wed, Oct 8, 2014 at 7:39 PM, Brian C. Huffman <
bhuffman@etinternational.com> wrote:
> Hmmm.. It seems that there's only one block pool per disk. So that
> won't help me. :-(
>
> Also, I see the blockpool directory names are all the same. Is that
> expected? So even if I put a larger disk in, I couldn't consolidate the
> smaller disk's blockpool directories?
> [hadoop@thor1 current]$ ls
> /data/data1/hadoop/yarn_data/hdfs/datanode/current
> BP-1408773897-172.17.1.1-1400769841207 VERSION
> [hadoop@thor1 current]$ ls
> /data/data2/hadoop/yarn_data/hdfs/datanode/current
> BP-1408773897-172.17.1.1-1400769841207 VERSION
> [hadoop@thor1 current]$ ls
> /data/data3/hadoop/yarn_data/hdfs/datanode/current
> BP-1408773897-172.17.1.1-1400769841207 VERSION
>
> Regards,
> Brian
>
>
> On 10/8/14, 7:14 AM, Aitor Cedres wrote:
>
>
> Hi Brian,
>
> I would try to move the Block Pools directories
> (BP-1408773897-172.17.1.1-1400769841207). You must shutdown your DataNode
> process before doing this operation.
>
> Regards,
>
> Aitor Cedrés
>
> On 8 October 2014 11:46, Brian C. Huffman <bh...@etinternational.com>
> wrote:
>
>> Can I move a whole subdir? Or does it have to be individual block
>> files / metadata?
>>
>> For example, I see this:
>> [hadoop@thor1 finalized]$ pwd
>>
>> /data/data2/hadoop/yarn_data/hdfs/datanode/current/BP-1408773897-172.17.1.1-1400769841207/current/finalized
>> [hadoop@thor1 finalized]$ du -sh subdir10/
>> 80G subdir10/
>>
>> So could I move subdir10 to the same location under /data/data3?
>>
>> Thanks,
>> Brian
>>
>>
>> Brian C. Huffman System Administrator ET International, Inc.On 10/8/14,
>> 4:44 AM, Aitor Cedres wrote:
>>
>>
>> Hi Brian,
>>
>> Hadoop does not balance the disks within a DataNode. If you ran out of
>> space and then add additional disks, you should shutdown the DataNode and
>> move manually a few files to the new disk.
>>
>> Regards,
>>
>> Aitor Cedrés
>>
>> On 6 October 2014 14:46, Brian C. Huffman <bhuffman@etinternational.com
>> > wrote:
>>
>>> All,
>>>
>>> I have a small hadoop cluster (2.5.0) with 4 datanodes and 3 data disks
>>> per node. Lately some of the volumes have been filling, but instead of
>>> moving to other configured volumes that *have* free space, it's giving
>>> errors in the datanode logs:
>>> 2014-10-03 11:52:44,989 ERROR
>>> org.apache.hadoop.hdfs.server.datanode.DataNode:
>>> thor2.xmen.eti:50010:DataXceiver error processing WRITE_BLOCK
>>> operation src: /172.17.1.3:35412 dst: /172.17.1.2:50010
>>> java.io.IOException: No space left on device
>>> at java.io.FileOutputStream.writeBytes(Native Method)
>>> at java.io.FileOutputStream.write(FileOutputStream.java:345)
>>> at
>>> org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:592)
>>> at
>>> org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:734)
>>> at
>>> org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:741)
>>> at
>>> org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:124)
>>> at
>>> org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:71)
>>> at
>>> org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:234)
>>> at java.lang.Thread.run(Thread.java:745)
>>>
>>> Unfortunately it's continuing to try to write and when it fails, it's
>>> passing the exception to the client.
>>>
>>> I did a restart and then it seemed to figure out that it should move to
>>> the next volume.
>>>
>>> Any suggestions to keep this from happening in the future?
>>>
>>> Also - could it be an issue that I have a small amount of non-HDFS data
>>> on those volumes?
>>>
>>> Thanks,
>>> Brian
>>>
>>>
>>
>>
>
>
Re: Datanode volume full, but not moving to free volume
Posted by Azuryy Yu <az...@gmail.com>.
yes, all blockpool directory names are all same is expected if you don't
configure HDFS federation.
On Wed, Oct 8, 2014 at 7:39 PM, Brian C. Huffman <
bhuffman@etinternational.com> wrote:
> Hmmm.. It seems that there's only one block pool per disk. So that
> won't help me. :-(
>
> Also, I see the blockpool directory names are all the same. Is that
> expected? So even if I put a larger disk in, I couldn't consolidate the
> smaller disk's blockpool directories?
> [hadoop@thor1 current]$ ls
> /data/data1/hadoop/yarn_data/hdfs/datanode/current
> BP-1408773897-172.17.1.1-1400769841207 VERSION
> [hadoop@thor1 current]$ ls
> /data/data2/hadoop/yarn_data/hdfs/datanode/current
> BP-1408773897-172.17.1.1-1400769841207 VERSION
> [hadoop@thor1 current]$ ls
> /data/data3/hadoop/yarn_data/hdfs/datanode/current
> BP-1408773897-172.17.1.1-1400769841207 VERSION
>
> Regards,
> Brian
>
>
> On 10/8/14, 7:14 AM, Aitor Cedres wrote:
>
>
> Hi Brian,
>
> I would try to move the Block Pools directories
> (BP-1408773897-172.17.1.1-1400769841207). You must shutdown your DataNode
> process before doing this operation.
>
> Regards,
>
> Aitor Cedrés
>
> On 8 October 2014 11:46, Brian C. Huffman <bh...@etinternational.com>
> wrote:
>
>> Can I move a whole subdir? Or does it have to be individual block
>> files / metadata?
>>
>> For example, I see this:
>> [hadoop@thor1 finalized]$ pwd
>>
>> /data/data2/hadoop/yarn_data/hdfs/datanode/current/BP-1408773897-172.17.1.1-1400769841207/current/finalized
>> [hadoop@thor1 finalized]$ du -sh subdir10/
>> 80G subdir10/
>>
>> So could I move subdir10 to the same location under /data/data3?
>>
>> Thanks,
>> Brian
>>
>>
>> Brian C. Huffman System Administrator ET International, Inc.On 10/8/14,
>> 4:44 AM, Aitor Cedres wrote:
>>
>>
>> Hi Brian,
>>
>> Hadoop does not balance the disks within a DataNode. If you ran out of
>> space and then add additional disks, you should shutdown the DataNode and
>> move manually a few files to the new disk.
>>
>> Regards,
>>
>> Aitor Cedrés
>>
>> On 6 October 2014 14:46, Brian C. Huffman <bhuffman@etinternational.com
>> > wrote:
>>
>>> All,
>>>
>>> I have a small hadoop cluster (2.5.0) with 4 datanodes and 3 data disks
>>> per node. Lately some of the volumes have been filling, but instead of
>>> moving to other configured volumes that *have* free space, it's giving
>>> errors in the datanode logs:
>>> 2014-10-03 11:52:44,989 ERROR
>>> org.apache.hadoop.hdfs.server.datanode.DataNode:
>>> thor2.xmen.eti:50010:DataXceiver error processing WRITE_BLOCK
>>> operation src: /172.17.1.3:35412 dst: /172.17.1.2:50010
>>> java.io.IOException: No space left on device
>>> at java.io.FileOutputStream.writeBytes(Native Method)
>>> at java.io.FileOutputStream.write(FileOutputStream.java:345)
>>> at
>>> org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:592)
>>> at
>>> org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:734)
>>> at
>>> org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:741)
>>> at
>>> org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:124)
>>> at
>>> org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:71)
>>> at
>>> org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:234)
>>> at java.lang.Thread.run(Thread.java:745)
>>>
>>> Unfortunately it's continuing to try to write and when it fails, it's
>>> passing the exception to the client.
>>>
>>> I did a restart and then it seemed to figure out that it should move to
>>> the next volume.
>>>
>>> Any suggestions to keep this from happening in the future?
>>>
>>> Also - could it be an issue that I have a small amount of non-HDFS data
>>> on those volumes?
>>>
>>> Thanks,
>>> Brian
>>>
>>>
>>
>>
>
>
Re: Datanode volume full, but not moving to free volume
Posted by Azuryy Yu <az...@gmail.com>.
yes, all blockpool directory names are all same is expected if you don't
configure HDFS federation.
On Wed, Oct 8, 2014 at 7:39 PM, Brian C. Huffman <
bhuffman@etinternational.com> wrote:
> Hmmm.. It seems that there's only one block pool per disk. So that
> won't help me. :-(
>
> Also, I see the blockpool directory names are all the same. Is that
> expected? So even if I put a larger disk in, I couldn't consolidate the
> smaller disk's blockpool directories?
> [hadoop@thor1 current]$ ls
> /data/data1/hadoop/yarn_data/hdfs/datanode/current
> BP-1408773897-172.17.1.1-1400769841207 VERSION
> [hadoop@thor1 current]$ ls
> /data/data2/hadoop/yarn_data/hdfs/datanode/current
> BP-1408773897-172.17.1.1-1400769841207 VERSION
> [hadoop@thor1 current]$ ls
> /data/data3/hadoop/yarn_data/hdfs/datanode/current
> BP-1408773897-172.17.1.1-1400769841207 VERSION
>
> Regards,
> Brian
>
>
> On 10/8/14, 7:14 AM, Aitor Cedres wrote:
>
>
> Hi Brian,
>
> I would try to move the Block Pools directories
> (BP-1408773897-172.17.1.1-1400769841207). You must shutdown your DataNode
> process before doing this operation.
>
> Regards,
>
> Aitor Cedrés
>
> On 8 October 2014 11:46, Brian C. Huffman <bh...@etinternational.com>
> wrote:
>
>> Can I move a whole subdir? Or does it have to be individual block
>> files / metadata?
>>
>> For example, I see this:
>> [hadoop@thor1 finalized]$ pwd
>>
>> /data/data2/hadoop/yarn_data/hdfs/datanode/current/BP-1408773897-172.17.1.1-1400769841207/current/finalized
>> [hadoop@thor1 finalized]$ du -sh subdir10/
>> 80G subdir10/
>>
>> So could I move subdir10 to the same location under /data/data3?
>>
>> Thanks,
>> Brian
>>
>>
>> Brian C. Huffman System Administrator ET International, Inc.On 10/8/14,
>> 4:44 AM, Aitor Cedres wrote:
>>
>>
>> Hi Brian,
>>
>> Hadoop does not balance the disks within a DataNode. If you ran out of
>> space and then add additional disks, you should shutdown the DataNode and
>> move manually a few files to the new disk.
>>
>> Regards,
>>
>> Aitor Cedrés
>>
>> On 6 October 2014 14:46, Brian C. Huffman <bhuffman@etinternational.com
>> > wrote:
>>
>>> All,
>>>
>>> I have a small hadoop cluster (2.5.0) with 4 datanodes and 3 data disks
>>> per node. Lately some of the volumes have been filling, but instead of
>>> moving to other configured volumes that *have* free space, it's giving
>>> errors in the datanode logs:
>>> 2014-10-03 11:52:44,989 ERROR
>>> org.apache.hadoop.hdfs.server.datanode.DataNode:
>>> thor2.xmen.eti:50010:DataXceiver error processing WRITE_BLOCK
>>> operation src: /172.17.1.3:35412 dst: /172.17.1.2:50010
>>> java.io.IOException: No space left on device
>>> at java.io.FileOutputStream.writeBytes(Native Method)
>>> at java.io.FileOutputStream.write(FileOutputStream.java:345)
>>> at
>>> org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:592)
>>> at
>>> org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:734)
>>> at
>>> org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:741)
>>> at
>>> org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:124)
>>> at
>>> org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:71)
>>> at
>>> org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:234)
>>> at java.lang.Thread.run(Thread.java:745)
>>>
>>> Unfortunately it's continuing to try to write and when it fails, it's
>>> passing the exception to the client.
>>>
>>> I did a restart and then it seemed to figure out that it should move to
>>> the next volume.
>>>
>>> Any suggestions to keep this from happening in the future?
>>>
>>> Also - could it be an issue that I have a small amount of non-HDFS data
>>> on those volumes?
>>>
>>> Thanks,
>>> Brian
>>>
>>>
>>
>>
>
>
Re: Datanode volume full, but not moving to free volume
Posted by Azuryy Yu <az...@gmail.com>.
yes, all blockpool directory names are all same is expected if you don't
configure HDFS federation.
On Wed, Oct 8, 2014 at 7:39 PM, Brian C. Huffman <
bhuffman@etinternational.com> wrote:
> Hmmm.. It seems that there's only one block pool per disk. So that
> won't help me. :-(
>
> Also, I see the blockpool directory names are all the same. Is that
> expected? So even if I put a larger disk in, I couldn't consolidate the
> smaller disk's blockpool directories?
> [hadoop@thor1 current]$ ls
> /data/data1/hadoop/yarn_data/hdfs/datanode/current
> BP-1408773897-172.17.1.1-1400769841207 VERSION
> [hadoop@thor1 current]$ ls
> /data/data2/hadoop/yarn_data/hdfs/datanode/current
> BP-1408773897-172.17.1.1-1400769841207 VERSION
> [hadoop@thor1 current]$ ls
> /data/data3/hadoop/yarn_data/hdfs/datanode/current
> BP-1408773897-172.17.1.1-1400769841207 VERSION
>
> Regards,
> Brian
>
>
> On 10/8/14, 7:14 AM, Aitor Cedres wrote:
>
>
> Hi Brian,
>
> I would try to move the Block Pools directories
> (BP-1408773897-172.17.1.1-1400769841207). You must shutdown your DataNode
> process before doing this operation.
>
> Regards,
>
> Aitor Cedrés
>
> On 8 October 2014 11:46, Brian C. Huffman <bh...@etinternational.com>
> wrote:
>
>> Can I move a whole subdir? Or does it have to be individual block
>> files / metadata?
>>
>> For example, I see this:
>> [hadoop@thor1 finalized]$ pwd
>>
>> /data/data2/hadoop/yarn_data/hdfs/datanode/current/BP-1408773897-172.17.1.1-1400769841207/current/finalized
>> [hadoop@thor1 finalized]$ du -sh subdir10/
>> 80G subdir10/
>>
>> So could I move subdir10 to the same location under /data/data3?
>>
>> Thanks,
>> Brian
>>
>>
>> Brian C. Huffman System Administrator ET International, Inc.On 10/8/14,
>> 4:44 AM, Aitor Cedres wrote:
>>
>>
>> Hi Brian,
>>
>> Hadoop does not balance the disks within a DataNode. If you ran out of
>> space and then add additional disks, you should shutdown the DataNode and
>> move manually a few files to the new disk.
>>
>> Regards,
>>
>> Aitor Cedrés
>>
>> On 6 October 2014 14:46, Brian C. Huffman <bhuffman@etinternational.com
>> > wrote:
>>
>>> All,
>>>
>>> I have a small hadoop cluster (2.5.0) with 4 datanodes and 3 data disks
>>> per node. Lately some of the volumes have been filling, but instead of
>>> moving to other configured volumes that *have* free space, it's giving
>>> errors in the datanode logs:
>>> 2014-10-03 11:52:44,989 ERROR
>>> org.apache.hadoop.hdfs.server.datanode.DataNode:
>>> thor2.xmen.eti:50010:DataXceiver error processing WRITE_BLOCK
>>> operation src: /172.17.1.3:35412 dst: /172.17.1.2:50010
>>> java.io.IOException: No space left on device
>>> at java.io.FileOutputStream.writeBytes(Native Method)
>>> at java.io.FileOutputStream.write(FileOutputStream.java:345)
>>> at
>>> org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:592)
>>> at
>>> org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:734)
>>> at
>>> org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:741)
>>> at
>>> org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:124)
>>> at
>>> org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:71)
>>> at
>>> org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:234)
>>> at java.lang.Thread.run(Thread.java:745)
>>>
>>> Unfortunately it's continuing to try to write and when it fails, it's
>>> passing the exception to the client.
>>>
>>> I did a restart and then it seemed to figure out that it should move to
>>> the next volume.
>>>
>>> Any suggestions to keep this from happening in the future?
>>>
>>> Also - could it be an issue that I have a small amount of non-HDFS data
>>> on those volumes?
>>>
>>> Thanks,
>>> Brian
>>>
>>>
>>
>>
>
>
Re: Datanode volume full, but not moving to free volume
Posted by "Brian C. Huffman" <bh...@etinternational.com>.
Hmmm.. It seems that there's only one block pool per disk. So that
won't help me. :-(
Also, I see the blockpool directory names are all the same. Is that
expected? So even if I put a larger disk in, I couldn't consolidate the
smaller disk's blockpool directories?
[hadoop@thor1 current]$ ls
/data/data1/hadoop/yarn_data/hdfs/datanode/current
BP-1408773897-172.17.1.1-1400769841207 VERSION
[hadoop@thor1 current]$ ls
/data/data2/hadoop/yarn_data/hdfs/datanode/current
BP-1408773897-172.17.1.1-1400769841207 VERSION
[hadoop@thor1 current]$ ls
/data/data3/hadoop/yarn_data/hdfs/datanode/current
BP-1408773897-172.17.1.1-1400769841207 VERSION
Regards,
Brian
On 10/8/14, 7:14 AM, Aitor Cedres wrote:
>
> Hi Brian,
>
> I would try to move the Block Pools directories
> (BP-1408773897-172.17.1.1-1400769841207). You must shutdown your
> DataNode process before doing this operation.
>
> Regards,
>
> Aitor Cedrés
>
> On 8 October 2014 11:46, Brian C. Huffman
> <bhuffman@etinternational.com <ma...@etinternational.com>>
> wrote:
>
> Can I move a whole subdir? Or does it have to be individual block
> files / metadata?
>
> For example, I see this:
> [hadoop@thor1 finalized]$ pwd
> /data/data2/hadoop/yarn_data/hdfs/datanode/current/BP-1408773897-172.17.1.1-1400769841207/current/finalized
> [hadoop@thor1 finalized]$ du -sh subdir10/
> 80G subdir10/
>
> So could I move subdir10 to the same location under /data/data3?
>
> Thanks,
> Brian
>
>
> Brian C. Huffman System Administrator ET International, Inc.On
> 10/8/14, 4:44 AM, Aitor Cedres wrote:
>>
>> Hi Brian,
>>
>> Hadoop does not balance the disks within a DataNode. If you ran
>> out of space and then add additional disks, you should shutdown
>> the DataNode and move manually a few files to the new disk.
>>
>> Regards,
>>
>> Aitor Cedrés
>>
>> On 6 October 2014 14:46, Brian C. Huffman
>> <bhuffman@etinternational.com
>> <ma...@etinternational.com>> wrote:
>>
>> All,
>>
>> I have a small hadoop cluster (2.5.0) with 4 datanodes and 3
>> data disks per node. Lately some of the volumes have been
>> filling, but instead of moving to other configured volumes
>> that *have* free space, it's giving errors in the datanode logs:
>> 2014-10-03 11:52:44,989 ERROR
>> org.apache.hadoop.hdfs.server.datanode.DataNode:
>> thor2.xmen.eti:50010:DataXceiver error processing WRITE_BLOCK
>> operation src: /172.17.1.3:35412 <http://172.17.1.3:35412>
>> dst: /172.17.1.2:50010 <http://172.17.1.2:50010>
>> java.io.IOException: No space left on device
>> at java.io.FileOutputStream.writeBytes(Native Method)
>> at java.io.FileOutputStream.write(FileOutputStream.java:345)
>> at
>> org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:592)
>> at
>> org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:734)
>> at
>> org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:741)
>> at
>> org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:124)
>> at
>> org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:71)
>> at
>> org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:234)
>> at java.lang.Thread.run(Thread.java:745)
>>
>> Unfortunately it's continuing to try to write and when it
>> fails, it's passing the exception to the client.
>>
>> I did a restart and then it seemed to figure out that it
>> should move to the next volume.
>>
>> Any suggestions to keep this from happening in the future?
>>
>> Also - could it be an issue that I have a small amount of
>> non-HDFS data on those volumes?
>>
>> Thanks,
>> Brian
>>
>>
>
>
Re: Datanode volume full, but not moving to free volume
Posted by "Brian C. Huffman" <bh...@etinternational.com>.
Hmmm.. It seems that there's only one block pool per disk. So that
won't help me. :-(
Also, I see the blockpool directory names are all the same. Is that
expected? So even if I put a larger disk in, I couldn't consolidate the
smaller disk's blockpool directories?
[hadoop@thor1 current]$ ls
/data/data1/hadoop/yarn_data/hdfs/datanode/current
BP-1408773897-172.17.1.1-1400769841207 VERSION
[hadoop@thor1 current]$ ls
/data/data2/hadoop/yarn_data/hdfs/datanode/current
BP-1408773897-172.17.1.1-1400769841207 VERSION
[hadoop@thor1 current]$ ls
/data/data3/hadoop/yarn_data/hdfs/datanode/current
BP-1408773897-172.17.1.1-1400769841207 VERSION
Regards,
Brian
On 10/8/14, 7:14 AM, Aitor Cedres wrote:
>
> Hi Brian,
>
> I would try to move the Block Pools directories
> (BP-1408773897-172.17.1.1-1400769841207). You must shutdown your
> DataNode process before doing this operation.
>
> Regards,
>
> Aitor Cedrés
>
> On 8 October 2014 11:46, Brian C. Huffman
> <bhuffman@etinternational.com <ma...@etinternational.com>>
> wrote:
>
> Can I move a whole subdir? Or does it have to be individual block
> files / metadata?
>
> For example, I see this:
> [hadoop@thor1 finalized]$ pwd
> /data/data2/hadoop/yarn_data/hdfs/datanode/current/BP-1408773897-172.17.1.1-1400769841207/current/finalized
> [hadoop@thor1 finalized]$ du -sh subdir10/
> 80G subdir10/
>
> So could I move subdir10 to the same location under /data/data3?
>
> Thanks,
> Brian
>
>
> Brian C. Huffman System Administrator ET International, Inc.On
> 10/8/14, 4:44 AM, Aitor Cedres wrote:
>>
>> Hi Brian,
>>
>> Hadoop does not balance the disks within a DataNode. If you ran
>> out of space and then add additional disks, you should shutdown
>> the DataNode and move manually a few files to the new disk.
>>
>> Regards,
>>
>> Aitor Cedrés
>>
>> On 6 October 2014 14:46, Brian C. Huffman
>> <bhuffman@etinternational.com
>> <ma...@etinternational.com>> wrote:
>>
>> All,
>>
>> I have a small hadoop cluster (2.5.0) with 4 datanodes and 3
>> data disks per node. Lately some of the volumes have been
>> filling, but instead of moving to other configured volumes
>> that *have* free space, it's giving errors in the datanode logs:
>> 2014-10-03 11:52:44,989 ERROR
>> org.apache.hadoop.hdfs.server.datanode.DataNode:
>> thor2.xmen.eti:50010:DataXceiver error processing WRITE_BLOCK
>> operation src: /172.17.1.3:35412 <http://172.17.1.3:35412>
>> dst: /172.17.1.2:50010 <http://172.17.1.2:50010>
>> java.io.IOException: No space left on device
>> at java.io.FileOutputStream.writeBytes(Native Method)
>> at java.io.FileOutputStream.write(FileOutputStream.java:345)
>> at
>> org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:592)
>> at
>> org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:734)
>> at
>> org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:741)
>> at
>> org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:124)
>> at
>> org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:71)
>> at
>> org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:234)
>> at java.lang.Thread.run(Thread.java:745)
>>
>> Unfortunately it's continuing to try to write and when it
>> fails, it's passing the exception to the client.
>>
>> I did a restart and then it seemed to figure out that it
>> should move to the next volume.
>>
>> Any suggestions to keep this from happening in the future?
>>
>> Also - could it be an issue that I have a small amount of
>> non-HDFS data on those volumes?
>>
>> Thanks,
>> Brian
>>
>>
>
>
Re: Datanode volume full, but not moving to free volume
Posted by "Brian C. Huffman" <bh...@etinternational.com>.
Hmmm.. It seems that there's only one block pool per disk. So that
won't help me. :-(
Also, I see the blockpool directory names are all the same. Is that
expected? So even if I put a larger disk in, I couldn't consolidate the
smaller disk's blockpool directories?
[hadoop@thor1 current]$ ls
/data/data1/hadoop/yarn_data/hdfs/datanode/current
BP-1408773897-172.17.1.1-1400769841207 VERSION
[hadoop@thor1 current]$ ls
/data/data2/hadoop/yarn_data/hdfs/datanode/current
BP-1408773897-172.17.1.1-1400769841207 VERSION
[hadoop@thor1 current]$ ls
/data/data3/hadoop/yarn_data/hdfs/datanode/current
BP-1408773897-172.17.1.1-1400769841207 VERSION
Regards,
Brian
On 10/8/14, 7:14 AM, Aitor Cedres wrote:
>
> Hi Brian,
>
> I would try to move the Block Pools directories
> (BP-1408773897-172.17.1.1-1400769841207). You must shutdown your
> DataNode process before doing this operation.
>
> Regards,
>
> Aitor Cedrés
>
> On 8 October 2014 11:46, Brian C. Huffman
> <bhuffman@etinternational.com <ma...@etinternational.com>>
> wrote:
>
> Can I move a whole subdir? Or does it have to be individual block
> files / metadata?
>
> For example, I see this:
> [hadoop@thor1 finalized]$ pwd
> /data/data2/hadoop/yarn_data/hdfs/datanode/current/BP-1408773897-172.17.1.1-1400769841207/current/finalized
> [hadoop@thor1 finalized]$ du -sh subdir10/
> 80G subdir10/
>
> So could I move subdir10 to the same location under /data/data3?
>
> Thanks,
> Brian
>
>
> Brian C. Huffman System Administrator ET International, Inc.On
> 10/8/14, 4:44 AM, Aitor Cedres wrote:
>>
>> Hi Brian,
>>
>> Hadoop does not balance the disks within a DataNode. If you ran
>> out of space and then add additional disks, you should shutdown
>> the DataNode and move manually a few files to the new disk.
>>
>> Regards,
>>
>> Aitor Cedrés
>>
>> On 6 October 2014 14:46, Brian C. Huffman
>> <bhuffman@etinternational.com
>> <ma...@etinternational.com>> wrote:
>>
>> All,
>>
>> I have a small hadoop cluster (2.5.0) with 4 datanodes and 3
>> data disks per node. Lately some of the volumes have been
>> filling, but instead of moving to other configured volumes
>> that *have* free space, it's giving errors in the datanode logs:
>> 2014-10-03 11:52:44,989 ERROR
>> org.apache.hadoop.hdfs.server.datanode.DataNode:
>> thor2.xmen.eti:50010:DataXceiver error processing WRITE_BLOCK
>> operation src: /172.17.1.3:35412 <http://172.17.1.3:35412>
>> dst: /172.17.1.2:50010 <http://172.17.1.2:50010>
>> java.io.IOException: No space left on device
>> at java.io.FileOutputStream.writeBytes(Native Method)
>> at java.io.FileOutputStream.write(FileOutputStream.java:345)
>> at
>> org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:592)
>> at
>> org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:734)
>> at
>> org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:741)
>> at
>> org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:124)
>> at
>> org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:71)
>> at
>> org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:234)
>> at java.lang.Thread.run(Thread.java:745)
>>
>> Unfortunately it's continuing to try to write and when it
>> fails, it's passing the exception to the client.
>>
>> I did a restart and then it seemed to figure out that it
>> should move to the next volume.
>>
>> Any suggestions to keep this from happening in the future?
>>
>> Also - could it be an issue that I have a small amount of
>> non-HDFS data on those volumes?
>>
>> Thanks,
>> Brian
>>
>>
>
>
Re: Datanode volume full, but not moving to free volume
Posted by "Brian C. Huffman" <bh...@etinternational.com>.
Hmmm.. It seems that there's only one block pool per disk. So that
won't help me. :-(
Also, I see the blockpool directory names are all the same. Is that
expected? So even if I put a larger disk in, I couldn't consolidate the
smaller disk's blockpool directories?
[hadoop@thor1 current]$ ls
/data/data1/hadoop/yarn_data/hdfs/datanode/current
BP-1408773897-172.17.1.1-1400769841207 VERSION
[hadoop@thor1 current]$ ls
/data/data2/hadoop/yarn_data/hdfs/datanode/current
BP-1408773897-172.17.1.1-1400769841207 VERSION
[hadoop@thor1 current]$ ls
/data/data3/hadoop/yarn_data/hdfs/datanode/current
BP-1408773897-172.17.1.1-1400769841207 VERSION
Regards,
Brian
On 10/8/14, 7:14 AM, Aitor Cedres wrote:
>
> Hi Brian,
>
> I would try to move the Block Pools directories
> (BP-1408773897-172.17.1.1-1400769841207). You must shutdown your
> DataNode process before doing this operation.
>
> Regards,
>
> Aitor Cedrés
>
> On 8 October 2014 11:46, Brian C. Huffman
> <bhuffman@etinternational.com <ma...@etinternational.com>>
> wrote:
>
> Can I move a whole subdir? Or does it have to be individual block
> files / metadata?
>
> For example, I see this:
> [hadoop@thor1 finalized]$ pwd
> /data/data2/hadoop/yarn_data/hdfs/datanode/current/BP-1408773897-172.17.1.1-1400769841207/current/finalized
> [hadoop@thor1 finalized]$ du -sh subdir10/
> 80G subdir10/
>
> So could I move subdir10 to the same location under /data/data3?
>
> Thanks,
> Brian
>
>
> Brian C. Huffman System Administrator ET International, Inc.On
> 10/8/14, 4:44 AM, Aitor Cedres wrote:
>>
>> Hi Brian,
>>
>> Hadoop does not balance the disks within a DataNode. If you ran
>> out of space and then add additional disks, you should shutdown
>> the DataNode and move manually a few files to the new disk.
>>
>> Regards,
>>
>> Aitor Cedrés
>>
>> On 6 October 2014 14:46, Brian C. Huffman
>> <bhuffman@etinternational.com
>> <ma...@etinternational.com>> wrote:
>>
>> All,
>>
>> I have a small hadoop cluster (2.5.0) with 4 datanodes and 3
>> data disks per node. Lately some of the volumes have been
>> filling, but instead of moving to other configured volumes
>> that *have* free space, it's giving errors in the datanode logs:
>> 2014-10-03 11:52:44,989 ERROR
>> org.apache.hadoop.hdfs.server.datanode.DataNode:
>> thor2.xmen.eti:50010:DataXceiver error processing WRITE_BLOCK
>> operation src: /172.17.1.3:35412 <http://172.17.1.3:35412>
>> dst: /172.17.1.2:50010 <http://172.17.1.2:50010>
>> java.io.IOException: No space left on device
>> at java.io.FileOutputStream.writeBytes(Native Method)
>> at java.io.FileOutputStream.write(FileOutputStream.java:345)
>> at
>> org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:592)
>> at
>> org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:734)
>> at
>> org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:741)
>> at
>> org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:124)
>> at
>> org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:71)
>> at
>> org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:234)
>> at java.lang.Thread.run(Thread.java:745)
>>
>> Unfortunately it's continuing to try to write and when it
>> fails, it's passing the exception to the client.
>>
>> I did a restart and then it seemed to figure out that it
>> should move to the next volume.
>>
>> Any suggestions to keep this from happening in the future?
>>
>> Also - could it be an issue that I have a small amount of
>> non-HDFS data on those volumes?
>>
>> Thanks,
>> Brian
>>
>>
>
>
Re: Datanode volume full, but not moving to free volume
Posted by Aitor Cedres <ac...@pivotal.io>.
Hi Brian,
I would try to move the Block Pools directories
(BP-1408773897-172.17.1.1-1400769841207). You must shutdown your DataNode
process before doing this operation.
Regards,
Aitor Cedrés
On 8 October 2014 11:46, Brian C. Huffman <bh...@etinternational.com>
wrote:
> Can I move a whole subdir? Or does it have to be individual block files
> / metadata?
>
> For example, I see this:
> [hadoop@thor1 finalized]$ pwd
>
> /data/data2/hadoop/yarn_data/hdfs/datanode/current/BP-1408773897-172.17.1.1-1400769841207/current/finalized
> [hadoop@thor1 finalized]$ du -sh subdir10/
> 80G subdir10/
>
> So could I move subdir10 to the same location under /data/data3?
>
> Thanks,
> Brian
>
>
> Brian C. Huffman System Administrator ET International, Inc.On 10/8/14,
> 4:44 AM, Aitor Cedres wrote:
>
>
> Hi Brian,
>
> Hadoop does not balance the disks within a DataNode. If you ran out of
> space and then add additional disks, you should shutdown the DataNode and
> move manually a few files to the new disk.
>
> Regards,
>
> Aitor Cedrés
>
> On 6 October 2014 14:46, Brian C. Huffman <bh...@etinternational.com>
> wrote:
>
>> All,
>>
>> I have a small hadoop cluster (2.5.0) with 4 datanodes and 3 data disks
>> per node. Lately some of the volumes have been filling, but instead of
>> moving to other configured volumes that *have* free space, it's giving
>> errors in the datanode logs:
>> 2014-10-03 11:52:44,989 ERROR
>> org.apache.hadoop.hdfs.server.datanode.DataNode:
>> thor2.xmen.eti:50010:DataXceiver error processing WRITE_BLOCK
>> operation src: /172.17.1.3:35412 dst: /172.17.1.2:50010
>> java.io.IOException: No space left on device
>> at java.io.FileOutputStream.writeBytes(Native Method)
>> at java.io.FileOutputStream.write(FileOutputStream.java:345)
>> at
>> org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:592)
>> at
>> org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:734)
>> at
>> org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:741)
>> at
>> org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:124)
>> at
>> org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:71)
>> at
>> org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:234)
>> at java.lang.Thread.run(Thread.java:745)
>>
>> Unfortunately it's continuing to try to write and when it fails, it's
>> passing the exception to the client.
>>
>> I did a restart and then it seemed to figure out that it should move to
>> the next volume.
>>
>> Any suggestions to keep this from happening in the future?
>>
>> Also - could it be an issue that I have a small amount of non-HDFS data
>> on those volumes?
>>
>> Thanks,
>> Brian
>>
>>
>
>
Re: Datanode volume full, but not moving to free volume
Posted by Aitor Cedres <ac...@pivotal.io>.
Hi Brian,
I would try to move the Block Pools directories
(BP-1408773897-172.17.1.1-1400769841207). You must shutdown your DataNode
process before doing this operation.
Regards,
Aitor Cedrés
On 8 October 2014 11:46, Brian C. Huffman <bh...@etinternational.com>
wrote:
> Can I move a whole subdir? Or does it have to be individual block files
> / metadata?
>
> For example, I see this:
> [hadoop@thor1 finalized]$ pwd
>
> /data/data2/hadoop/yarn_data/hdfs/datanode/current/BP-1408773897-172.17.1.1-1400769841207/current/finalized
> [hadoop@thor1 finalized]$ du -sh subdir10/
> 80G subdir10/
>
> So could I move subdir10 to the same location under /data/data3?
>
> Thanks,
> Brian
>
>
> Brian C. Huffman System Administrator ET International, Inc.On 10/8/14,
> 4:44 AM, Aitor Cedres wrote:
>
>
> Hi Brian,
>
> Hadoop does not balance the disks within a DataNode. If you ran out of
> space and then add additional disks, you should shutdown the DataNode and
> move manually a few files to the new disk.
>
> Regards,
>
> Aitor Cedrés
>
> On 6 October 2014 14:46, Brian C. Huffman <bh...@etinternational.com>
> wrote:
>
>> All,
>>
>> I have a small hadoop cluster (2.5.0) with 4 datanodes and 3 data disks
>> per node. Lately some of the volumes have been filling, but instead of
>> moving to other configured volumes that *have* free space, it's giving
>> errors in the datanode logs:
>> 2014-10-03 11:52:44,989 ERROR
>> org.apache.hadoop.hdfs.server.datanode.DataNode:
>> thor2.xmen.eti:50010:DataXceiver error processing WRITE_BLOCK
>> operation src: /172.17.1.3:35412 dst: /172.17.1.2:50010
>> java.io.IOException: No space left on device
>> at java.io.FileOutputStream.writeBytes(Native Method)
>> at java.io.FileOutputStream.write(FileOutputStream.java:345)
>> at
>> org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:592)
>> at
>> org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:734)
>> at
>> org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:741)
>> at
>> org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:124)
>> at
>> org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:71)
>> at
>> org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:234)
>> at java.lang.Thread.run(Thread.java:745)
>>
>> Unfortunately it's continuing to try to write and when it fails, it's
>> passing the exception to the client.
>>
>> I did a restart and then it seemed to figure out that it should move to
>> the next volume.
>>
>> Any suggestions to keep this from happening in the future?
>>
>> Also - could it be an issue that I have a small amount of non-HDFS data
>> on those volumes?
>>
>> Thanks,
>> Brian
>>
>>
>
>
Re: Datanode volume full, but not moving to free volume
Posted by Aitor Cedres <ac...@pivotal.io>.
Hi Brian,
I would try to move the Block Pools directories
(BP-1408773897-172.17.1.1-1400769841207). You must shutdown your DataNode
process before doing this operation.
Regards,
Aitor Cedrés
On 8 October 2014 11:46, Brian C. Huffman <bh...@etinternational.com>
wrote:
> Can I move a whole subdir? Or does it have to be individual block files
> / metadata?
>
> For example, I see this:
> [hadoop@thor1 finalized]$ pwd
>
> /data/data2/hadoop/yarn_data/hdfs/datanode/current/BP-1408773897-172.17.1.1-1400769841207/current/finalized
> [hadoop@thor1 finalized]$ du -sh subdir10/
> 80G subdir10/
>
> So could I move subdir10 to the same location under /data/data3?
>
> Thanks,
> Brian
>
>
> Brian C. Huffman System Administrator ET International, Inc.On 10/8/14,
> 4:44 AM, Aitor Cedres wrote:
>
>
> Hi Brian,
>
> Hadoop does not balance the disks within a DataNode. If you ran out of
> space and then add additional disks, you should shutdown the DataNode and
> move manually a few files to the new disk.
>
> Regards,
>
> Aitor Cedrés
>
> On 6 October 2014 14:46, Brian C. Huffman <bh...@etinternational.com>
> wrote:
>
>> All,
>>
>> I have a small hadoop cluster (2.5.0) with 4 datanodes and 3 data disks
>> per node. Lately some of the volumes have been filling, but instead of
>> moving to other configured volumes that *have* free space, it's giving
>> errors in the datanode logs:
>> 2014-10-03 11:52:44,989 ERROR
>> org.apache.hadoop.hdfs.server.datanode.DataNode:
>> thor2.xmen.eti:50010:DataXceiver error processing WRITE_BLOCK
>> operation src: /172.17.1.3:35412 dst: /172.17.1.2:50010
>> java.io.IOException: No space left on device
>> at java.io.FileOutputStream.writeBytes(Native Method)
>> at java.io.FileOutputStream.write(FileOutputStream.java:345)
>> at
>> org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:592)
>> at
>> org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:734)
>> at
>> org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:741)
>> at
>> org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:124)
>> at
>> org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:71)
>> at
>> org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:234)
>> at java.lang.Thread.run(Thread.java:745)
>>
>> Unfortunately it's continuing to try to write and when it fails, it's
>> passing the exception to the client.
>>
>> I did a restart and then it seemed to figure out that it should move to
>> the next volume.
>>
>> Any suggestions to keep this from happening in the future?
>>
>> Also - could it be an issue that I have a small amount of non-HDFS data
>> on those volumes?
>>
>> Thanks,
>> Brian
>>
>>
>
>
Re: Datanode volume full, but not moving to free volume
Posted by Aitor Cedres <ac...@pivotal.io>.
Hi Brian,
I would try to move the Block Pools directories
(BP-1408773897-172.17.1.1-1400769841207). You must shutdown your DataNode
process before doing this operation.
Regards,
Aitor Cedrés
On 8 October 2014 11:46, Brian C. Huffman <bh...@etinternational.com>
wrote:
> Can I move a whole subdir? Or does it have to be individual block files
> / metadata?
>
> For example, I see this:
> [hadoop@thor1 finalized]$ pwd
>
> /data/data2/hadoop/yarn_data/hdfs/datanode/current/BP-1408773897-172.17.1.1-1400769841207/current/finalized
> [hadoop@thor1 finalized]$ du -sh subdir10/
> 80G subdir10/
>
> So could I move subdir10 to the same location under /data/data3?
>
> Thanks,
> Brian
>
>
> Brian C. Huffman System Administrator ET International, Inc.On 10/8/14,
> 4:44 AM, Aitor Cedres wrote:
>
>
> Hi Brian,
>
> Hadoop does not balance the disks within a DataNode. If you ran out of
> space and then add additional disks, you should shutdown the DataNode and
> move manually a few files to the new disk.
>
> Regards,
>
> Aitor Cedrés
>
> On 6 October 2014 14:46, Brian C. Huffman <bh...@etinternational.com>
> wrote:
>
>> All,
>>
>> I have a small hadoop cluster (2.5.0) with 4 datanodes and 3 data disks
>> per node. Lately some of the volumes have been filling, but instead of
>> moving to other configured volumes that *have* free space, it's giving
>> errors in the datanode logs:
>> 2014-10-03 11:52:44,989 ERROR
>> org.apache.hadoop.hdfs.server.datanode.DataNode:
>> thor2.xmen.eti:50010:DataXceiver error processing WRITE_BLOCK
>> operation src: /172.17.1.3:35412 dst: /172.17.1.2:50010
>> java.io.IOException: No space left on device
>> at java.io.FileOutputStream.writeBytes(Native Method)
>> at java.io.FileOutputStream.write(FileOutputStream.java:345)
>> at
>> org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:592)
>> at
>> org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:734)
>> at
>> org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:741)
>> at
>> org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:124)
>> at
>> org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:71)
>> at
>> org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:234)
>> at java.lang.Thread.run(Thread.java:745)
>>
>> Unfortunately it's continuing to try to write and when it fails, it's
>> passing the exception to the client.
>>
>> I did a restart and then it seemed to figure out that it should move to
>> the next volume.
>>
>> Any suggestions to keep this from happening in the future?
>>
>> Also - could it be an issue that I have a small amount of non-HDFS data
>> on those volumes?
>>
>> Thanks,
>> Brian
>>
>>
>
>
Re: Datanode volume full, but not moving to free volume
Posted by "Brian C. Huffman" <bh...@etinternational.com>.
Can I move a whole subdir? Or does it have to be individual block files
/ metadata?
For example, I see this:
[hadoop@thor1 finalized]$ pwd
/data/data2/hadoop/yarn_data/hdfs/datanode/current/BP-1408773897-172.17.1.1-1400769841207/current/finalized
[hadoop@thor1 finalized]$ du -sh subdir10/
80G subdir10/
So could I move subdir10 to the same location under /data/data3?
Thanks,
Brian
Brian C. Huffman System Administrator ET International, Inc.On 10/8/14,
4:44 AM, Aitor Cedres wrote:
>
> Hi Brian,
>
> Hadoop does not balance the disks within a DataNode. If you ran out of
> space and then add additional disks, you should shutdown the DataNode
> and move manually a few files to the new disk.
>
> Regards,
>
> Aitor Cedrés
>
> On 6 October 2014 14:46, Brian C. Huffman
> <bhuffman@etinternational.com <ma...@etinternational.com>>
> wrote:
>
> All,
>
> I have a small hadoop cluster (2.5.0) with 4 datanodes and 3 data
> disks per node. Lately some of the volumes have been filling, but
> instead of moving to other configured volumes that *have* free
> space, it's giving errors in the datanode logs:
> 2014-10-03 11:52:44,989 ERROR
> org.apache.hadoop.hdfs.server.datanode.DataNode:
> thor2.xmen.eti:50010:DataXceiver error processing WRITE_BLOCK
> operation src: /172.17.1.3:35412 <http://172.17.1.3:35412> dst:
> /172.17.1.2:50010 <http://172.17.1.2:50010>
> java.io.IOException: No space left on device
> at java.io.FileOutputStream.writeBytes(Native Method)
> at java.io.FileOutputStream.write(FileOutputStream.java:345)
> at
> org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:592)
> at
> org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:734)
> at
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:741)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:124)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:71)
> at
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:234)
> at java.lang.Thread.run(Thread.java:745)
>
> Unfortunately it's continuing to try to write and when it fails,
> it's passing the exception to the client.
>
> I did a restart and then it seemed to figure out that it should
> move to the next volume.
>
> Any suggestions to keep this from happening in the future?
>
> Also - could it be an issue that I have a small amount of non-HDFS
> data on those volumes?
>
> Thanks,
> Brian
>
>
Re: Datanode volume full, but not moving to free volume
Posted by "Brian C. Huffman" <bh...@etinternational.com>.
Can I move a whole subdir? Or does it have to be individual block files
/ metadata?
For example, I see this:
[hadoop@thor1 finalized]$ pwd
/data/data2/hadoop/yarn_data/hdfs/datanode/current/BP-1408773897-172.17.1.1-1400769841207/current/finalized
[hadoop@thor1 finalized]$ du -sh subdir10/
80G subdir10/
So could I move subdir10 to the same location under /data/data3?
Thanks,
Brian
Brian C. Huffman System Administrator ET International, Inc.On 10/8/14,
4:44 AM, Aitor Cedres wrote:
>
> Hi Brian,
>
> Hadoop does not balance the disks within a DataNode. If you ran out of
> space and then add additional disks, you should shutdown the DataNode
> and move manually a few files to the new disk.
>
> Regards,
>
> Aitor Cedrés
>
> On 6 October 2014 14:46, Brian C. Huffman
> <bhuffman@etinternational.com <ma...@etinternational.com>>
> wrote:
>
> All,
>
> I have a small hadoop cluster (2.5.0) with 4 datanodes and 3 data
> disks per node. Lately some of the volumes have been filling, but
> instead of moving to other configured volumes that *have* free
> space, it's giving errors in the datanode logs:
> 2014-10-03 11:52:44,989 ERROR
> org.apache.hadoop.hdfs.server.datanode.DataNode:
> thor2.xmen.eti:50010:DataXceiver error processing WRITE_BLOCK
> operation src: /172.17.1.3:35412 <http://172.17.1.3:35412> dst:
> /172.17.1.2:50010 <http://172.17.1.2:50010>
> java.io.IOException: No space left on device
> at java.io.FileOutputStream.writeBytes(Native Method)
> at java.io.FileOutputStream.write(FileOutputStream.java:345)
> at
> org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:592)
> at
> org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:734)
> at
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:741)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:124)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:71)
> at
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:234)
> at java.lang.Thread.run(Thread.java:745)
>
> Unfortunately it's continuing to try to write and when it fails,
> it's passing the exception to the client.
>
> I did a restart and then it seemed to figure out that it should
> move to the next volume.
>
> Any suggestions to keep this from happening in the future?
>
> Also - could it be an issue that I have a small amount of non-HDFS
> data on those volumes?
>
> Thanks,
> Brian
>
>
Re: Datanode volume full, but not moving to free volume
Posted by "Brian C. Huffman" <bh...@etinternational.com>.
Can I move a whole subdir? Or does it have to be individual block files
/ metadata?
For example, I see this:
[hadoop@thor1 finalized]$ pwd
/data/data2/hadoop/yarn_data/hdfs/datanode/current/BP-1408773897-172.17.1.1-1400769841207/current/finalized
[hadoop@thor1 finalized]$ du -sh subdir10/
80G subdir10/
So could I move subdir10 to the same location under /data/data3?
Thanks,
Brian
Brian C. Huffman System Administrator ET International, Inc.On 10/8/14,
4:44 AM, Aitor Cedres wrote:
>
> Hi Brian,
>
> Hadoop does not balance the disks within a DataNode. If you ran out of
> space and then add additional disks, you should shutdown the DataNode
> and move manually a few files to the new disk.
>
> Regards,
>
> Aitor Cedrés
>
> On 6 October 2014 14:46, Brian C. Huffman
> <bhuffman@etinternational.com <ma...@etinternational.com>>
> wrote:
>
> All,
>
> I have a small hadoop cluster (2.5.0) with 4 datanodes and 3 data
> disks per node. Lately some of the volumes have been filling, but
> instead of moving to other configured volumes that *have* free
> space, it's giving errors in the datanode logs:
> 2014-10-03 11:52:44,989 ERROR
> org.apache.hadoop.hdfs.server.datanode.DataNode:
> thor2.xmen.eti:50010:DataXceiver error processing WRITE_BLOCK
> operation src: /172.17.1.3:35412 <http://172.17.1.3:35412> dst:
> /172.17.1.2:50010 <http://172.17.1.2:50010>
> java.io.IOException: No space left on device
> at java.io.FileOutputStream.writeBytes(Native Method)
> at java.io.FileOutputStream.write(FileOutputStream.java:345)
> at
> org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:592)
> at
> org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:734)
> at
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:741)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:124)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:71)
> at
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:234)
> at java.lang.Thread.run(Thread.java:745)
>
> Unfortunately it's continuing to try to write and when it fails,
> it's passing the exception to the client.
>
> I did a restart and then it seemed to figure out that it should
> move to the next volume.
>
> Any suggestions to keep this from happening in the future?
>
> Also - could it be an issue that I have a small amount of non-HDFS
> data on those volumes?
>
> Thanks,
> Brian
>
>
Re: Datanode volume full, but not moving to free volume
Posted by "Brian C. Huffman" <bh...@etinternational.com>.
Can I move a whole subdir? Or does it have to be individual block files
/ metadata?
For example, I see this:
[hadoop@thor1 finalized]$ pwd
/data/data2/hadoop/yarn_data/hdfs/datanode/current/BP-1408773897-172.17.1.1-1400769841207/current/finalized
[hadoop@thor1 finalized]$ du -sh subdir10/
80G subdir10/
So could I move subdir10 to the same location under /data/data3?
Thanks,
Brian
Brian C. Huffman System Administrator ET International, Inc.On 10/8/14,
4:44 AM, Aitor Cedres wrote:
>
> Hi Brian,
>
> Hadoop does not balance the disks within a DataNode. If you ran out of
> space and then add additional disks, you should shutdown the DataNode
> and move manually a few files to the new disk.
>
> Regards,
>
> Aitor Cedrés
>
> On 6 October 2014 14:46, Brian C. Huffman
> <bhuffman@etinternational.com <ma...@etinternational.com>>
> wrote:
>
> All,
>
> I have a small hadoop cluster (2.5.0) with 4 datanodes and 3 data
> disks per node. Lately some of the volumes have been filling, but
> instead of moving to other configured volumes that *have* free
> space, it's giving errors in the datanode logs:
> 2014-10-03 11:52:44,989 ERROR
> org.apache.hadoop.hdfs.server.datanode.DataNode:
> thor2.xmen.eti:50010:DataXceiver error processing WRITE_BLOCK
> operation src: /172.17.1.3:35412 <http://172.17.1.3:35412> dst:
> /172.17.1.2:50010 <http://172.17.1.2:50010>
> java.io.IOException: No space left on device
> at java.io.FileOutputStream.writeBytes(Native Method)
> at java.io.FileOutputStream.write(FileOutputStream.java:345)
> at
> org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:592)
> at
> org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:734)
> at
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:741)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:124)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:71)
> at
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:234)
> at java.lang.Thread.run(Thread.java:745)
>
> Unfortunately it's continuing to try to write and when it fails,
> it's passing the exception to the client.
>
> I did a restart and then it seemed to figure out that it should
> move to the next volume.
>
> Any suggestions to keep this from happening in the future?
>
> Also - could it be an issue that I have a small amount of non-HDFS
> data on those volumes?
>
> Thanks,
> Brian
>
>
Re: Datanode volume full, but not moving to free volume
Posted by Azuryy Yu <az...@gmail.com>.
Hi Brian,
Did you try set dfs.datanode.fsdataset.volume.choosing.policy to
org.apache.hadoop.hdfs.server.datanode.fsdataset.AvailableSpaceVolumeChoosingPolicy
then there are some other related options with this policy. you can google
it.
On Wed, Oct 8, 2014 at 4:44 PM, Aitor Cedres <ac...@pivotal.io> wrote:
>
> Hi Brian,
>
> Hadoop does not balance the disks within a DataNode. If you ran out of
> space and then add additional disks, you should shutdown the DataNode and
> move manually a few files to the new disk.
>
> Regards,
>
> Aitor Cedrés
>
>
> On 6 October 2014 14:46, Brian C. Huffman <bh...@etinternational.com>
> wrote:
>
>> All,
>>
>> I have a small hadoop cluster (2.5.0) with 4 datanodes and 3 data disks
>> per node. Lately some of the volumes have been filling, but instead of
>> moving to other configured volumes that *have* free space, it's giving
>> errors in the datanode logs:
>> 2014-10-03 11:52:44,989 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode:
>> thor2.xmen.eti:50010:DataXceiver error processing WRITE_BLOCK
>> operation src: /172.17.1.3:35412 dst: /172.17.1.2:50010
>> java.io.IOException: No space left on device
>> at java.io.FileOutputStream.writeBytes(Native Method)
>> at java.io.FileOutputStream.write(FileOutputStream.java:345)
>> at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.
>> receivePacket(BlockReceiver.java:592)
>> at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.
>> receiveBlock(BlockReceiver.java:734)
>> at org.apache.hadoop.hdfs.server.datanode.DataXceiver.
>> writeBlock(DataXceiver.java:741)
>> at org.apache.hadoop.hdfs.protocol.datatransfer.
>> Receiver.opWriteBlock(Receiver.java:124)
>> at org.apache.hadoop.hdfs.protocol.datatransfer.
>> Receiver.processOp(Receiver.java:71)
>> at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(
>> DataXceiver.java:234)
>> at java.lang.Thread.run(Thread.java:745)
>>
>> Unfortunately it's continuing to try to write and when it fails, it's
>> passing the exception to the client.
>>
>> I did a restart and then it seemed to figure out that it should move to
>> the next volume.
>>
>> Any suggestions to keep this from happening in the future?
>>
>> Also - could it be an issue that I have a small amount of non-HDFS data
>> on those volumes?
>>
>> Thanks,
>> Brian
>>
>>
>
Re: Datanode volume full, but not moving to free volume
Posted by Azuryy Yu <az...@gmail.com>.
Hi Brian,
Did you try set dfs.datanode.fsdataset.volume.choosing.policy to
org.apache.hadoop.hdfs.server.datanode.fsdataset.AvailableSpaceVolumeChoosingPolicy
then there are some other related options with this policy. you can google
it.
On Wed, Oct 8, 2014 at 4:44 PM, Aitor Cedres <ac...@pivotal.io> wrote:
>
> Hi Brian,
>
> Hadoop does not balance the disks within a DataNode. If you ran out of
> space and then add additional disks, you should shutdown the DataNode and
> move manually a few files to the new disk.
>
> Regards,
>
> Aitor Cedrés
>
>
> On 6 October 2014 14:46, Brian C. Huffman <bh...@etinternational.com>
> wrote:
>
>> All,
>>
>> I have a small hadoop cluster (2.5.0) with 4 datanodes and 3 data disks
>> per node. Lately some of the volumes have been filling, but instead of
>> moving to other configured volumes that *have* free space, it's giving
>> errors in the datanode logs:
>> 2014-10-03 11:52:44,989 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode:
>> thor2.xmen.eti:50010:DataXceiver error processing WRITE_BLOCK
>> operation src: /172.17.1.3:35412 dst: /172.17.1.2:50010
>> java.io.IOException: No space left on device
>> at java.io.FileOutputStream.writeBytes(Native Method)
>> at java.io.FileOutputStream.write(FileOutputStream.java:345)
>> at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.
>> receivePacket(BlockReceiver.java:592)
>> at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.
>> receiveBlock(BlockReceiver.java:734)
>> at org.apache.hadoop.hdfs.server.datanode.DataXceiver.
>> writeBlock(DataXceiver.java:741)
>> at org.apache.hadoop.hdfs.protocol.datatransfer.
>> Receiver.opWriteBlock(Receiver.java:124)
>> at org.apache.hadoop.hdfs.protocol.datatransfer.
>> Receiver.processOp(Receiver.java:71)
>> at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(
>> DataXceiver.java:234)
>> at java.lang.Thread.run(Thread.java:745)
>>
>> Unfortunately it's continuing to try to write and when it fails, it's
>> passing the exception to the client.
>>
>> I did a restart and then it seemed to figure out that it should move to
>> the next volume.
>>
>> Any suggestions to keep this from happening in the future?
>>
>> Also - could it be an issue that I have a small amount of non-HDFS data
>> on those volumes?
>>
>> Thanks,
>> Brian
>>
>>
>
Re: Datanode volume full, but not moving to free volume
Posted by Azuryy Yu <az...@gmail.com>.
Hi Brian,
Did you try set dfs.datanode.fsdataset.volume.choosing.policy to
org.apache.hadoop.hdfs.server.datanode.fsdataset.AvailableSpaceVolumeChoosingPolicy
then there are some other related options with this policy. you can google
it.
On Wed, Oct 8, 2014 at 4:44 PM, Aitor Cedres <ac...@pivotal.io> wrote:
>
> Hi Brian,
>
> Hadoop does not balance the disks within a DataNode. If you ran out of
> space and then add additional disks, you should shutdown the DataNode and
> move manually a few files to the new disk.
>
> Regards,
>
> Aitor Cedrés
>
>
> On 6 October 2014 14:46, Brian C. Huffman <bh...@etinternational.com>
> wrote:
>
>> All,
>>
>> I have a small hadoop cluster (2.5.0) with 4 datanodes and 3 data disks
>> per node. Lately some of the volumes have been filling, but instead of
>> moving to other configured volumes that *have* free space, it's giving
>> errors in the datanode logs:
>> 2014-10-03 11:52:44,989 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode:
>> thor2.xmen.eti:50010:DataXceiver error processing WRITE_BLOCK
>> operation src: /172.17.1.3:35412 dst: /172.17.1.2:50010
>> java.io.IOException: No space left on device
>> at java.io.FileOutputStream.writeBytes(Native Method)
>> at java.io.FileOutputStream.write(FileOutputStream.java:345)
>> at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.
>> receivePacket(BlockReceiver.java:592)
>> at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.
>> receiveBlock(BlockReceiver.java:734)
>> at org.apache.hadoop.hdfs.server.datanode.DataXceiver.
>> writeBlock(DataXceiver.java:741)
>> at org.apache.hadoop.hdfs.protocol.datatransfer.
>> Receiver.opWriteBlock(Receiver.java:124)
>> at org.apache.hadoop.hdfs.protocol.datatransfer.
>> Receiver.processOp(Receiver.java:71)
>> at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(
>> DataXceiver.java:234)
>> at java.lang.Thread.run(Thread.java:745)
>>
>> Unfortunately it's continuing to try to write and when it fails, it's
>> passing the exception to the client.
>>
>> I did a restart and then it seemed to figure out that it should move to
>> the next volume.
>>
>> Any suggestions to keep this from happening in the future?
>>
>> Also - could it be an issue that I have a small amount of non-HDFS data
>> on those volumes?
>>
>> Thanks,
>> Brian
>>
>>
>
Re: Datanode volume full, but not moving to free volume
Posted by Aitor Cedres <ac...@pivotal.io>.
Hi Brian,
Hadoop does not balance the disks within a DataNode. If you ran out of
space and then add additional disks, you should shutdown the DataNode and
move manually a few files to the new disk.
Regards,
Aitor Cedrés
On 6 October 2014 14:46, Brian C. Huffman <bh...@etinternational.com>
wrote:
> All,
>
> I have a small hadoop cluster (2.5.0) with 4 datanodes and 3 data disks
> per node. Lately some of the volumes have been filling, but instead of
> moving to other configured volumes that *have* free space, it's giving
> errors in the datanode logs:
> 2014-10-03 11:52:44,989 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode:
> thor2.xmen.eti:50010:DataXceiver error processing WRITE_BLOCK
> operation src: /172.17.1.3:35412 dst: /172.17.1.2:50010
> java.io.IOException: No space left on device
> at java.io.FileOutputStream.writeBytes(Native Method)
> at java.io.FileOutputStream.write(FileOutputStream.java:345)
> at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.
> receivePacket(BlockReceiver.java:592)
> at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.
> receiveBlock(BlockReceiver.java:734)
> at org.apache.hadoop.hdfs.server.datanode.DataXceiver.
> writeBlock(DataXceiver.java:741)
> at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(
> Receiver.java:124)
> at org.apache.hadoop.hdfs.protocol.datatransfer.
> Receiver.processOp(Receiver.java:71)
> at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(
> DataXceiver.java:234)
> at java.lang.Thread.run(Thread.java:745)
>
> Unfortunately it's continuing to try to write and when it fails, it's
> passing the exception to the client.
>
> I did a restart and then it seemed to figure out that it should move to
> the next volume.
>
> Any suggestions to keep this from happening in the future?
>
> Also - could it be an issue that I have a small amount of non-HDFS data on
> those volumes?
>
> Thanks,
> Brian
>
>
Re: Datanode volume full, but not moving to free volume
Posted by Aitor Cedres <ac...@pivotal.io>.
Hi Brian,
Hadoop does not balance the disks within a DataNode. If you ran out of
space and then add additional disks, you should shutdown the DataNode and
move manually a few files to the new disk.
Regards,
Aitor Cedrés
On 6 October 2014 14:46, Brian C. Huffman <bh...@etinternational.com>
wrote:
> All,
>
> I have a small hadoop cluster (2.5.0) with 4 datanodes and 3 data disks
> per node. Lately some of the volumes have been filling, but instead of
> moving to other configured volumes that *have* free space, it's giving
> errors in the datanode logs:
> 2014-10-03 11:52:44,989 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode:
> thor2.xmen.eti:50010:DataXceiver error processing WRITE_BLOCK
> operation src: /172.17.1.3:35412 dst: /172.17.1.2:50010
> java.io.IOException: No space left on device
> at java.io.FileOutputStream.writeBytes(Native Method)
> at java.io.FileOutputStream.write(FileOutputStream.java:345)
> at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.
> receivePacket(BlockReceiver.java:592)
> at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.
> receiveBlock(BlockReceiver.java:734)
> at org.apache.hadoop.hdfs.server.datanode.DataXceiver.
> writeBlock(DataXceiver.java:741)
> at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(
> Receiver.java:124)
> at org.apache.hadoop.hdfs.protocol.datatransfer.
> Receiver.processOp(Receiver.java:71)
> at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(
> DataXceiver.java:234)
> at java.lang.Thread.run(Thread.java:745)
>
> Unfortunately it's continuing to try to write and when it fails, it's
> passing the exception to the client.
>
> I did a restart and then it seemed to figure out that it should move to
> the next volume.
>
> Any suggestions to keep this from happening in the future?
>
> Also - could it be an issue that I have a small amount of non-HDFS data on
> those volumes?
>
> Thanks,
> Brian
>
>
Re: Datanode volume full, but not moving to free volume
Posted by Aitor Cedres <ac...@pivotal.io>.
Hi Brian,
Hadoop does not balance the disks within a DataNode. If you ran out of
space and then add additional disks, you should shutdown the DataNode and
move manually a few files to the new disk.
Regards,
Aitor Cedrés
On 6 October 2014 14:46, Brian C. Huffman <bh...@etinternational.com>
wrote:
> All,
>
> I have a small hadoop cluster (2.5.0) with 4 datanodes and 3 data disks
> per node. Lately some of the volumes have been filling, but instead of
> moving to other configured volumes that *have* free space, it's giving
> errors in the datanode logs:
> 2014-10-03 11:52:44,989 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode:
> thor2.xmen.eti:50010:DataXceiver error processing WRITE_BLOCK
> operation src: /172.17.1.3:35412 dst: /172.17.1.2:50010
> java.io.IOException: No space left on device
> at java.io.FileOutputStream.writeBytes(Native Method)
> at java.io.FileOutputStream.write(FileOutputStream.java:345)
> at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.
> receivePacket(BlockReceiver.java:592)
> at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.
> receiveBlock(BlockReceiver.java:734)
> at org.apache.hadoop.hdfs.server.datanode.DataXceiver.
> writeBlock(DataXceiver.java:741)
> at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(
> Receiver.java:124)
> at org.apache.hadoop.hdfs.protocol.datatransfer.
> Receiver.processOp(Receiver.java:71)
> at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(
> DataXceiver.java:234)
> at java.lang.Thread.run(Thread.java:745)
>
> Unfortunately it's continuing to try to write and when it fails, it's
> passing the exception to the client.
>
> I did a restart and then it seemed to figure out that it should move to
> the next volume.
>
> Any suggestions to keep this from happening in the future?
>
> Also - could it be an issue that I have a small amount of non-HDFS data on
> those volumes?
>
> Thanks,
> Brian
>
>
Re: Datanode volume full, but not moving to free volume
Posted by Aitor Cedres <ac...@pivotal.io>.
Hi Brian,
Hadoop does not balance the disks within a DataNode. If you ran out of
space and then add additional disks, you should shutdown the DataNode and
move manually a few files to the new disk.
Regards,
Aitor Cedrés
On 6 October 2014 14:46, Brian C. Huffman <bh...@etinternational.com>
wrote:
> All,
>
> I have a small hadoop cluster (2.5.0) with 4 datanodes and 3 data disks
> per node. Lately some of the volumes have been filling, but instead of
> moving to other configured volumes that *have* free space, it's giving
> errors in the datanode logs:
> 2014-10-03 11:52:44,989 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode:
> thor2.xmen.eti:50010:DataXceiver error processing WRITE_BLOCK
> operation src: /172.17.1.3:35412 dst: /172.17.1.2:50010
> java.io.IOException: No space left on device
> at java.io.FileOutputStream.writeBytes(Native Method)
> at java.io.FileOutputStream.write(FileOutputStream.java:345)
> at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.
> receivePacket(BlockReceiver.java:592)
> at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.
> receiveBlock(BlockReceiver.java:734)
> at org.apache.hadoop.hdfs.server.datanode.DataXceiver.
> writeBlock(DataXceiver.java:741)
> at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(
> Receiver.java:124)
> at org.apache.hadoop.hdfs.protocol.datatransfer.
> Receiver.processOp(Receiver.java:71)
> at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(
> DataXceiver.java:234)
> at java.lang.Thread.run(Thread.java:745)
>
> Unfortunately it's continuing to try to write and when it fails, it's
> passing the exception to the client.
>
> I did a restart and then it seemed to figure out that it should move to
> the next volume.
>
> Any suggestions to keep this from happening in the future?
>
> Also - could it be an issue that I have a small amount of non-HDFS data on
> those volumes?
>
> Thanks,
> Brian
>
>