You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by He Chen <ai...@gmail.com> on 2011/03/01 17:40:02 UTC
hadoop balancer
Hi all
I met a problem when I try to balance certain hdfs directory among the
clusters. For example, I have a directory "/user/xxx/", and there 100
blocks. I want to balance them among my 5 nodes clusters. Each node has 40
blocks (2 replicas). The problem is about transfer block from one datanode
to another. Actually, I followed the balancer's method. However, it always
waits for the response of destination datanode and halt. I attached the
code:
.................................................
Socket sock = new Socket();
DataOutputStream out = null;
DataInputStream in = null;
try{
sock.connect(NetUtils.createSocketAddr(
target.getName()), HdfsConstants.READ_TIMEOUT);
sock.setKeepAlive(true);
System.out.println(sock.isConnected());
out = new DataOutputStream( new BufferedOutputStream(
sock.getOutputStream(), FSConstants.BUFFER_SIZE));
out.writeShort(DataTransferProtocol.DATA_TRANSFER_VERSION);
out.writeByte(DataTransferProtocol.OP_REPLACE_BLOCK);
out.writeLong(block2move.getBlockId());
out.writeLong(block2move.getGenerationStamp());
Text.writeString(out, source.getStorageID());
System.out.println("Ready to move");
source.write(out);
System.out.println("Write to output Stream");
out.flush();
System.out.println("out has been flushed!");
in = new DataInputStream( new BufferedInputStream(
sock.getInputStream(), FSConstants.BUFFER_SIZE));
It stop here and wait for response.
short status = in.readShort();
System.out.println("Got the response from input stream!"+status);
if (status != DataTransferProtocol.OP_STATUS_SUCCESS) {
throw new IOException("block move is failed\t"+status);
}
} catch (IOException e) {
LOG.warn("Error moving block "+block2move.getBlockId()+
" from " + source.getName() + " to " +
target.getName() + " through " +
source.getName() +
": "+e.toString());
} finally {
IOUtils.closeStream(out);
IOUtils.closeStream(in);
IOUtils.closeSocket(sock);
}
..........................................
Any reply will be appreciated. Thank you in advance!
Chen
Re: hadoop balancer
Posted by He Chen <ai...@gmail.com>.
Thank you very much Icebergs.
I rewrite the balancer. Now, given a directory like "/user/foo/", I can
balance the blocks under this directory evenly to every node in the cluster.
Best wishes!
Chen
On Thu, Mar 3, 2011 at 11:14 PM, icebergs <hk...@gmail.com> wrote:
> try this command
> hadoop fs -setrep -R -w 2 xx
> maybe help
>
> 2011/3/2 He Chen <ai...@gmail.com>
>
> > Hi all
> >
> > I met a problem when I try to balance certain hdfs directory among the
> > clusters. For example, I have a directory "/user/xxx/", and there 100
> > blocks. I want to balance them among my 5 nodes clusters. Each node has
> 40
> > blocks (2 replicas). The problem is about transfer block from one
> datanode
> > to another. Actually, I followed the balancer's method. However, it
> always
> > waits for the response of destination datanode and halt. I attached the
> > code:
> > .................................................
> >
> > Socket sock = new Socket();
> >
> > DataOutputStream out = null;
> >
> > DataInputStream in = null;
> >
> > try{
> >
> > sock.connect(NetUtils.createSocketAddr(
> >
> > target.getName()), HdfsConstants.READ_TIMEOUT);
> >
> > sock.setKeepAlive(true);
> >
> > System.out.println(sock.isConnected());
> >
> > out = new DataOutputStream( new BufferedOutputStream(
> >
> > sock.getOutputStream(), FSConstants.BUFFER_SIZE));
> >
> > out.writeShort(DataTransferProtocol.DATA_TRANSFER_VERSION);
> >
> > out.writeByte(DataTransferProtocol.OP_REPLACE_BLOCK);
> >
> > out.writeLong(block2move.getBlockId());
> >
> > out.writeLong(block2move.getGenerationStamp());
> >
> > Text.writeString(out, source.getStorageID());
> >
> > System.out.println("Ready to move");
> >
> > source.write(out);
> >
> > System.out.println("Write to output Stream");
> >
> > out.flush();
> >
> > System.out.println("out has been flushed!");
> >
> > in = new DataInputStream( new BufferedInputStream(
> >
> > sock.getInputStream(), FSConstants.BUFFER_SIZE));
> >
> > It stop here and wait for response.
> >
> > short status = in.readShort();
> >
> > System.out.println("Got the response from input stream!"+status);
> >
> > if (status != DataTransferProtocol.OP_STATUS_SUCCESS) {
> >
> > throw new IOException("block move is failed\t"+status);
> >
> > }
> >
> >
> >
> > } catch (IOException e) {
> >
> > LOG.warn("Error moving block "+block2move.getBlockId()+
> >
> > " from " + source.getName() + " to " +
> >
> > target.getName() + " through " +
> >
> > source.getName() +
> >
> > ": "+e.toString());
> >
> >
> > } finally {
> >
> > IOUtils.closeStream(out);
> >
> > IOUtils.closeStream(in);
> >
> > IOUtils.closeSocket(sock);
> > }
> > ..........................................
> >
> > Any reply will be appreciated. Thank you in advance!
> >
> > Chen
> >
>
Re: hadoop balancer
Posted by icebergs <hk...@gmail.com>.
try this command
hadoop fs -setrep -R -w 2 xx
maybe help
2011/3/2 He Chen <ai...@gmail.com>
> Hi all
>
> I met a problem when I try to balance certain hdfs directory among the
> clusters. For example, I have a directory "/user/xxx/", and there 100
> blocks. I want to balance them among my 5 nodes clusters. Each node has 40
> blocks (2 replicas). The problem is about transfer block from one datanode
> to another. Actually, I followed the balancer's method. However, it always
> waits for the response of destination datanode and halt. I attached the
> code:
> .................................................
>
> Socket sock = new Socket();
>
> DataOutputStream out = null;
>
> DataInputStream in = null;
>
> try{
>
> sock.connect(NetUtils.createSocketAddr(
>
> target.getName()), HdfsConstants.READ_TIMEOUT);
>
> sock.setKeepAlive(true);
>
> System.out.println(sock.isConnected());
>
> out = new DataOutputStream( new BufferedOutputStream(
>
> sock.getOutputStream(), FSConstants.BUFFER_SIZE));
>
> out.writeShort(DataTransferProtocol.DATA_TRANSFER_VERSION);
>
> out.writeByte(DataTransferProtocol.OP_REPLACE_BLOCK);
>
> out.writeLong(block2move.getBlockId());
>
> out.writeLong(block2move.getGenerationStamp());
>
> Text.writeString(out, source.getStorageID());
>
> System.out.println("Ready to move");
>
> source.write(out);
>
> System.out.println("Write to output Stream");
>
> out.flush();
>
> System.out.println("out has been flushed!");
>
> in = new DataInputStream( new BufferedInputStream(
>
> sock.getInputStream(), FSConstants.BUFFER_SIZE));
>
> It stop here and wait for response.
>
> short status = in.readShort();
>
> System.out.println("Got the response from input stream!"+status);
>
> if (status != DataTransferProtocol.OP_STATUS_SUCCESS) {
>
> throw new IOException("block move is failed\t"+status);
>
> }
>
>
>
> } catch (IOException e) {
>
> LOG.warn("Error moving block "+block2move.getBlockId()+
>
> " from " + source.getName() + " to " +
>
> target.getName() + " through " +
>
> source.getName() +
>
> ": "+e.toString());
>
>
> } finally {
>
> IOUtils.closeStream(out);
>
> IOUtils.closeStream(in);
>
> IOUtils.closeSocket(sock);
> }
> ..........................................
>
> Any reply will be appreciated. Thank you in advance!
>
> Chen
>