You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-user@hadoop.apache.org by Mohammad Mustaqeem <3m...@gmail.com> on 2013/05/04 17:27:58 UTC
Block placement Policy
I have read somewhere that a user can specified his own ReplicaPlacementPolicy.
How can I specify my own ReplicaPlacementPolicy?
I you have any sample ReplicaPlacementPolicy, then please share it..
--
*With regards ---*
*Mohammad Mustaqeem*,
M.Tech (CSE)
MNNIT Allahabad
9026604270
Re: Block placement Policy
Posted by Rajat Sharma <fs...@gmail.com>.
Have a look at org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.
This is default placement policy, you should be able to extend this to
implement your own policy and set config parameter
"dfs.block.replicator.classname" to point to your class
On Sat, May 4, 2013 at 8:57 PM, Mohammad Mustaqeem
<3m...@gmail.com> wrote:
> I have read somewhere that a user can specified his own ReplicaPlacementPolicy.
> How can I specify my own ReplicaPlacementPolicy?
> I you have any sample ReplicaPlacementPolicy, then please share it..
>
> --
> *With regards ---*
> *Mohammad Mustaqeem*,
> M.Tech (CSE)
> MNNIT Allahabad
> 9026604270
Re: Block placement Policy
Posted by Rajat Sharma <fs...@gmail.com>.
Have a look at org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.
This is default placement policy, you should be able to extend this to
implement your own policy and set config parameter
"dfs.block.replicator.classname" to point to your class
On Sat, May 4, 2013 at 8:57 PM, Mohammad Mustaqeem
<3m...@gmail.com> wrote:
> I have read somewhere that a user can specified his own ReplicaPlacementPolicy.
> How can I specify my own ReplicaPlacementPolicy?
> I you have any sample ReplicaPlacementPolicy, then please share it..
>
> --
> *With regards ---*
> *Mohammad Mustaqeem*,
> M.Tech (CSE)
> MNNIT Allahabad
> 9026604270
Re: Block placement Policy
Posted by Mohammad Mustaqeem <3m...@gmail.com>.
Do you know, how to use it?
Sorry, I am very new :(
--
*With regards ---*
*Mohammad Mustaqeem*,
M.Tech (CSE)
MNNIT Allahabad
9026604270
Re: Block placement Policy
Posted by Mohammad Mustaqeem <3m...@gmail.com>.
Do you know, how to use it?
Sorry, I am very new :(
--
*With regards ---*
*Mohammad Mustaqeem*,
M.Tech (CSE)
MNNIT Allahabad
9026604270
RE: Block placement Policy
Posted by Ivan Mitic <iv...@microsoft.com>.
Hi Mohammad,
You will have to implement BlockPlacementPolicy interface (abstract class). There are two existing implementations in Hadoop trunk, BlockPlacementPolicyDefault and BlockPlacementPolicyWithNodeGroup.
To configure Hadoop to use your policy, define dfs.block.replicator.classname in hdfs-site.xml. For example:
<property>
<name>dfs.block.replicator.classname</name>
<value>org.apache.hadoop.hdfs.server.namenode.AzureBlockPlacementPolicy</value>
</property>
Usually, you will also want to have your own script that computes the node topology/location (look for net.topology.script.file.name in trunk, or topology.script.file.name in branch-1).
Hope this helps,
Ivan
From: Mohammad Tariq [mailto:dontariq@gmail.com]
Sent: Saturday, May 04, 2013 8:35 AM
To: user@hadoop.apache.org
Subject: Re: Block placement Policy
You might find this useful : https://issues.apache.org/jira/browse/HDFS-385
Warm Regards,
Tariq
https://mtariq.jux.com/
cloudfront.blogspot.com<http://cloudfront.blogspot.com>
On Sat, May 4, 2013 at 8:57 PM, Mohammad Mustaqeem <3m...@gmail.com>> wrote:
I have read somewhere that a user can specified his own ReplicaPlacementPolicy.
How can I specify my own ReplicaPlacementPolicy?
I you have any sample ReplicaPlacementPolicy, then please share it..
--
*With regards ---*
*Mohammad Mustaqeem*,
M.Tech (CSE)
MNNIT Allahabad
9026604270
RE: Block placement Policy
Posted by Ivan Mitic <iv...@microsoft.com>.
Hi Mohammad,
You will have to implement BlockPlacementPolicy interface (abstract class). There are two existing implementations in Hadoop trunk, BlockPlacementPolicyDefault and BlockPlacementPolicyWithNodeGroup.
To configure Hadoop to use your policy, define dfs.block.replicator.classname in hdfs-site.xml. For example:
<property>
<name>dfs.block.replicator.classname</name>
<value>org.apache.hadoop.hdfs.server.namenode.AzureBlockPlacementPolicy</value>
</property>
Usually, you will also want to have your own script that computes the node topology/location (look for net.topology.script.file.name in trunk, or topology.script.file.name in branch-1).
Hope this helps,
Ivan
From: Mohammad Tariq [mailto:dontariq@gmail.com]
Sent: Saturday, May 04, 2013 8:35 AM
To: user@hadoop.apache.org
Subject: Re: Block placement Policy
You might find this useful : https://issues.apache.org/jira/browse/HDFS-385
Warm Regards,
Tariq
https://mtariq.jux.com/
cloudfront.blogspot.com<http://cloudfront.blogspot.com>
On Sat, May 4, 2013 at 8:57 PM, Mohammad Mustaqeem <3m...@gmail.com>> wrote:
I have read somewhere that a user can specified his own ReplicaPlacementPolicy.
How can I specify my own ReplicaPlacementPolicy?
I you have any sample ReplicaPlacementPolicy, then please share it..
--
*With regards ---*
*Mohammad Mustaqeem*,
M.Tech (CSE)
MNNIT Allahabad
9026604270
RE: Block placement Policy
Posted by Ivan Mitic <iv...@microsoft.com>.
Hi Mohammad,
You will have to implement BlockPlacementPolicy interface (abstract class). There are two existing implementations in Hadoop trunk, BlockPlacementPolicyDefault and BlockPlacementPolicyWithNodeGroup.
To configure Hadoop to use your policy, define dfs.block.replicator.classname in hdfs-site.xml. For example:
<property>
<name>dfs.block.replicator.classname</name>
<value>org.apache.hadoop.hdfs.server.namenode.AzureBlockPlacementPolicy</value>
</property>
Usually, you will also want to have your own script that computes the node topology/location (look for net.topology.script.file.name in trunk, or topology.script.file.name in branch-1).
Hope this helps,
Ivan
From: Mohammad Tariq [mailto:dontariq@gmail.com]
Sent: Saturday, May 04, 2013 8:35 AM
To: user@hadoop.apache.org
Subject: Re: Block placement Policy
You might find this useful : https://issues.apache.org/jira/browse/HDFS-385
Warm Regards,
Tariq
https://mtariq.jux.com/
cloudfront.blogspot.com<http://cloudfront.blogspot.com>
On Sat, May 4, 2013 at 8:57 PM, Mohammad Mustaqeem <3m...@gmail.com>> wrote:
I have read somewhere that a user can specified his own ReplicaPlacementPolicy.
How can I specify my own ReplicaPlacementPolicy?
I you have any sample ReplicaPlacementPolicy, then please share it..
--
*With regards ---*
*Mohammad Mustaqeem*,
M.Tech (CSE)
MNNIT Allahabad
9026604270
RE: Block placement Policy
Posted by Ivan Mitic <iv...@microsoft.com>.
Hi Mohammad,
You will have to implement BlockPlacementPolicy interface (abstract class). There are two existing implementations in Hadoop trunk, BlockPlacementPolicyDefault and BlockPlacementPolicyWithNodeGroup.
To configure Hadoop to use your policy, define dfs.block.replicator.classname in hdfs-site.xml. For example:
<property>
<name>dfs.block.replicator.classname</name>
<value>org.apache.hadoop.hdfs.server.namenode.AzureBlockPlacementPolicy</value>
</property>
Usually, you will also want to have your own script that computes the node topology/location (look for net.topology.script.file.name in trunk, or topology.script.file.name in branch-1).
Hope this helps,
Ivan
From: Mohammad Tariq [mailto:dontariq@gmail.com]
Sent: Saturday, May 04, 2013 8:35 AM
To: user@hadoop.apache.org
Subject: Re: Block placement Policy
You might find this useful : https://issues.apache.org/jira/browse/HDFS-385
Warm Regards,
Tariq
https://mtariq.jux.com/
cloudfront.blogspot.com<http://cloudfront.blogspot.com>
On Sat, May 4, 2013 at 8:57 PM, Mohammad Mustaqeem <3m...@gmail.com>> wrote:
I have read somewhere that a user can specified his own ReplicaPlacementPolicy.
How can I specify my own ReplicaPlacementPolicy?
I you have any sample ReplicaPlacementPolicy, then please share it..
--
*With regards ---*
*Mohammad Mustaqeem*,
M.Tech (CSE)
MNNIT Allahabad
9026604270
Re: Block placement Policy
Posted by Mohammad Mustaqeem <3m...@gmail.com>.
Do you know, how to use it?
Sorry, I am very new :(
--
*With regards ---*
*Mohammad Mustaqeem*,
M.Tech (CSE)
MNNIT Allahabad
9026604270
Re: Block placement Policy
Posted by Mohammad Mustaqeem <3m...@gmail.com>.
Do you know, how to use it?
Sorry, I am very new :(
--
*With regards ---*
*Mohammad Mustaqeem*,
M.Tech (CSE)
MNNIT Allahabad
9026604270
Re: Block placement Policy
Posted by Mohammad Tariq <do...@gmail.com>.
You might find this useful : https://issues.apache.org/jira/browse/HDFS-385
Warm Regards,
Tariq
https://mtariq.jux.com/
cloudfront.blogspot.com
On Sat, May 4, 2013 at 8:57 PM, Mohammad Mustaqeem
<3m...@gmail.com>wrote:
> I have read somewhere that a user can specified his own
> ReplicaPlacementPolicy.
> How can I specify my own ReplicaPlacementPolicy?
> I you have any sample ReplicaPlacementPolicy, then please share it..
>
> --
> *With regards ---*
> *Mohammad Mustaqeem*,
> M.Tech (CSE)
> MNNIT Allahabad
> 9026604270
>
Re: Block placement Policy
Posted by Rajat Sharma <fs...@gmail.com>.
Have a look at org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.
This is default placement policy, you should be able to extend this to
implement your own policy and set config parameter
"dfs.block.replicator.classname" to point to your class
On Sat, May 4, 2013 at 8:57 PM, Mohammad Mustaqeem
<3m...@gmail.com> wrote:
> I have read somewhere that a user can specified his own ReplicaPlacementPolicy.
> How can I specify my own ReplicaPlacementPolicy?
> I you have any sample ReplicaPlacementPolicy, then please share it..
>
> --
> *With regards ---*
> *Mohammad Mustaqeem*,
> M.Tech (CSE)
> MNNIT Allahabad
> 9026604270
Re: Block placement Policy
Posted by Mohammad Tariq <do...@gmail.com>.
You might find this useful : https://issues.apache.org/jira/browse/HDFS-385
Warm Regards,
Tariq
https://mtariq.jux.com/
cloudfront.blogspot.com
On Sat, May 4, 2013 at 8:57 PM, Mohammad Mustaqeem
<3m...@gmail.com>wrote:
> I have read somewhere that a user can specified his own
> ReplicaPlacementPolicy.
> How can I specify my own ReplicaPlacementPolicy?
> I you have any sample ReplicaPlacementPolicy, then please share it..
>
> --
> *With regards ---*
> *Mohammad Mustaqeem*,
> M.Tech (CSE)
> MNNIT Allahabad
> 9026604270
>
Re: Block placement Policy
Posted by Mohammad Tariq <do...@gmail.com>.
You might find this useful : https://issues.apache.org/jira/browse/HDFS-385
Warm Regards,
Tariq
https://mtariq.jux.com/
cloudfront.blogspot.com
On Sat, May 4, 2013 at 8:57 PM, Mohammad Mustaqeem
<3m...@gmail.com>wrote:
> I have read somewhere that a user can specified his own
> ReplicaPlacementPolicy.
> How can I specify my own ReplicaPlacementPolicy?
> I you have any sample ReplicaPlacementPolicy, then please share it..
>
> --
> *With regards ---*
> *Mohammad Mustaqeem*,
> M.Tech (CSE)
> MNNIT Allahabad
> 9026604270
>
Re: Block placement Policy
Posted by Mohammad Tariq <do...@gmail.com>.
You might find this useful : https://issues.apache.org/jira/browse/HDFS-385
Warm Regards,
Tariq
https://mtariq.jux.com/
cloudfront.blogspot.com
On Sat, May 4, 2013 at 8:57 PM, Mohammad Mustaqeem
<3m...@gmail.com>wrote:
> I have read somewhere that a user can specified his own
> ReplicaPlacementPolicy.
> How can I specify my own ReplicaPlacementPolicy?
> I you have any sample ReplicaPlacementPolicy, then please share it..
>
> --
> *With regards ---*
> *Mohammad Mustaqeem*,
> M.Tech (CSE)
> MNNIT Allahabad
> 9026604270
>
Re: Block placement Policy
Posted by Rajat Sharma <fs...@gmail.com>.
Have a look at org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.
This is default placement policy, you should be able to extend this to
implement your own policy and set config parameter
"dfs.block.replicator.classname" to point to your class
On Sat, May 4, 2013 at 8:57 PM, Mohammad Mustaqeem
<3m...@gmail.com> wrote:
> I have read somewhere that a user can specified his own ReplicaPlacementPolicy.
> How can I specify my own ReplicaPlacementPolicy?
> I you have any sample ReplicaPlacementPolicy, then please share it..
>
> --
> *With regards ---*
> *Mohammad Mustaqeem*,
> M.Tech (CSE)
> MNNIT Allahabad
> 9026604270
Re: Block placement Policy
Posted by Rajat Sharma <fs...@gmail.com>.
Have a look at org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.
This is default placement policy, you should be able to extend this to
implement your own policy and set config parameter
"dfs.block.replicator.classname" to point to your class
On Sat, May 4, 2013 at 8:57 PM, Mohammad Mustaqeem
<3m...@gmail.com> wrote:
> I have read somewhere that a user can specified his own ReplicaPlacementPolicy.
> How can I specify my own ReplicaPlacementPolicy?
> I you have any sample ReplicaPlacementPolicy, then please share it..
>
> --
> *With regards ---*
> *Mohammad Mustaqeem*,
> M.Tech (CSE)
> MNNIT Allahabad
> 9026604270