Re:

[Wei-Chiu Chuang <we...@cloudera.com.INVALID>]

+Hdfs-dev <hd...@hadoop.apache.org>
Hi Shuubham,

Just like to clarify a bit. What's the purpose of this work? Is this for
the general block placement policy in HDFS, or the
balancer/mover/diskbalancer, or decommissioning/recommissioning? Block
placement is determined by NameNode. Do you intend to shorten the time to
decide where a block is placed? Do you want to reduce the time such that
re-replication takes less time?

I'm asking this because I don't think there's ever a placementmonitor or a
blockmonitor class.

On Wed, Dec 19, 2018 at 10:36 PM Shuubham Ojha <sh...@gmail.com>
wrote:

> Hello All,
>
>                        I am Shuubham Ojha a graduate researcher with the
> University Of Melbourne. We have developed a block placement strategy which
> optimises delay associated with reconstruction. As a result of this
> optimisation problem, we get a placement matrix for blocks which tells us
> which block has to be placed at which node. We have been able to implement
> this strategy in Hadoop 2 by tweaking the file *placementmonitor.java*
> and *blockmover.java* where *placementmonitor.java* monitors the
> placement process and calls *blockmover.java* when the placement is not
> according the strategy. However, I can't find any such file analogous to
> *placementmonitor.java* in Hadoop 3 although I think that the closest
> file which performs this function is *balancer.java* located in
> hadoop-hdfs-project. Can anybody please provide me more information on this
> front?
>
>
> Warm Regards,
>
> Shuubham Ojha
>
> University Of Melbourne,
>
> Victoria, Australia- 3010
>

Re:

[Shuubham Ojha <sh...@gmail.com>]

Hi,
          Actually we just want to implement our block placement strategy
in hadoop 3 and compare results. I think that we should see some
improvements. Can you shed some more light on the difference between
general block placement policy and balancer / mover thing.

Warm regards,
Shuubham Ojha

On Thu, Dec 20, 2018 at 9:24 PM Wei-Chiu Chuang <we...@cloudera.com>
wrote:

> +Hdfs-dev <hd...@hadoop.apache.org>
> Hi Shuubham,
>
> Just like to clarify a bit. What's the purpose of this work? Is this for
> the general block placement policy in HDFS, or the
> balancer/mover/diskbalancer, or decommissioning/recommissioning? Block
> placement is determined by NameNode. Do you intend to shorten the time to
> decide where a block is placed? Do you want to reduce the time such that
> re-replication takes less time?
>
> I'm asking this because I don't think there's ever a placementmonitor or a
> blockmonitor class.
>
> On Wed, Dec 19, 2018 at 10:36 PM Shuubham Ojha <sh...@gmail.com>
> wrote:
>
>> Hello All,
>>
>>                        I am Shuubham Ojha a graduate researcher with the
>> University Of Melbourne. We have developed a block placement strategy which
>> optimises delay associated with reconstruction. As a result of this
>> optimisation problem, we get a placement matrix for blocks which tells us
>> which block has to be placed at which node. We have been able to implement
>> this strategy in Hadoop 2 by tweaking the file *placementmonitor.java*
>> and *blockmover.java* where *placementmonitor.java* monitors the
>> placement process and calls *blockmover.java* when the placement is not
>> according the strategy. However, I can't find any such file analogous to
>> *placementmonitor.java* in Hadoop 3 although I think that the closest
>> file which performs this function is *balancer.java* located in
>> hadoop-hdfs-project. Can anybody please provide me more information on this
>> front?
>>
>>
>> Warm Regards,
>>
>> Shuubham Ojha
>>
>> University Of Melbourne,
>>
>> Victoria, Australia- 3010
>>
>