You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-user@hadoop.apache.org by Shinichi Yamashita <sh...@gmail.com> on 2012/10/08 15:23:41 UTC
How to change topology
Hi,
I know that DataNode and TaskTracker must restart to change topology.
Is there the method to execute the topology change without restart of
DataNode and TaskTracker?
In other words, can I change the topology by a command?
Thanks in advance!
Shinichi
Re: How to change topology
Posted by Shinichi Yamashita <sh...@gmail.com>.
Hi Steve,
Thank you for sharing information of JIRA with me.
Topology easy setting and confirmation will be necessary.
I use JIRA for an idea about toplogy setting and confirmation.
Regards,
Shinichi
2012/10/10 Ted Dunning <td...@maprtech.com>
>
>
> On Tue, Oct 9, 2012 at 12:17 PM, Steve Loughran <st...@hortonworks.com>wrote:
>
>>
>>
>> On 9 October 2012 16:51, Shinichi Yamashita <sh...@gmail.com> wrote:
>>
>>> Hi Steve,
>>>
>>> Thank you for your reply.
>>>
>>>
>>> > no, it's the Namenode and JobTracker that needs to be restarted;
>>> > they are the bits that care where the boxes are.
>>>
>>> I confirmed it in my cluster, and I understood it as follows.
>>> First, the resolved node information is recorded in ConcurrentHashMap.
>>> Next same node toplogy decision, HashMap data is given priority to over
>>> a script.
>>> The HashMap data become invalid by restart of NameNode and JobTracker.
>>>
>>
>> That's right. There's been lots of discussion on adding live topology
>> resetting.
>>
>
> For what it's worth, there is at least one Hadoop derived distribution
> which has live topology resetting already.
>
> I can follow up offline with anyone who wants more information.
>
>
Re: How to change topology
Posted by Shinichi Yamashita <sh...@gmail.com>.
Hi Steve,
Thank you for sharing information of JIRA with me.
Topology easy setting and confirmation will be necessary.
I use JIRA for an idea about toplogy setting and confirmation.
Regards,
Shinichi
2012/10/10 Ted Dunning <td...@maprtech.com>
>
>
> On Tue, Oct 9, 2012 at 12:17 PM, Steve Loughran <st...@hortonworks.com>wrote:
>
>>
>>
>> On 9 October 2012 16:51, Shinichi Yamashita <sh...@gmail.com> wrote:
>>
>>> Hi Steve,
>>>
>>> Thank you for your reply.
>>>
>>>
>>> > no, it's the Namenode and JobTracker that needs to be restarted;
>>> > they are the bits that care where the boxes are.
>>>
>>> I confirmed it in my cluster, and I understood it as follows.
>>> First, the resolved node information is recorded in ConcurrentHashMap.
>>> Next same node toplogy decision, HashMap data is given priority to over
>>> a script.
>>> The HashMap data become invalid by restart of NameNode and JobTracker.
>>>
>>
>> That's right. There's been lots of discussion on adding live topology
>> resetting.
>>
>
> For what it's worth, there is at least one Hadoop derived distribution
> which has live topology resetting already.
>
> I can follow up offline with anyone who wants more information.
>
>
Re: How to change topology
Posted by Shinichi Yamashita <sh...@gmail.com>.
Hi Steve,
Thank you for sharing information of JIRA with me.
Topology easy setting and confirmation will be necessary.
I use JIRA for an idea about toplogy setting and confirmation.
Regards,
Shinichi
2012/10/10 Ted Dunning <td...@maprtech.com>
>
>
> On Tue, Oct 9, 2012 at 12:17 PM, Steve Loughran <st...@hortonworks.com>wrote:
>
>>
>>
>> On 9 October 2012 16:51, Shinichi Yamashita <sh...@gmail.com> wrote:
>>
>>> Hi Steve,
>>>
>>> Thank you for your reply.
>>>
>>>
>>> > no, it's the Namenode and JobTracker that needs to be restarted;
>>> > they are the bits that care where the boxes are.
>>>
>>> I confirmed it in my cluster, and I understood it as follows.
>>> First, the resolved node information is recorded in ConcurrentHashMap.
>>> Next same node toplogy decision, HashMap data is given priority to over
>>> a script.
>>> The HashMap data become invalid by restart of NameNode and JobTracker.
>>>
>>
>> That's right. There's been lots of discussion on adding live topology
>> resetting.
>>
>
> For what it's worth, there is at least one Hadoop derived distribution
> which has live topology resetting already.
>
> I can follow up offline with anyone who wants more information.
>
>
Re: How to change topology
Posted by Shinichi Yamashita <sh...@gmail.com>.
Hi Steve,
Thank you for sharing information of JIRA with me.
Topology easy setting and confirmation will be necessary.
I use JIRA for an idea about toplogy setting and confirmation.
Regards,
Shinichi
2012/10/10 Ted Dunning <td...@maprtech.com>
>
>
> On Tue, Oct 9, 2012 at 12:17 PM, Steve Loughran <st...@hortonworks.com>wrote:
>
>>
>>
>> On 9 October 2012 16:51, Shinichi Yamashita <sh...@gmail.com> wrote:
>>
>>> Hi Steve,
>>>
>>> Thank you for your reply.
>>>
>>>
>>> > no, it's the Namenode and JobTracker that needs to be restarted;
>>> > they are the bits that care where the boxes are.
>>>
>>> I confirmed it in my cluster, and I understood it as follows.
>>> First, the resolved node information is recorded in ConcurrentHashMap.
>>> Next same node toplogy decision, HashMap data is given priority to over
>>> a script.
>>> The HashMap data become invalid by restart of NameNode and JobTracker.
>>>
>>
>> That's right. There's been lots of discussion on adding live topology
>> resetting.
>>
>
> For what it's worth, there is at least one Hadoop derived distribution
> which has live topology resetting already.
>
> I can follow up offline with anyone who wants more information.
>
>
Re: How to change topology
Posted by Ted Dunning <td...@maprtech.com>.
On Tue, Oct 9, 2012 at 12:17 PM, Steve Loughran <st...@hortonworks.com>wrote:
>
>
> On 9 October 2012 16:51, Shinichi Yamashita <sh...@gmail.com> wrote:
>
>> Hi Steve,
>>
>> Thank you for your reply.
>>
>>
>> > no, it's the Namenode and JobTracker that needs to be restarted;
>> > they are the bits that care where the boxes are.
>>
>> I confirmed it in my cluster, and I understood it as follows.
>> First, the resolved node information is recorded in ConcurrentHashMap.
>> Next same node toplogy decision, HashMap data is given priority to over a
>> script.
>> The HashMap data become invalid by restart of NameNode and JobTracker.
>>
>
> That's right. There's been lots of discussion on adding live topology
> resetting.
>
For what it's worth, there is at least one Hadoop derived distribution
which has live topology resetting already.
I can follow up offline with anyone who wants more information.
Re: How to change topology
Posted by Ted Dunning <td...@maprtech.com>.
On Tue, Oct 9, 2012 at 12:17 PM, Steve Loughran <st...@hortonworks.com>wrote:
>
>
> On 9 October 2012 16:51, Shinichi Yamashita <sh...@gmail.com> wrote:
>
>> Hi Steve,
>>
>> Thank you for your reply.
>>
>>
>> > no, it's the Namenode and JobTracker that needs to be restarted;
>> > they are the bits that care where the boxes are.
>>
>> I confirmed it in my cluster, and I understood it as follows.
>> First, the resolved node information is recorded in ConcurrentHashMap.
>> Next same node toplogy decision, HashMap data is given priority to over a
>> script.
>> The HashMap data become invalid by restart of NameNode and JobTracker.
>>
>
> That's right. There's been lots of discussion on adding live topology
> resetting.
>
For what it's worth, there is at least one Hadoop derived distribution
which has live topology resetting already.
I can follow up offline with anyone who wants more information.
Re: How to change topology
Posted by Ted Dunning <td...@maprtech.com>.
On Tue, Oct 9, 2012 at 12:17 PM, Steve Loughran <st...@hortonworks.com>wrote:
>
>
> On 9 October 2012 16:51, Shinichi Yamashita <sh...@gmail.com> wrote:
>
>> Hi Steve,
>>
>> Thank you for your reply.
>>
>>
>> > no, it's the Namenode and JobTracker that needs to be restarted;
>> > they are the bits that care where the boxes are.
>>
>> I confirmed it in my cluster, and I understood it as follows.
>> First, the resolved node information is recorded in ConcurrentHashMap.
>> Next same node toplogy decision, HashMap data is given priority to over a
>> script.
>> The HashMap data become invalid by restart of NameNode and JobTracker.
>>
>
> That's right. There's been lots of discussion on adding live topology
> resetting.
>
For what it's worth, there is at least one Hadoop derived distribution
which has live topology resetting already.
I can follow up offline with anyone who wants more information.
Re: How to change topology
Posted by Ted Dunning <td...@maprtech.com>.
On Tue, Oct 9, 2012 at 12:17 PM, Steve Loughran <st...@hortonworks.com>wrote:
>
>
> On 9 October 2012 16:51, Shinichi Yamashita <sh...@gmail.com> wrote:
>
>> Hi Steve,
>>
>> Thank you for your reply.
>>
>>
>> > no, it's the Namenode and JobTracker that needs to be restarted;
>> > they are the bits that care where the boxes are.
>>
>> I confirmed it in my cluster, and I understood it as follows.
>> First, the resolved node information is recorded in ConcurrentHashMap.
>> Next same node toplogy decision, HashMap data is given priority to over a
>> script.
>> The HashMap data become invalid by restart of NameNode and JobTracker.
>>
>
> That's right. There's been lots of discussion on adding live topology
> resetting.
>
For what it's worth, there is at least one Hadoop derived distribution
which has live topology resetting already.
I can follow up offline with anyone who wants more information.
Re: How to change topology
Posted by Steve Loughran <st...@hortonworks.com>.
On 9 October 2012 16:51, Shinichi Yamashita <sh...@gmail.com> wrote:
> Hi Steve,
>
> Thank you for your reply.
>
>
> > no, it's the Namenode and JobTracker that needs to be restarted;
> > they are the bits that care where the boxes are.
>
> I confirmed it in my cluster, and I understood it as follows.
> First, the resolved node information is recorded in ConcurrentHashMap.
> Next same node toplogy decision, HashMap data is given priority to over a
> script.
> The HashMap data become invalid by restart of NameNode and JobTracker.
>
That's right. There's been lots of discussion on adding live topology
resetting.
There's also a large unfinished JIRA in my name to do better topology work
in Hadoop 2,
https://issues.apache.org/jira/browse/HADOOP-8231
Including the ability to do local checks of a topology script
https://issues.apache.org/jira/browse/HADOOP-8232
Supporting topology dump and reset from any of the manager nodes would come
after.
-steve
Re: How to change topology
Posted by Steve Loughran <st...@hortonworks.com>.
On 9 October 2012 16:51, Shinichi Yamashita <sh...@gmail.com> wrote:
> Hi Steve,
>
> Thank you for your reply.
>
>
> > no, it's the Namenode and JobTracker that needs to be restarted;
> > they are the bits that care where the boxes are.
>
> I confirmed it in my cluster, and I understood it as follows.
> First, the resolved node information is recorded in ConcurrentHashMap.
> Next same node toplogy decision, HashMap data is given priority to over a
> script.
> The HashMap data become invalid by restart of NameNode and JobTracker.
>
That's right. There's been lots of discussion on adding live topology
resetting.
There's also a large unfinished JIRA in my name to do better topology work
in Hadoop 2,
https://issues.apache.org/jira/browse/HADOOP-8231
Including the ability to do local checks of a topology script
https://issues.apache.org/jira/browse/HADOOP-8232
Supporting topology dump and reset from any of the manager nodes would come
after.
-steve
Re: How to change topology
Posted by Steve Loughran <st...@hortonworks.com>.
On 9 October 2012 16:51, Shinichi Yamashita <sh...@gmail.com> wrote:
> Hi Steve,
>
> Thank you for your reply.
>
>
> > no, it's the Namenode and JobTracker that needs to be restarted;
> > they are the bits that care where the boxes are.
>
> I confirmed it in my cluster, and I understood it as follows.
> First, the resolved node information is recorded in ConcurrentHashMap.
> Next same node toplogy decision, HashMap data is given priority to over a
> script.
> The HashMap data become invalid by restart of NameNode and JobTracker.
>
That's right. There's been lots of discussion on adding live topology
resetting.
There's also a large unfinished JIRA in my name to do better topology work
in Hadoop 2,
https://issues.apache.org/jira/browse/HADOOP-8231
Including the ability to do local checks of a topology script
https://issues.apache.org/jira/browse/HADOOP-8232
Supporting topology dump and reset from any of the manager nodes would come
after.
-steve
Re: How to change topology
Posted by Steve Loughran <st...@hortonworks.com>.
On 9 October 2012 16:51, Shinichi Yamashita <sh...@gmail.com> wrote:
> Hi Steve,
>
> Thank you for your reply.
>
>
> > no, it's the Namenode and JobTracker that needs to be restarted;
> > they are the bits that care where the boxes are.
>
> I confirmed it in my cluster, and I understood it as follows.
> First, the resolved node information is recorded in ConcurrentHashMap.
> Next same node toplogy decision, HashMap data is given priority to over a
> script.
> The HashMap data become invalid by restart of NameNode and JobTracker.
>
That's right. There's been lots of discussion on adding live topology
resetting.
There's also a large unfinished JIRA in my name to do better topology work
in Hadoop 2,
https://issues.apache.org/jira/browse/HADOOP-8231
Including the ability to do local checks of a topology script
https://issues.apache.org/jira/browse/HADOOP-8232
Supporting topology dump and reset from any of the manager nodes would come
after.
-steve
Re: How to change topology
Posted by Shinichi Yamashita <sh...@gmail.com>.
Hi Steve,
Thank you for your reply.
> no, it's the Namenode and JobTracker that needs to be restarted;
> they are the bits that care where the boxes are.
I confirmed it in my cluster, and I understood it as follows.
First, the resolved node information is recorded in ConcurrentHashMap.
Next same node toplogy decision, HashMap data is given priority to over a
script.
The HashMap data become invalid by restart of NameNode and JobTracker.
Regards,
Shinichi
2012/10/9 Steve Loughran <st...@hortonworks.com>
>
>
> On 8 October 2012 14:23, Shinichi Yamashita <sh...@gmail.com> wrote:
>
>> Hi,
>>
>> I know that DataNode and TaskTracker must restart to change topology.
>>
>
> no, it's the Namenode and JobTracker that needs to be restarted; they are
> the bits that care where the boxes are.
>
>
>> Is there the method to execute the topology change without restart of
>> DataNode and TaskTracker?
>> In other words, can I change the topology by a command?
>>
>
> No -the map gets cached, and when the services start up they do quick
> check to see if the topology is flat or spread across switches -and use
> that flag to make various placement decisions.
>
> steve
>
>
>
Re: How to change topology
Posted by Shinichi Yamashita <sh...@gmail.com>.
Hi Steve,
Thank you for your reply.
> no, it's the Namenode and JobTracker that needs to be restarted;
> they are the bits that care where the boxes are.
I confirmed it in my cluster, and I understood it as follows.
First, the resolved node information is recorded in ConcurrentHashMap.
Next same node toplogy decision, HashMap data is given priority to over a
script.
The HashMap data become invalid by restart of NameNode and JobTracker.
Regards,
Shinichi
2012/10/9 Steve Loughran <st...@hortonworks.com>
>
>
> On 8 October 2012 14:23, Shinichi Yamashita <sh...@gmail.com> wrote:
>
>> Hi,
>>
>> I know that DataNode and TaskTracker must restart to change topology.
>>
>
> no, it's the Namenode and JobTracker that needs to be restarted; they are
> the bits that care where the boxes are.
>
>
>> Is there the method to execute the topology change without restart of
>> DataNode and TaskTracker?
>> In other words, can I change the topology by a command?
>>
>
> No -the map gets cached, and when the services start up they do quick
> check to see if the topology is flat or spread across switches -and use
> that flag to make various placement decisions.
>
> steve
>
>
>
Re: How to change topology
Posted by Shinichi Yamashita <sh...@gmail.com>.
Hi Steve,
Thank you for your reply.
> no, it's the Namenode and JobTracker that needs to be restarted;
> they are the bits that care where the boxes are.
I confirmed it in my cluster, and I understood it as follows.
First, the resolved node information is recorded in ConcurrentHashMap.
Next same node toplogy decision, HashMap data is given priority to over a
script.
The HashMap data become invalid by restart of NameNode and JobTracker.
Regards,
Shinichi
2012/10/9 Steve Loughran <st...@hortonworks.com>
>
>
> On 8 October 2012 14:23, Shinichi Yamashita <sh...@gmail.com> wrote:
>
>> Hi,
>>
>> I know that DataNode and TaskTracker must restart to change topology.
>>
>
> no, it's the Namenode and JobTracker that needs to be restarted; they are
> the bits that care where the boxes are.
>
>
>> Is there the method to execute the topology change without restart of
>> DataNode and TaskTracker?
>> In other words, can I change the topology by a command?
>>
>
> No -the map gets cached, and when the services start up they do quick
> check to see if the topology is flat or spread across switches -and use
> that flag to make various placement decisions.
>
> steve
>
>
>
Re: How to change topology
Posted by Shinichi Yamashita <sh...@gmail.com>.
Hi Steve,
Thank you for your reply.
> no, it's the Namenode and JobTracker that needs to be restarted;
> they are the bits that care where the boxes are.
I confirmed it in my cluster, and I understood it as follows.
First, the resolved node information is recorded in ConcurrentHashMap.
Next same node toplogy decision, HashMap data is given priority to over a
script.
The HashMap data become invalid by restart of NameNode and JobTracker.
Regards,
Shinichi
2012/10/9 Steve Loughran <st...@hortonworks.com>
>
>
> On 8 October 2012 14:23, Shinichi Yamashita <sh...@gmail.com> wrote:
>
>> Hi,
>>
>> I know that DataNode and TaskTracker must restart to change topology.
>>
>
> no, it's the Namenode and JobTracker that needs to be restarted; they are
> the bits that care where the boxes are.
>
>
>> Is there the method to execute the topology change without restart of
>> DataNode and TaskTracker?
>> In other words, can I change the topology by a command?
>>
>
> No -the map gets cached, and when the services start up they do quick
> check to see if the topology is flat or spread across switches -and use
> that flag to make various placement decisions.
>
> steve
>
>
>
Re: How to change topology
Posted by Steve Loughran <st...@hortonworks.com>.
On 8 October 2012 14:23, Shinichi Yamashita <sh...@gmail.com> wrote:
> Hi,
>
> I know that DataNode and TaskTracker must restart to change topology.
>
no, it's the Namenode and JobTracker that needs to be restarted; they are
the bits that care where the boxes are.
> Is there the method to execute the topology change without restart of
> DataNode and TaskTracker?
> In other words, can I change the topology by a command?
>
No -the map gets cached, and when the services start up they do quick check
to see if the topology is flat or spread across switches -and use that flag
to make various placement decisions.
steve
Re: How to change topology
Posted by Steve Loughran <st...@hortonworks.com>.
On 8 October 2012 14:23, Shinichi Yamashita <sh...@gmail.com> wrote:
> Hi,
>
> I know that DataNode and TaskTracker must restart to change topology.
>
no, it's the Namenode and JobTracker that needs to be restarted; they are
the bits that care where the boxes are.
> Is there the method to execute the topology change without restart of
> DataNode and TaskTracker?
> In other words, can I change the topology by a command?
>
No -the map gets cached, and when the services start up they do quick check
to see if the topology is flat or spread across switches -and use that flag
to make various placement decisions.
steve
Re: How to change topology
Posted by Steve Loughran <st...@hortonworks.com>.
On 8 October 2012 14:23, Shinichi Yamashita <sh...@gmail.com> wrote:
> Hi,
>
> I know that DataNode and TaskTracker must restart to change topology.
>
no, it's the Namenode and JobTracker that needs to be restarted; they are
the bits that care where the boxes are.
> Is there the method to execute the topology change without restart of
> DataNode and TaskTracker?
> In other words, can I change the topology by a command?
>
No -the map gets cached, and when the services start up they do quick check
to see if the topology is flat or spread across switches -and use that flag
to make various placement decisions.
steve
Re: How to change topology
Posted by Steve Loughran <st...@hortonworks.com>.
On 8 October 2012 14:23, Shinichi Yamashita <sh...@gmail.com> wrote:
> Hi,
>
> I know that DataNode and TaskTracker must restart to change topology.
>
no, it's the Namenode and JobTracker that needs to be restarted; they are
the bits that care where the boxes are.
> Is there the method to execute the topology change without restart of
> DataNode and TaskTracker?
> In other words, can I change the topology by a command?
>
No -the map gets cached, and when the services start up they do quick check
to see if the topology is flat or spread across switches -and use that flag
to make various placement decisions.
steve