You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hadoop.apache.org by Nick Niemeyer <nn...@riotgames.com> on 2013/06/06 22:48:16 UTC
Please explain FSNamesystemState TotalLoad
Can someone please explain what TotalLoad represents below? Thanks for your response in advance!
Version: hadoop-0.20-namenode-0.20.2+923.197-1
Example pulled from the output of via the name node:
# curl -i http://localhost:50070/jmx
{
"name" : "hadoop:service=NameNode,name=FSNamesystemState",
"modelerType" : "org.apache.hadoop.hdfs.server.namenode.FSNamesystem",
"CapacityTotal" : #,
"CapacityUsed" : #,
"CapacityRemaining" : #,
"TotalLoad" : #,
"BlocksTotal" : #,
"FilesTotal" : #,
"PendingReplicationBlocks" : 0,
"UnderReplicatedBlocks" : 0,
"ScheduledReplicationBlocks" : 0,
"FSState" : "Operational"
}
Thanks,
Nick
Re: Please explain FSNamesystemState TotalLoad
Posted by Suresh Srinivas <su...@hortonworks.com>.
On Fri, Jun 7, 2013 at 9:10 AM, Nick Niemeyer <nn...@riotgames.com>wrote:
> Regarding TotalLoad, what would be normal operating tolerances per node
> for this metric? When should one become concerned? Thanks again to
> everyone participating in this community. :)
>
>
Why do you want to be concered :) I have not seen many issues related to
high TotalLoad.
This is mainly useful in terms of understanding how many concurrent
jobs/file accesses are happening and how busy datanodes are. When you are
debugging issues where cluster slow down due to overload, or correlating a
run of big jobs, this is useful. Knowing what it represent, you would find
many other uses as well.
> From: Suresh Srinivas <su...@hortonworks.com>
> Reply-To: "user@hadoop.apache.org" <us...@hadoop.apache.org>
> Date: Thursday, June 6, 2013 4:14 PM
> To: "hdfs-user@hadoop.apache.org" <us...@hadoop.apache.org>
> Subject: Re: Please explain FSNamesystemState TotalLoad
>
> It is the total number of transceivers (readers and writers) reported
> by all the datanodes. Datanode reports this count in periodic heartbeat to
> the namenode.
>
>
> On Thu, Jun 6, 2013 at 1:48 PM, Nick Niemeyer <nn...@riotgames.com>wrote:
>
>> Can someone please explain what TotalLoad represents below? Thanks
>> for your response in advance!
>>
>> Version: hadoop-0.20-namenode-0.20.2+923.197-1
>>
>> Example pulled from the output of via the name node:
>> # curl -i http://localhost:50070/jmx
>>
>> {
>> "name" : "hadoop:service=NameNode,name=FSNamesystemState",
>> "modelerType" : "org.apache.hadoop.hdfs.server.namenode.FSNamesystem",
>> "CapacityTotal" : #,
>> "CapacityUsed" : #,
>> "CapacityRemaining" : #,
>> * "TotalLoad" : #,*
>> "BlocksTotal" : #,
>> "FilesTotal" : #,
>> "PendingReplicationBlocks" : 0,
>> "UnderReplicatedBlocks" : 0,
>> "ScheduledReplicationBlocks" : 0,
>> "FSState" : "Operational"
>> }
>>
>>
>> Thanks,
>> Nick
>>
>
>
>
> --
> http://hortonworks.com/download/
>
--
http://hortonworks.com/download/
Re: Please explain FSNamesystemState TotalLoad
Posted by Suresh Srinivas <su...@hortonworks.com>.
On Fri, Jun 7, 2013 at 9:10 AM, Nick Niemeyer <nn...@riotgames.com>wrote:
> Regarding TotalLoad, what would be normal operating tolerances per node
> for this metric? When should one become concerned? Thanks again to
> everyone participating in this community. :)
>
>
Why do you want to be concered :) I have not seen many issues related to
high TotalLoad.
This is mainly useful in terms of understanding how many concurrent
jobs/file accesses are happening and how busy datanodes are. When you are
debugging issues where cluster slow down due to overload, or correlating a
run of big jobs, this is useful. Knowing what it represent, you would find
many other uses as well.
> From: Suresh Srinivas <su...@hortonworks.com>
> Reply-To: "user@hadoop.apache.org" <us...@hadoop.apache.org>
> Date: Thursday, June 6, 2013 4:14 PM
> To: "hdfs-user@hadoop.apache.org" <us...@hadoop.apache.org>
> Subject: Re: Please explain FSNamesystemState TotalLoad
>
> It is the total number of transceivers (readers and writers) reported
> by all the datanodes. Datanode reports this count in periodic heartbeat to
> the namenode.
>
>
> On Thu, Jun 6, 2013 at 1:48 PM, Nick Niemeyer <nn...@riotgames.com>wrote:
>
>> Can someone please explain what TotalLoad represents below? Thanks
>> for your response in advance!
>>
>> Version: hadoop-0.20-namenode-0.20.2+923.197-1
>>
>> Example pulled from the output of via the name node:
>> # curl -i http://localhost:50070/jmx
>>
>> {
>> "name" : "hadoop:service=NameNode,name=FSNamesystemState",
>> "modelerType" : "org.apache.hadoop.hdfs.server.namenode.FSNamesystem",
>> "CapacityTotal" : #,
>> "CapacityUsed" : #,
>> "CapacityRemaining" : #,
>> * "TotalLoad" : #,*
>> "BlocksTotal" : #,
>> "FilesTotal" : #,
>> "PendingReplicationBlocks" : 0,
>> "UnderReplicatedBlocks" : 0,
>> "ScheduledReplicationBlocks" : 0,
>> "FSState" : "Operational"
>> }
>>
>>
>> Thanks,
>> Nick
>>
>
>
>
> --
> http://hortonworks.com/download/
>
--
http://hortonworks.com/download/
Re: Please explain FSNamesystemState TotalLoad
Posted by Suresh Srinivas <su...@hortonworks.com>.
On Fri, Jun 7, 2013 at 9:10 AM, Nick Niemeyer <nn...@riotgames.com>wrote:
> Regarding TotalLoad, what would be normal operating tolerances per node
> for this metric? When should one become concerned? Thanks again to
> everyone participating in this community. :)
>
>
Why do you want to be concered :) I have not seen many issues related to
high TotalLoad.
This is mainly useful in terms of understanding how many concurrent
jobs/file accesses are happening and how busy datanodes are. When you are
debugging issues where cluster slow down due to overload, or correlating a
run of big jobs, this is useful. Knowing what it represent, you would find
many other uses as well.
> From: Suresh Srinivas <su...@hortonworks.com>
> Reply-To: "user@hadoop.apache.org" <us...@hadoop.apache.org>
> Date: Thursday, June 6, 2013 4:14 PM
> To: "hdfs-user@hadoop.apache.org" <us...@hadoop.apache.org>
> Subject: Re: Please explain FSNamesystemState TotalLoad
>
> It is the total number of transceivers (readers and writers) reported
> by all the datanodes. Datanode reports this count in periodic heartbeat to
> the namenode.
>
>
> On Thu, Jun 6, 2013 at 1:48 PM, Nick Niemeyer <nn...@riotgames.com>wrote:
>
>> Can someone please explain what TotalLoad represents below? Thanks
>> for your response in advance!
>>
>> Version: hadoop-0.20-namenode-0.20.2+923.197-1
>>
>> Example pulled from the output of via the name node:
>> # curl -i http://localhost:50070/jmx
>>
>> {
>> "name" : "hadoop:service=NameNode,name=FSNamesystemState",
>> "modelerType" : "org.apache.hadoop.hdfs.server.namenode.FSNamesystem",
>> "CapacityTotal" : #,
>> "CapacityUsed" : #,
>> "CapacityRemaining" : #,
>> * "TotalLoad" : #,*
>> "BlocksTotal" : #,
>> "FilesTotal" : #,
>> "PendingReplicationBlocks" : 0,
>> "UnderReplicatedBlocks" : 0,
>> "ScheduledReplicationBlocks" : 0,
>> "FSState" : "Operational"
>> }
>>
>>
>> Thanks,
>> Nick
>>
>
>
>
> --
> http://hortonworks.com/download/
>
--
http://hortonworks.com/download/
Re: Please explain FSNamesystemState TotalLoad
Posted by Suresh Srinivas <su...@hortonworks.com>.
On Fri, Jun 7, 2013 at 9:10 AM, Nick Niemeyer <nn...@riotgames.com>wrote:
> Regarding TotalLoad, what would be normal operating tolerances per node
> for this metric? When should one become concerned? Thanks again to
> everyone participating in this community. :)
>
>
Why do you want to be concered :) I have not seen many issues related to
high TotalLoad.
This is mainly useful in terms of understanding how many concurrent
jobs/file accesses are happening and how busy datanodes are. When you are
debugging issues where cluster slow down due to overload, or correlating a
run of big jobs, this is useful. Knowing what it represent, you would find
many other uses as well.
> From: Suresh Srinivas <su...@hortonworks.com>
> Reply-To: "user@hadoop.apache.org" <us...@hadoop.apache.org>
> Date: Thursday, June 6, 2013 4:14 PM
> To: "hdfs-user@hadoop.apache.org" <us...@hadoop.apache.org>
> Subject: Re: Please explain FSNamesystemState TotalLoad
>
> It is the total number of transceivers (readers and writers) reported
> by all the datanodes. Datanode reports this count in periodic heartbeat to
> the namenode.
>
>
> On Thu, Jun 6, 2013 at 1:48 PM, Nick Niemeyer <nn...@riotgames.com>wrote:
>
>> Can someone please explain what TotalLoad represents below? Thanks
>> for your response in advance!
>>
>> Version: hadoop-0.20-namenode-0.20.2+923.197-1
>>
>> Example pulled from the output of via the name node:
>> # curl -i http://localhost:50070/jmx
>>
>> {
>> "name" : "hadoop:service=NameNode,name=FSNamesystemState",
>> "modelerType" : "org.apache.hadoop.hdfs.server.namenode.FSNamesystem",
>> "CapacityTotal" : #,
>> "CapacityUsed" : #,
>> "CapacityRemaining" : #,
>> * "TotalLoad" : #,*
>> "BlocksTotal" : #,
>> "FilesTotal" : #,
>> "PendingReplicationBlocks" : 0,
>> "UnderReplicatedBlocks" : 0,
>> "ScheduledReplicationBlocks" : 0,
>> "FSState" : "Operational"
>> }
>>
>>
>> Thanks,
>> Nick
>>
>
>
>
> --
> http://hortonworks.com/download/
>
--
http://hortonworks.com/download/
Re: Please explain FSNamesystemState TotalLoad
Posted by Nick Niemeyer <nn...@riotgames.com>.
Regarding TotalLoad, what would be normal operating tolerances per node for this metric? When should one become concerned? Thanks again to everyone participating in this community. :)
Nick
From: Suresh Srinivas <su...@hortonworks.com>>
Reply-To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <us...@hadoop.apache.org>>
Date: Thursday, June 6, 2013 4:14 PM
To: "hdfs-user@hadoop.apache.org<ma...@hadoop.apache.org>" <us...@hadoop.apache.org>>
Subject: Re: Please explain FSNamesystemState TotalLoad
It is the total number of transceivers (readers and writers) reported by all the datanodes. Datanode reports this count in periodic heartbeat to the namenode.
On Thu, Jun 6, 2013 at 1:48 PM, Nick Niemeyer <nn...@riotgames.com>> wrote:
Can someone please explain what TotalLoad represents below? Thanks for your response in advance!
Version: hadoop-0.20-namenode-0.20.2+923.197-1
Example pulled from the output of via the name node:
# curl -i http://localhost:50070/jmx
{
"name" : "hadoop:service=NameNode,name=FSNamesystemState",
"modelerType" : "org.apache.hadoop.hdfs.server.namenode.FSNamesystem",
"CapacityTotal" : #,
"CapacityUsed" : #,
"CapacityRemaining" : #,
"TotalLoad" : #,
"BlocksTotal" : #,
"FilesTotal" : #,
"PendingReplicationBlocks" : 0,
"UnderReplicatedBlocks" : 0,
"ScheduledReplicationBlocks" : 0,
"FSState" : "Operational"
}
Thanks,
Nick
--
http://hortonworks.com/download/
Re: Please explain FSNamesystemState TotalLoad
Posted by Nick Niemeyer <nn...@riotgames.com>.
Regarding TotalLoad, what would be normal operating tolerances per node for this metric? When should one become concerned? Thanks again to everyone participating in this community. :)
Nick
From: Suresh Srinivas <su...@hortonworks.com>>
Reply-To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <us...@hadoop.apache.org>>
Date: Thursday, June 6, 2013 4:14 PM
To: "hdfs-user@hadoop.apache.org<ma...@hadoop.apache.org>" <us...@hadoop.apache.org>>
Subject: Re: Please explain FSNamesystemState TotalLoad
It is the total number of transceivers (readers and writers) reported by all the datanodes. Datanode reports this count in periodic heartbeat to the namenode.
On Thu, Jun 6, 2013 at 1:48 PM, Nick Niemeyer <nn...@riotgames.com>> wrote:
Can someone please explain what TotalLoad represents below? Thanks for your response in advance!
Version: hadoop-0.20-namenode-0.20.2+923.197-1
Example pulled from the output of via the name node:
# curl -i http://localhost:50070/jmx
{
"name" : "hadoop:service=NameNode,name=FSNamesystemState",
"modelerType" : "org.apache.hadoop.hdfs.server.namenode.FSNamesystem",
"CapacityTotal" : #,
"CapacityUsed" : #,
"CapacityRemaining" : #,
"TotalLoad" : #,
"BlocksTotal" : #,
"FilesTotal" : #,
"PendingReplicationBlocks" : 0,
"UnderReplicatedBlocks" : 0,
"ScheduledReplicationBlocks" : 0,
"FSState" : "Operational"
}
Thanks,
Nick
--
http://hortonworks.com/download/
Re: Please explain FSNamesystemState TotalLoad
Posted by Nick Niemeyer <nn...@riotgames.com>.
Regarding TotalLoad, what would be normal operating tolerances per node for this metric? When should one become concerned? Thanks again to everyone participating in this community. :)
Nick
From: Suresh Srinivas <su...@hortonworks.com>>
Reply-To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <us...@hadoop.apache.org>>
Date: Thursday, June 6, 2013 4:14 PM
To: "hdfs-user@hadoop.apache.org<ma...@hadoop.apache.org>" <us...@hadoop.apache.org>>
Subject: Re: Please explain FSNamesystemState TotalLoad
It is the total number of transceivers (readers and writers) reported by all the datanodes. Datanode reports this count in periodic heartbeat to the namenode.
On Thu, Jun 6, 2013 at 1:48 PM, Nick Niemeyer <nn...@riotgames.com>> wrote:
Can someone please explain what TotalLoad represents below? Thanks for your response in advance!
Version: hadoop-0.20-namenode-0.20.2+923.197-1
Example pulled from the output of via the name node:
# curl -i http://localhost:50070/jmx
{
"name" : "hadoop:service=NameNode,name=FSNamesystemState",
"modelerType" : "org.apache.hadoop.hdfs.server.namenode.FSNamesystem",
"CapacityTotal" : #,
"CapacityUsed" : #,
"CapacityRemaining" : #,
"TotalLoad" : #,
"BlocksTotal" : #,
"FilesTotal" : #,
"PendingReplicationBlocks" : 0,
"UnderReplicatedBlocks" : 0,
"ScheduledReplicationBlocks" : 0,
"FSState" : "Operational"
}
Thanks,
Nick
--
http://hortonworks.com/download/
Re: Please explain FSNamesystemState TotalLoad
Posted by Nick Niemeyer <nn...@riotgames.com>.
Regarding TotalLoad, what would be normal operating tolerances per node for this metric? When should one become concerned? Thanks again to everyone participating in this community. :)
Nick
From: Suresh Srinivas <su...@hortonworks.com>>
Reply-To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <us...@hadoop.apache.org>>
Date: Thursday, June 6, 2013 4:14 PM
To: "hdfs-user@hadoop.apache.org<ma...@hadoop.apache.org>" <us...@hadoop.apache.org>>
Subject: Re: Please explain FSNamesystemState TotalLoad
It is the total number of transceivers (readers and writers) reported by all the datanodes. Datanode reports this count in periodic heartbeat to the namenode.
On Thu, Jun 6, 2013 at 1:48 PM, Nick Niemeyer <nn...@riotgames.com>> wrote:
Can someone please explain what TotalLoad represents below? Thanks for your response in advance!
Version: hadoop-0.20-namenode-0.20.2+923.197-1
Example pulled from the output of via the name node:
# curl -i http://localhost:50070/jmx
{
"name" : "hadoop:service=NameNode,name=FSNamesystemState",
"modelerType" : "org.apache.hadoop.hdfs.server.namenode.FSNamesystem",
"CapacityTotal" : #,
"CapacityUsed" : #,
"CapacityRemaining" : #,
"TotalLoad" : #,
"BlocksTotal" : #,
"FilesTotal" : #,
"PendingReplicationBlocks" : 0,
"UnderReplicatedBlocks" : 0,
"ScheduledReplicationBlocks" : 0,
"FSState" : "Operational"
}
Thanks,
Nick
--
http://hortonworks.com/download/
Re: Please explain FSNamesystemState TotalLoad
Posted by Suresh Srinivas <su...@hortonworks.com>.
It is the total number of transceivers (readers and writers) reported by
all the datanodes. Datanode reports this count in periodic heartbeat to the
namenode.
On Thu, Jun 6, 2013 at 1:48 PM, Nick Niemeyer <nn...@riotgames.com>wrote:
> Can someone please explain what TotalLoad represents below? Thanks for
> your response in advance!
>
> Version: hadoop-0.20-namenode-0.20.2+923.197-1
>
> Example pulled from the output of via the name node:
> # curl -i http://localhost:50070/jmx
>
> {
> "name" : "hadoop:service=NameNode,name=FSNamesystemState",
> "modelerType" : "org.apache.hadoop.hdfs.server.namenode.FSNamesystem",
> "CapacityTotal" : #,
> "CapacityUsed" : #,
> "CapacityRemaining" : #,
> * "TotalLoad" : #,*
> "BlocksTotal" : #,
> "FilesTotal" : #,
> "PendingReplicationBlocks" : 0,
> "UnderReplicatedBlocks" : 0,
> "ScheduledReplicationBlocks" : 0,
> "FSState" : "Operational"
> }
>
>
> Thanks,
> Nick
>
--
http://hortonworks.com/download/
Re: Please explain FSNamesystemState TotalLoad
Posted by Suresh Srinivas <su...@hortonworks.com>.
It is the total number of transceivers (readers and writers) reported by
all the datanodes. Datanode reports this count in periodic heartbeat to the
namenode.
On Thu, Jun 6, 2013 at 1:48 PM, Nick Niemeyer <nn...@riotgames.com>wrote:
> Can someone please explain what TotalLoad represents below? Thanks for
> your response in advance!
>
> Version: hadoop-0.20-namenode-0.20.2+923.197-1
>
> Example pulled from the output of via the name node:
> # curl -i http://localhost:50070/jmx
>
> {
> "name" : "hadoop:service=NameNode,name=FSNamesystemState",
> "modelerType" : "org.apache.hadoop.hdfs.server.namenode.FSNamesystem",
> "CapacityTotal" : #,
> "CapacityUsed" : #,
> "CapacityRemaining" : #,
> * "TotalLoad" : #,*
> "BlocksTotal" : #,
> "FilesTotal" : #,
> "PendingReplicationBlocks" : 0,
> "UnderReplicatedBlocks" : 0,
> "ScheduledReplicationBlocks" : 0,
> "FSState" : "Operational"
> }
>
>
> Thanks,
> Nick
>
--
http://hortonworks.com/download/
Re: Please explain FSNamesystemState TotalLoad
Posted by Suresh Srinivas <su...@hortonworks.com>.
It is the total number of transceivers (readers and writers) reported by
all the datanodes. Datanode reports this count in periodic heartbeat to the
namenode.
On Thu, Jun 6, 2013 at 1:48 PM, Nick Niemeyer <nn...@riotgames.com>wrote:
> Can someone please explain what TotalLoad represents below? Thanks for
> your response in advance!
>
> Version: hadoop-0.20-namenode-0.20.2+923.197-1
>
> Example pulled from the output of via the name node:
> # curl -i http://localhost:50070/jmx
>
> {
> "name" : "hadoop:service=NameNode,name=FSNamesystemState",
> "modelerType" : "org.apache.hadoop.hdfs.server.namenode.FSNamesystem",
> "CapacityTotal" : #,
> "CapacityUsed" : #,
> "CapacityRemaining" : #,
> * "TotalLoad" : #,*
> "BlocksTotal" : #,
> "FilesTotal" : #,
> "PendingReplicationBlocks" : 0,
> "UnderReplicatedBlocks" : 0,
> "ScheduledReplicationBlocks" : 0,
> "FSState" : "Operational"
> }
>
>
> Thanks,
> Nick
>
--
http://hortonworks.com/download/
Re: Please explain FSNamesystemState TotalLoad
Posted by Suresh Srinivas <su...@hortonworks.com>.
It is the total number of transceivers (readers and writers) reported by
all the datanodes. Datanode reports this count in periodic heartbeat to the
namenode.
On Thu, Jun 6, 2013 at 1:48 PM, Nick Niemeyer <nn...@riotgames.com>wrote:
> Can someone please explain what TotalLoad represents below? Thanks for
> your response in advance!
>
> Version: hadoop-0.20-namenode-0.20.2+923.197-1
>
> Example pulled from the output of via the name node:
> # curl -i http://localhost:50070/jmx
>
> {
> "name" : "hadoop:service=NameNode,name=FSNamesystemState",
> "modelerType" : "org.apache.hadoop.hdfs.server.namenode.FSNamesystem",
> "CapacityTotal" : #,
> "CapacityUsed" : #,
> "CapacityRemaining" : #,
> * "TotalLoad" : #,*
> "BlocksTotal" : #,
> "FilesTotal" : #,
> "PendingReplicationBlocks" : 0,
> "UnderReplicatedBlocks" : 0,
> "ScheduledReplicationBlocks" : 0,
> "FSState" : "Operational"
> }
>
>
> Thanks,
> Nick
>
--
http://hortonworks.com/download/