You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hadoop.apache.org by Brian Mason <br...@gabey.com> on 2013/06/06 14:51:56 UTC
Management API
I am looking for a way to access a list of Nodes, Compute, Data etc .. My
application is not running on the name node. It is remote. The 2.0 Yarn
API look like they may be useful, but I am not on 2.0 and cannot move to
2,0 anytime soon.
DFSClient.java looks useful, but its not in the API docs so I am not sure
how to use it or even if I should.
Any pointers would be helpful.
Thanks,
Re: Management API
Posted by Suresh Srinivas <su...@hortonworks.com>.
Namenode exposes all these information over JMX. The same is also exposed
over http, which you could use for your use case.
See http://<nn:httpport>/jmx to list all the properties exposed. Typically
http port is 50070. You will find the information you looking for at:
http://<nn:httpport>/jmx?Hadoop:service=NameNode,name=NameNodeInfo
On Thu, Jun 6, 2013 at 9:17 AM, Brian Mason <br...@gabey.com> wrote:
> Mostly looking for a list of data nodes. I am making web page to display
> some info about data nodes and I need a list. I can screen scrape the JSP
> GUI, I was just hoping for something more elegant.
>
>
>
> On Thu, Jun 6, 2013 at 8:40 AM, John Lilley <jo...@redpoint.net>wrote:
>
>> What resources are you trying to access? ****
>>
>> Do you want to monitor the system status?****
>>
>> Do you want to read/write HDFS as a client? ****
>>
>> Do you want to run your application on the Hadoop cluster? ****
>>
>> John****
>>
>> ** **
>>
>> ** **
>>
>> *From:* Brian Mason [mailto:brian@gabey.com]
>> *Sent:* Thursday, June 06, 2013 6:52 AM
>> *To:* user@hadoop.apache.org
>> *Subject:* Management API****
>>
>> ** **
>>
>> I am looking for a way to access a list of Nodes, Compute, Data etc ..
>> My application is not running on the name node. It is remote. The 2.0
>> Yarn API look like they may be useful, but I am not on 2.0 and cannot move
>> to 2,0 anytime soon.****
>>
>> ** **
>>
>> DFSClient.java looks useful, but its not in the API docs so I am not sure
>> how to use it or even if I should.****
>>
>> Any pointers would be helpful. ****
>>
>> ** **
>>
>> Thanks,****
>>
>
>
--
http://hortonworks.com/download/
Re: Management API
Posted by Suresh Srinivas <su...@hortonworks.com>.
Namenode exposes all these information over JMX. The same is also exposed
over http, which you could use for your use case.
See http://<nn:httpport>/jmx to list all the properties exposed. Typically
http port is 50070. You will find the information you looking for at:
http://<nn:httpport>/jmx?Hadoop:service=NameNode,name=NameNodeInfo
On Thu, Jun 6, 2013 at 9:17 AM, Brian Mason <br...@gabey.com> wrote:
> Mostly looking for a list of data nodes. I am making web page to display
> some info about data nodes and I need a list. I can screen scrape the JSP
> GUI, I was just hoping for something more elegant.
>
>
>
> On Thu, Jun 6, 2013 at 8:40 AM, John Lilley <jo...@redpoint.net>wrote:
>
>> What resources are you trying to access? ****
>>
>> Do you want to monitor the system status?****
>>
>> Do you want to read/write HDFS as a client? ****
>>
>> Do you want to run your application on the Hadoop cluster? ****
>>
>> John****
>>
>> ** **
>>
>> ** **
>>
>> *From:* Brian Mason [mailto:brian@gabey.com]
>> *Sent:* Thursday, June 06, 2013 6:52 AM
>> *To:* user@hadoop.apache.org
>> *Subject:* Management API****
>>
>> ** **
>>
>> I am looking for a way to access a list of Nodes, Compute, Data etc ..
>> My application is not running on the name node. It is remote. The 2.0
>> Yarn API look like they may be useful, but I am not on 2.0 and cannot move
>> to 2,0 anytime soon.****
>>
>> ** **
>>
>> DFSClient.java looks useful, but its not in the API docs so I am not sure
>> how to use it or even if I should.****
>>
>> Any pointers would be helpful. ****
>>
>> ** **
>>
>> Thanks,****
>>
>
>
--
http://hortonworks.com/download/
Re: Management API
Posted by Suresh Srinivas <su...@hortonworks.com>.
Namenode exposes all these information over JMX. The same is also exposed
over http, which you could use for your use case.
See http://<nn:httpport>/jmx to list all the properties exposed. Typically
http port is 50070. You will find the information you looking for at:
http://<nn:httpport>/jmx?Hadoop:service=NameNode,name=NameNodeInfo
On Thu, Jun 6, 2013 at 9:17 AM, Brian Mason <br...@gabey.com> wrote:
> Mostly looking for a list of data nodes. I am making web page to display
> some info about data nodes and I need a list. I can screen scrape the JSP
> GUI, I was just hoping for something more elegant.
>
>
>
> On Thu, Jun 6, 2013 at 8:40 AM, John Lilley <jo...@redpoint.net>wrote:
>
>> What resources are you trying to access? ****
>>
>> Do you want to monitor the system status?****
>>
>> Do you want to read/write HDFS as a client? ****
>>
>> Do you want to run your application on the Hadoop cluster? ****
>>
>> John****
>>
>> ** **
>>
>> ** **
>>
>> *From:* Brian Mason [mailto:brian@gabey.com]
>> *Sent:* Thursday, June 06, 2013 6:52 AM
>> *To:* user@hadoop.apache.org
>> *Subject:* Management API****
>>
>> ** **
>>
>> I am looking for a way to access a list of Nodes, Compute, Data etc ..
>> My application is not running on the name node. It is remote. The 2.0
>> Yarn API look like they may be useful, but I am not on 2.0 and cannot move
>> to 2,0 anytime soon.****
>>
>> ** **
>>
>> DFSClient.java looks useful, but its not in the API docs so I am not sure
>> how to use it or even if I should.****
>>
>> Any pointers would be helpful. ****
>>
>> ** **
>>
>> Thanks,****
>>
>
>
--
http://hortonworks.com/download/
Re: Management API
Posted by Suresh Srinivas <su...@hortonworks.com>.
Namenode exposes all these information over JMX. The same is also exposed
over http, which you could use for your use case.
See http://<nn:httpport>/jmx to list all the properties exposed. Typically
http port is 50070. You will find the information you looking for at:
http://<nn:httpport>/jmx?Hadoop:service=NameNode,name=NameNodeInfo
On Thu, Jun 6, 2013 at 9:17 AM, Brian Mason <br...@gabey.com> wrote:
> Mostly looking for a list of data nodes. I am making web page to display
> some info about data nodes and I need a list. I can screen scrape the JSP
> GUI, I was just hoping for something more elegant.
>
>
>
> On Thu, Jun 6, 2013 at 8:40 AM, John Lilley <jo...@redpoint.net>wrote:
>
>> What resources are you trying to access? ****
>>
>> Do you want to monitor the system status?****
>>
>> Do you want to read/write HDFS as a client? ****
>>
>> Do you want to run your application on the Hadoop cluster? ****
>>
>> John****
>>
>> ** **
>>
>> ** **
>>
>> *From:* Brian Mason [mailto:brian@gabey.com]
>> *Sent:* Thursday, June 06, 2013 6:52 AM
>> *To:* user@hadoop.apache.org
>> *Subject:* Management API****
>>
>> ** **
>>
>> I am looking for a way to access a list of Nodes, Compute, Data etc ..
>> My application is not running on the name node. It is remote. The 2.0
>> Yarn API look like they may be useful, but I am not on 2.0 and cannot move
>> to 2,0 anytime soon.****
>>
>> ** **
>>
>> DFSClient.java looks useful, but its not in the API docs so I am not sure
>> how to use it or even if I should.****
>>
>> Any pointers would be helpful. ****
>>
>> ** **
>>
>> Thanks,****
>>
>
>
--
http://hortonworks.com/download/
Re: Management API
Posted by Brian Mason <br...@gabey.com>.
Mostly looking for a list of data nodes. I am making web page to display
some info about data nodes and I need a list. I can screen scrape the JSP
GUI, I was just hoping for something more elegant.
On Thu, Jun 6, 2013 at 8:40 AM, John Lilley <jo...@redpoint.net>wrote:
> What resources are you trying to access? ****
>
> Do you want to monitor the system status?****
>
> Do you want to read/write HDFS as a client? ****
>
> Do you want to run your application on the Hadoop cluster? ****
>
> John****
>
> ** **
>
> ** **
>
> *From:* Brian Mason [mailto:brian@gabey.com]
> *Sent:* Thursday, June 06, 2013 6:52 AM
> *To:* user@hadoop.apache.org
> *Subject:* Management API****
>
> ** **
>
> I am looking for a way to access a list of Nodes, Compute, Data etc .. My
> application is not running on the name node. It is remote. The 2.0 Yarn
> API look like they may be useful, but I am not on 2.0 and cannot move to
> 2,0 anytime soon.****
>
> ** **
>
> DFSClient.java looks useful, but its not in the API docs so I am not sure
> how to use it or even if I should.****
>
> Any pointers would be helpful. ****
>
> ** **
>
> Thanks,****
>
Re: Management API
Posted by Brian Mason <br...@gabey.com>.
Mostly looking for a list of data nodes. I am making web page to display
some info about data nodes and I need a list. I can screen scrape the JSP
GUI, I was just hoping for something more elegant.
On Thu, Jun 6, 2013 at 8:40 AM, John Lilley <jo...@redpoint.net>wrote:
> What resources are you trying to access? ****
>
> Do you want to monitor the system status?****
>
> Do you want to read/write HDFS as a client? ****
>
> Do you want to run your application on the Hadoop cluster? ****
>
> John****
>
> ** **
>
> ** **
>
> *From:* Brian Mason [mailto:brian@gabey.com]
> *Sent:* Thursday, June 06, 2013 6:52 AM
> *To:* user@hadoop.apache.org
> *Subject:* Management API****
>
> ** **
>
> I am looking for a way to access a list of Nodes, Compute, Data etc .. My
> application is not running on the name node. It is remote. The 2.0 Yarn
> API look like they may be useful, but I am not on 2.0 and cannot move to
> 2,0 anytime soon.****
>
> ** **
>
> DFSClient.java looks useful, but its not in the API docs so I am not sure
> how to use it or even if I should.****
>
> Any pointers would be helpful. ****
>
> ** **
>
> Thanks,****
>
Re: Management API
Posted by Brian Mason <br...@gabey.com>.
Mostly looking for a list of data nodes. I am making web page to display
some info about data nodes and I need a list. I can screen scrape the JSP
GUI, I was just hoping for something more elegant.
On Thu, Jun 6, 2013 at 8:40 AM, John Lilley <jo...@redpoint.net>wrote:
> What resources are you trying to access? ****
>
> Do you want to monitor the system status?****
>
> Do you want to read/write HDFS as a client? ****
>
> Do you want to run your application on the Hadoop cluster? ****
>
> John****
>
> ** **
>
> ** **
>
> *From:* Brian Mason [mailto:brian@gabey.com]
> *Sent:* Thursday, June 06, 2013 6:52 AM
> *To:* user@hadoop.apache.org
> *Subject:* Management API****
>
> ** **
>
> I am looking for a way to access a list of Nodes, Compute, Data etc .. My
> application is not running on the name node. It is remote. The 2.0 Yarn
> API look like they may be useful, but I am not on 2.0 and cannot move to
> 2,0 anytime soon.****
>
> ** **
>
> DFSClient.java looks useful, but its not in the API docs so I am not sure
> how to use it or even if I should.****
>
> Any pointers would be helpful. ****
>
> ** **
>
> Thanks,****
>
Re: Management API
Posted by Brian Mason <br...@gabey.com>.
Mostly looking for a list of data nodes. I am making web page to display
some info about data nodes and I need a list. I can screen scrape the JSP
GUI, I was just hoping for something more elegant.
On Thu, Jun 6, 2013 at 8:40 AM, John Lilley <jo...@redpoint.net>wrote:
> What resources are you trying to access? ****
>
> Do you want to monitor the system status?****
>
> Do you want to read/write HDFS as a client? ****
>
> Do you want to run your application on the Hadoop cluster? ****
>
> John****
>
> ** **
>
> ** **
>
> *From:* Brian Mason [mailto:brian@gabey.com]
> *Sent:* Thursday, June 06, 2013 6:52 AM
> *To:* user@hadoop.apache.org
> *Subject:* Management API****
>
> ** **
>
> I am looking for a way to access a list of Nodes, Compute, Data etc .. My
> application is not running on the name node. It is remote. The 2.0 Yarn
> API look like they may be useful, but I am not on 2.0 and cannot move to
> 2,0 anytime soon.****
>
> ** **
>
> DFSClient.java looks useful, but its not in the API docs so I am not sure
> how to use it or even if I should.****
>
> Any pointers would be helpful. ****
>
> ** **
>
> Thanks,****
>
RE: Management API
Posted by John Lilley <jo...@redpoint.net>.
What resources are you trying to access?
Do you want to monitor the system status?
Do you want to read/write HDFS as a client?
Do you want to run your application on the Hadoop cluster?
John
From: Brian Mason [mailto:brian@gabey.com]
Sent: Thursday, June 06, 2013 6:52 AM
To: user@hadoop.apache.org
Subject: Management API
I am looking for a way to access a list of Nodes, Compute, Data etc .. My application is not running on the name node. It is remote. The 2.0 Yarn API look like they may be useful, but I am not on 2.0 and cannot move to 2,0 anytime soon.
DFSClient.java looks useful, but its not in the API docs so I am not sure how to use it or even if I should.
Any pointers would be helpful.
Thanks,
Re: Management API
Posted by Arun C Murthy <ac...@hortonworks.com>.
You might want to look at Apache Ambari - it's got a pretty comprehensive set of APIs of the kind you seek.
https://github.com/apache/ambari/blob/trunk/ambari-server/docs/api/v1/index.md#resources
Arun
On Jun 6, 2013, at 5:51 AM, Brian Mason wrote:
> I am looking for a way to access a list of Nodes, Compute, Data etc .. My application is not running on the name node. It is remote. The 2.0 Yarn API look like they may be useful, but I am not on 2.0 and cannot move to 2,0 anytime soon.
>
> DFSClient.java looks useful, but its not in the API docs so I am not sure how to use it or even if I should.
> Any pointers would be helpful.
>
> Thanks,
--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/
Re: Management API
Posted by MARCOS MEDRADO RUBINELLI <ma...@buscapecompany.com>.
Rita,
There aren't any specs as far as I know, but in my experience the interface is stable enough from version to version, with the occasional extra field added here or there. If you query specifically for the beans you want (e.g. http://namenode:50070/jmx?get=Hadoop:service=NameNode,name=NameNodeInfo::LiveNodes ) and build in some flexibility, you shouldn't have any problems.
Regards,
Marcos
On 09-06-2013 11:30, Rita wrote:
Are there any specs for the JSON schema?
On Thu, Jun 6, 2013 at 9:49 AM, MARCOS MEDRADO RUBINELLI <ma...@buscapecompany.com>> wrote:
Brian,
If you have access to the web UI, you can get those metrics in JSON from the JMXJsonServlet. Try hitting http://namenode_hostname:50070/jmx?qry=Hadoop:* and http://jobtracker_v1_hostname:50030/jmx?qry=hadoop:*
It isn't as extensive as other options, but if you just need a snapshot of node capacity and utilization, it's pretty handy. I used it to plug some basic warnings into Nagios.
Regards,
Marcos
On 06-06-2013 09:51, Brian Mason wrote:
I am looking for a way to access a list of Nodes, Compute, Data etc .. My application is not running on the name node. It is remote. The 2.0 Yarn API look like they may be useful, but I am not on 2.0 and cannot move to 2,0 anytime soon.
DFSClient.java looks useful, but its not in the API docs so I am not sure how to use it or even if I should.
Any pointers would be helpful.
Thanks,
--
--- Get your facts first, then you can distort them as you please.--
Re: Management API
Posted by MARCOS MEDRADO RUBINELLI <ma...@buscapecompany.com>.
Rita,
There aren't any specs as far as I know, but in my experience the interface is stable enough from version to version, with the occasional extra field added here or there. If you query specifically for the beans you want (e.g. http://namenode:50070/jmx?get=Hadoop:service=NameNode,name=NameNodeInfo::LiveNodes ) and build in some flexibility, you shouldn't have any problems.
Regards,
Marcos
On 09-06-2013 11:30, Rita wrote:
Are there any specs for the JSON schema?
On Thu, Jun 6, 2013 at 9:49 AM, MARCOS MEDRADO RUBINELLI <ma...@buscapecompany.com>> wrote:
Brian,
If you have access to the web UI, you can get those metrics in JSON from the JMXJsonServlet. Try hitting http://namenode_hostname:50070/jmx?qry=Hadoop:* and http://jobtracker_v1_hostname:50030/jmx?qry=hadoop:*
It isn't as extensive as other options, but if you just need a snapshot of node capacity and utilization, it's pretty handy. I used it to plug some basic warnings into Nagios.
Regards,
Marcos
On 06-06-2013 09:51, Brian Mason wrote:
I am looking for a way to access a list of Nodes, Compute, Data etc .. My application is not running on the name node. It is remote. The 2.0 Yarn API look like they may be useful, but I am not on 2.0 and cannot move to 2,0 anytime soon.
DFSClient.java looks useful, but its not in the API docs so I am not sure how to use it or even if I should.
Any pointers would be helpful.
Thanks,
--
--- Get your facts first, then you can distort them as you please.--
Re: Management API
Posted by MARCOS MEDRADO RUBINELLI <ma...@buscapecompany.com>.
Rita,
There aren't any specs as far as I know, but in my experience the interface is stable enough from version to version, with the occasional extra field added here or there. If you query specifically for the beans you want (e.g. http://namenode:50070/jmx?get=Hadoop:service=NameNode,name=NameNodeInfo::LiveNodes ) and build in some flexibility, you shouldn't have any problems.
Regards,
Marcos
On 09-06-2013 11:30, Rita wrote:
Are there any specs for the JSON schema?
On Thu, Jun 6, 2013 at 9:49 AM, MARCOS MEDRADO RUBINELLI <ma...@buscapecompany.com>> wrote:
Brian,
If you have access to the web UI, you can get those metrics in JSON from the JMXJsonServlet. Try hitting http://namenode_hostname:50070/jmx?qry=Hadoop:* and http://jobtracker_v1_hostname:50030/jmx?qry=hadoop:*
It isn't as extensive as other options, but if you just need a snapshot of node capacity and utilization, it's pretty handy. I used it to plug some basic warnings into Nagios.
Regards,
Marcos
On 06-06-2013 09:51, Brian Mason wrote:
I am looking for a way to access a list of Nodes, Compute, Data etc .. My application is not running on the name node. It is remote. The 2.0 Yarn API look like they may be useful, but I am not on 2.0 and cannot move to 2,0 anytime soon.
DFSClient.java looks useful, but its not in the API docs so I am not sure how to use it or even if I should.
Any pointers would be helpful.
Thanks,
--
--- Get your facts first, then you can distort them as you please.--
Re: Management API
Posted by MARCOS MEDRADO RUBINELLI <ma...@buscapecompany.com>.
Rita,
There aren't any specs as far as I know, but in my experience the interface is stable enough from version to version, with the occasional extra field added here or there. If you query specifically for the beans you want (e.g. http://namenode:50070/jmx?get=Hadoop:service=NameNode,name=NameNodeInfo::LiveNodes ) and build in some flexibility, you shouldn't have any problems.
Regards,
Marcos
On 09-06-2013 11:30, Rita wrote:
Are there any specs for the JSON schema?
On Thu, Jun 6, 2013 at 9:49 AM, MARCOS MEDRADO RUBINELLI <ma...@buscapecompany.com>> wrote:
Brian,
If you have access to the web UI, you can get those metrics in JSON from the JMXJsonServlet. Try hitting http://namenode_hostname:50070/jmx?qry=Hadoop:* and http://jobtracker_v1_hostname:50030/jmx?qry=hadoop:*
It isn't as extensive as other options, but if you just need a snapshot of node capacity and utilization, it's pretty handy. I used it to plug some basic warnings into Nagios.
Regards,
Marcos
On 06-06-2013 09:51, Brian Mason wrote:
I am looking for a way to access a list of Nodes, Compute, Data etc .. My application is not running on the name node. It is remote. The 2.0 Yarn API look like they may be useful, but I am not on 2.0 and cannot move to 2,0 anytime soon.
DFSClient.java looks useful, but its not in the API docs so I am not sure how to use it or even if I should.
Any pointers would be helpful.
Thanks,
--
--- Get your facts first, then you can distort them as you please.--
Re: Management API
Posted by Rita <rm...@gmail.com>.
Are there any specs for the JSON schema?
On Thu, Jun 6, 2013 at 9:49 AM, MARCOS MEDRADO RUBINELLI <
marcosm@buscapecompany.com> wrote:
> Brian,
>
> If you have access to the web UI, you can get those metrics in JSON from
> the JMXJsonServlet. Try hitting
> http://namenode_hostname:50070/jmx?qry=Hadoop:* and
> http://jobtracker_v1_hostname:50030/jmx?qry=hadoop:*
>
> It isn't as extensive as other options, but if you just need a snapshot of
> node capacity and utilization, it's pretty handy. I used it to plug some
> basic warnings into Nagios.
>
> Regards,
> Marcos
>
>
> On 06-06-2013 09:51, Brian Mason wrote:
>
> I am looking for a way to access a list of Nodes, Compute, Data etc .. My
> application is not running on the name node. It is remote. The 2.0 Yarn
> API look like they may be useful, but I am not on 2.0 and cannot move to
> 2,0 anytime soon.
>
> DFSClient.java looks useful, but its not in the API docs so I am not
> sure how to use it or even if I should.
> Any pointers would be helpful.
>
> Thanks,
>
>
>
--
--- Get your facts first, then you can distort them as you please.--
Re: Management API
Posted by Rita <rm...@gmail.com>.
Are there any specs for the JSON schema?
On Thu, Jun 6, 2013 at 9:49 AM, MARCOS MEDRADO RUBINELLI <
marcosm@buscapecompany.com> wrote:
> Brian,
>
> If you have access to the web UI, you can get those metrics in JSON from
> the JMXJsonServlet. Try hitting
> http://namenode_hostname:50070/jmx?qry=Hadoop:* and
> http://jobtracker_v1_hostname:50030/jmx?qry=hadoop:*
>
> It isn't as extensive as other options, but if you just need a snapshot of
> node capacity and utilization, it's pretty handy. I used it to plug some
> basic warnings into Nagios.
>
> Regards,
> Marcos
>
>
> On 06-06-2013 09:51, Brian Mason wrote:
>
> I am looking for a way to access a list of Nodes, Compute, Data etc .. My
> application is not running on the name node. It is remote. The 2.0 Yarn
> API look like they may be useful, but I am not on 2.0 and cannot move to
> 2,0 anytime soon.
>
> DFSClient.java looks useful, but its not in the API docs so I am not
> sure how to use it or even if I should.
> Any pointers would be helpful.
>
> Thanks,
>
>
>
--
--- Get your facts first, then you can distort them as you please.--
Re: Management API
Posted by Rita <rm...@gmail.com>.
Are there any specs for the JSON schema?
On Thu, Jun 6, 2013 at 9:49 AM, MARCOS MEDRADO RUBINELLI <
marcosm@buscapecompany.com> wrote:
> Brian,
>
> If you have access to the web UI, you can get those metrics in JSON from
> the JMXJsonServlet. Try hitting
> http://namenode_hostname:50070/jmx?qry=Hadoop:* and
> http://jobtracker_v1_hostname:50030/jmx?qry=hadoop:*
>
> It isn't as extensive as other options, but if you just need a snapshot of
> node capacity and utilization, it's pretty handy. I used it to plug some
> basic warnings into Nagios.
>
> Regards,
> Marcos
>
>
> On 06-06-2013 09:51, Brian Mason wrote:
>
> I am looking for a way to access a list of Nodes, Compute, Data etc .. My
> application is not running on the name node. It is remote. The 2.0 Yarn
> API look like they may be useful, but I am not on 2.0 and cannot move to
> 2,0 anytime soon.
>
> DFSClient.java looks useful, but its not in the API docs so I am not
> sure how to use it or even if I should.
> Any pointers would be helpful.
>
> Thanks,
>
>
>
--
--- Get your facts first, then you can distort them as you please.--
Re: Management API
Posted by Rita <rm...@gmail.com>.
Are there any specs for the JSON schema?
On Thu, Jun 6, 2013 at 9:49 AM, MARCOS MEDRADO RUBINELLI <
marcosm@buscapecompany.com> wrote:
> Brian,
>
> If you have access to the web UI, you can get those metrics in JSON from
> the JMXJsonServlet. Try hitting
> http://namenode_hostname:50070/jmx?qry=Hadoop:* and
> http://jobtracker_v1_hostname:50030/jmx?qry=hadoop:*
>
> It isn't as extensive as other options, but if you just need a snapshot of
> node capacity and utilization, it's pretty handy. I used it to plug some
> basic warnings into Nagios.
>
> Regards,
> Marcos
>
>
> On 06-06-2013 09:51, Brian Mason wrote:
>
> I am looking for a way to access a list of Nodes, Compute, Data etc .. My
> application is not running on the name node. It is remote. The 2.0 Yarn
> API look like they may be useful, but I am not on 2.0 and cannot move to
> 2,0 anytime soon.
>
> DFSClient.java looks useful, but its not in the API docs so I am not
> sure how to use it or even if I should.
> Any pointers would be helpful.
>
> Thanks,
>
>
>
--
--- Get your facts first, then you can distort them as you please.--
Re: Management API
Posted by MARCOS MEDRADO RUBINELLI <ma...@buscapecompany.com>.
Brian,
If you have access to the web UI, you can get those metrics in JSON from the JMXJsonServlet. Try hitting http://namenode_hostname:50070/jmx?qry=Hadoop:* and http://jobtracker_v1_hostname:50030/jmx?qry=hadoop:*
It isn't as extensive as other options, but if you just need a snapshot of node capacity and utilization, it's pretty handy. I used it to plug some basic warnings into Nagios.
Regards,
Marcos
On 06-06-2013 09:51, Brian Mason wrote:
I am looking for a way to access a list of Nodes, Compute, Data etc .. My application is not running on the name node. It is remote. The 2.0 Yarn API look like they may be useful, but I am not on 2.0 and cannot move to 2,0 anytime soon.
DFSClient.java looks useful, but its not in the API docs so I am not sure how to use it or even if I should.
Any pointers would be helpful.
Thanks,
Re: Management API
Posted by MARCOS MEDRADO RUBINELLI <ma...@buscapecompany.com>.
Brian,
If you have access to the web UI, you can get those metrics in JSON from the JMXJsonServlet. Try hitting http://namenode_hostname:50070/jmx?qry=Hadoop:* and http://jobtracker_v1_hostname:50030/jmx?qry=hadoop:*
It isn't as extensive as other options, but if you just need a snapshot of node capacity and utilization, it's pretty handy. I used it to plug some basic warnings into Nagios.
Regards,
Marcos
On 06-06-2013 09:51, Brian Mason wrote:
I am looking for a way to access a list of Nodes, Compute, Data etc .. My application is not running on the name node. It is remote. The 2.0 Yarn API look like they may be useful, but I am not on 2.0 and cannot move to 2,0 anytime soon.
DFSClient.java looks useful, but its not in the API docs so I am not sure how to use it or even if I should.
Any pointers would be helpful.
Thanks,
Re: Management API
Posted by MARCOS MEDRADO RUBINELLI <ma...@buscapecompany.com>.
Brian,
If you have access to the web UI, you can get those metrics in JSON from the JMXJsonServlet. Try hitting http://namenode_hostname:50070/jmx?qry=Hadoop:* and http://jobtracker_v1_hostname:50030/jmx?qry=hadoop:*
It isn't as extensive as other options, but if you just need a snapshot of node capacity and utilization, it's pretty handy. I used it to plug some basic warnings into Nagios.
Regards,
Marcos
On 06-06-2013 09:51, Brian Mason wrote:
I am looking for a way to access a list of Nodes, Compute, Data etc .. My application is not running on the name node. It is remote. The 2.0 Yarn API look like they may be useful, but I am not on 2.0 and cannot move to 2,0 anytime soon.
DFSClient.java looks useful, but its not in the API docs so I am not sure how to use it or even if I should.
Any pointers would be helpful.
Thanks,
Re: Management API
Posted by Arun C Murthy <ac...@hortonworks.com>.
You might want to look at Apache Ambari - it's got a pretty comprehensive set of APIs of the kind you seek.
https://github.com/apache/ambari/blob/trunk/ambari-server/docs/api/v1/index.md#resources
Arun
On Jun 6, 2013, at 5:51 AM, Brian Mason wrote:
> I am looking for a way to access a list of Nodes, Compute, Data etc .. My application is not running on the name node. It is remote. The 2.0 Yarn API look like they may be useful, but I am not on 2.0 and cannot move to 2,0 anytime soon.
>
> DFSClient.java looks useful, but its not in the API docs so I am not sure how to use it or even if I should.
> Any pointers would be helpful.
>
> Thanks,
--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/
RE: Management API
Posted by John Lilley <jo...@redpoint.net>.
What resources are you trying to access?
Do you want to monitor the system status?
Do you want to read/write HDFS as a client?
Do you want to run your application on the Hadoop cluster?
John
From: Brian Mason [mailto:brian@gabey.com]
Sent: Thursday, June 06, 2013 6:52 AM
To: user@hadoop.apache.org
Subject: Management API
I am looking for a way to access a list of Nodes, Compute, Data etc .. My application is not running on the name node. It is remote. The 2.0 Yarn API look like they may be useful, but I am not on 2.0 and cannot move to 2,0 anytime soon.
DFSClient.java looks useful, but its not in the API docs so I am not sure how to use it or even if I should.
Any pointers would be helpful.
Thanks,
Re: Management API
Posted by MARCOS MEDRADO RUBINELLI <ma...@buscapecompany.com>.
Brian,
If you have access to the web UI, you can get those metrics in JSON from the JMXJsonServlet. Try hitting http://namenode_hostname:50070/jmx?qry=Hadoop:* and http://jobtracker_v1_hostname:50030/jmx?qry=hadoop:*
It isn't as extensive as other options, but if you just need a snapshot of node capacity and utilization, it's pretty handy. I used it to plug some basic warnings into Nagios.
Regards,
Marcos
On 06-06-2013 09:51, Brian Mason wrote:
I am looking for a way to access a list of Nodes, Compute, Data etc .. My application is not running on the name node. It is remote. The 2.0 Yarn API look like they may be useful, but I am not on 2.0 and cannot move to 2,0 anytime soon.
DFSClient.java looks useful, but its not in the API docs so I am not sure how to use it or even if I should.
Any pointers would be helpful.
Thanks,
Re: Management API
Posted by Arun C Murthy <ac...@hortonworks.com>.
You might want to look at Apache Ambari - it's got a pretty comprehensive set of APIs of the kind you seek.
https://github.com/apache/ambari/blob/trunk/ambari-server/docs/api/v1/index.md#resources
Arun
On Jun 6, 2013, at 5:51 AM, Brian Mason wrote:
> I am looking for a way to access a list of Nodes, Compute, Data etc .. My application is not running on the name node. It is remote. The 2.0 Yarn API look like they may be useful, but I am not on 2.0 and cannot move to 2,0 anytime soon.
>
> DFSClient.java looks useful, but its not in the API docs so I am not sure how to use it or even if I should.
> Any pointers would be helpful.
>
> Thanks,
--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/
Re: Management API
Posted by Arun C Murthy <ac...@hortonworks.com>.
You might want to look at Apache Ambari - it's got a pretty comprehensive set of APIs of the kind you seek.
https://github.com/apache/ambari/blob/trunk/ambari-server/docs/api/v1/index.md#resources
Arun
On Jun 6, 2013, at 5:51 AM, Brian Mason wrote:
> I am looking for a way to access a list of Nodes, Compute, Data etc .. My application is not running on the name node. It is remote. The 2.0 Yarn API look like they may be useful, but I am not on 2.0 and cannot move to 2,0 anytime soon.
>
> DFSClient.java looks useful, but its not in the API docs so I am not sure how to use it or even if I should.
> Any pointers would be helpful.
>
> Thanks,
--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/
RE: Management API
Posted by John Lilley <jo...@redpoint.net>.
What resources are you trying to access?
Do you want to monitor the system status?
Do you want to read/write HDFS as a client?
Do you want to run your application on the Hadoop cluster?
John
From: Brian Mason [mailto:brian@gabey.com]
Sent: Thursday, June 06, 2013 6:52 AM
To: user@hadoop.apache.org
Subject: Management API
I am looking for a way to access a list of Nodes, Compute, Data etc .. My application is not running on the name node. It is remote. The 2.0 Yarn API look like they may be useful, but I am not on 2.0 and cannot move to 2,0 anytime soon.
DFSClient.java looks useful, but its not in the API docs so I am not sure how to use it or even if I should.
Any pointers would be helpful.
Thanks,
RE: Management API
Posted by John Lilley <jo...@redpoint.net>.
What resources are you trying to access?
Do you want to monitor the system status?
Do you want to read/write HDFS as a client?
Do you want to run your application on the Hadoop cluster?
John
From: Brian Mason [mailto:brian@gabey.com]
Sent: Thursday, June 06, 2013 6:52 AM
To: user@hadoop.apache.org
Subject: Management API
I am looking for a way to access a list of Nodes, Compute, Data etc .. My application is not running on the name node. It is remote. The 2.0 Yarn API look like they may be useful, but I am not on 2.0 and cannot move to 2,0 anytime soon.
DFSClient.java looks useful, but its not in the API docs so I am not sure how to use it or even if I should.
Any pointers would be helpful.
Thanks,