You are viewing a plain text version of this content. The canonical link for it is here.
Posted to yarn-dev@hadoop.apache.org by Charlie Wright <ch...@live.ca> on 2016/03/12 21:00:43 UTC
Question about QueueMetrics/Cluster Metrics
Is there a way to get the root QueueMetrics on the client side?-Charles.
Re: Question about QueueMetrics/Cluster Metrics
Posted by Nathan Roberts <nr...@yahoo-inc.com.INVALID>.
Yes, YarnClient should get you access to current capacity information for the queues.
Nathan
On Saturday, March 12, 2016 2:37 PM, Charlie Wright <ch...@live.ca> wrote:
This may sound dumb, but I am not sure how to go about using the REST API. Is there a guide somewhere?
Also, I found that the YarnClientImpl has a getRootQueueInfos() method that returns a List<QueueInfo> which you can call getQueueStatistics() on - which returns QueueStatistics, which may give me what I need w/o the REST API (it seemingly includes all the metrics that QueueMetrics does). Can you confirm whether this works/is what I am looking for?
Thanks for the prompt response.
Charles.
> Date: Sat, 12 Mar 2016 20:22:40 +0000
> From: nroberts@yahoo-inc.com.INVALID
> To: yarn-dev@hadoop.apache.org
> Subject: Re: Question about QueueMetrics/Cluster Metrics
>
> Would the REST API get you what you need? https://hadoop.apache.org/docs/r2.7.2/hadoop-yarn/hadoop-yarn-site/ResourceManagerRest.html
> Nathan
>
>
> On Saturday, March 12, 2016 2:11 PM, Charlie Wright <ch...@live.ca> wrote:
>
>
> Well, what I have been trying to is send a "Cluster Pressure Event" to the Spark Applications running on a cluster to allow for dynamic re-sizing of Spark applications running on the cluster. It is a 4th year project that I am working on for my university program. We have the Spark side implemented where spark applications are set to have a fair share value for the number of executors along with the already implemented max and min values. This way, a spark app will grow until it has it's max and then throttle back to it's fair share if a Cluster Pressure Event is triggered. The motivation behind this, is to allow for interactive jobs to come onto the cluster - so that they don't have to wait for the other jobs to complete. Currently the only way for a spark application to "shrink down" is if the executors time out for inactivity. We have defined cluster pressure as (pending resources + allocated resources / total resources > 1) meaning that there has been more requests for resources than what is available - so we need to release some.
> I had attempted previously to do this inside the resource manager - but I could not figure out how to get events to the client from the RM. So now I am trying to do this client side, but I don't have access to this data (all I need is available memory, pending memory, and total memory available - or something similar).
> If you could provide some insight it would be greatly appreciated. I have been having trouble with this for months now.
> Thanks,Charles.
>
> > From: daniel@cloudera.com
> > To: yarn-dev@hadoop.apache.org
> > CC: charliewright@live.ca
> > Subject: Re: Question about QueueMetrics/Cluster Metrics
> > Date: Sat, 12 Mar 2016 12:05:52 -0800
> >
> > On 3/12/16 12:00 PM, Charlie Wright wrote:
> > > Is there a way to get the root QueueMetrics on the client side?-Charles.
> >
> > The QueueMetrics objects aren't exposed outside of the RM. What are you
> > trying to accomplish? If you just want to access the metrics data, you
> > can configure a RollingFileSystemSink to write the metrics into HDFS so
> > you can pick them up.
> >
> > Daniel
>
>
>
RE: Question about QueueMetrics/Cluster Metrics
Posted by Charlie Wright <ch...@live.ca>.
This may sound dumb, but I am not sure how to go about using the REST API. Is there a guide somewhere?
Also, I found that the YarnClientImpl has a getRootQueueInfos() method that returns a List<QueueInfo> which you can call getQueueStatistics() on - which returns QueueStatistics, which may give me what I need w/o the REST API (it seemingly includes all the metrics that QueueMetrics does). Can you confirm whether this works/is what I am looking for?
Thanks for the prompt response.
Charles.
> Date: Sat, 12 Mar 2016 20:22:40 +0000
> From: nroberts@yahoo-inc.com.INVALID
> To: yarn-dev@hadoop.apache.org
> Subject: Re: Question about QueueMetrics/Cluster Metrics
>
> Would the REST API get you what you need? https://hadoop.apache.org/docs/r2.7.2/hadoop-yarn/hadoop-yarn-site/ResourceManagerRest.html
> Nathan
>
>
> On Saturday, March 12, 2016 2:11 PM, Charlie Wright <ch...@live.ca> wrote:
>
>
> Well, what I have been trying to is send a "Cluster Pressure Event" to the Spark Applications running on a cluster to allow for dynamic re-sizing of Spark applications running on the cluster. It is a 4th year project that I am working on for my university program. We have the Spark side implemented where spark applications are set to have a fair share value for the number of executors along with the already implemented max and min values. This way, a spark app will grow until it has it's max and then throttle back to it's fair share if a Cluster Pressure Event is triggered. The motivation behind this, is to allow for interactive jobs to come onto the cluster - so that they don't have to wait for the other jobs to complete. Currently the only way for a spark application to "shrink down" is if the executors time out for inactivity. We have defined cluster pressure as (pending resources + allocated resources / total resources > 1) meaning that there has been more requests for resources than what is available - so we need to release some.
> I had attempted previously to do this inside the resource manager - but I could not figure out how to get events to the client from the RM. So now I am trying to do this client side, but I don't have access to this data (all I need is available memory, pending memory, and total memory available - or something similar).
> If you could provide some insight it would be greatly appreciated. I have been having trouble with this for months now.
> Thanks,Charles.
>
> > From: daniel@cloudera.com
> > To: yarn-dev@hadoop.apache.org
> > CC: charliewright@live.ca
> > Subject: Re: Question about QueueMetrics/Cluster Metrics
> > Date: Sat, 12 Mar 2016 12:05:52 -0800
> >
> > On 3/12/16 12:00 PM, Charlie Wright wrote:
> > > Is there a way to get the root QueueMetrics on the client side?-Charles.
> >
> > The QueueMetrics objects aren't exposed outside of the RM. What are you
> > trying to accomplish? If you just want to access the metrics data, you
> > can configure a RollingFileSystemSink to write the metrics into HDFS so
> > you can pick them up.
> >
> > Daniel
>
>
>
Re: Question about QueueMetrics/Cluster Metrics
Posted by Nathan Roberts <nr...@yahoo-inc.com.INVALID>.
Would the REST API get you what you need? https://hadoop.apache.org/docs/r2.7.2/hadoop-yarn/hadoop-yarn-site/ResourceManagerRest.html
Nathan
On Saturday, March 12, 2016 2:11 PM, Charlie Wright <ch...@live.ca> wrote:
Well, what I have been trying to is send a "Cluster Pressure Event" to the Spark Applications running on a cluster to allow for dynamic re-sizing of Spark applications running on the cluster. It is a 4th year project that I am working on for my university program. We have the Spark side implemented where spark applications are set to have a fair share value for the number of executors along with the already implemented max and min values. This way, a spark app will grow until it has it's max and then throttle back to it's fair share if a Cluster Pressure Event is triggered. The motivation behind this, is to allow for interactive jobs to come onto the cluster - so that they don't have to wait for the other jobs to complete. Currently the only way for a spark application to "shrink down" is if the executors time out for inactivity. We have defined cluster pressure as (pending resources + allocated resources / total resources > 1) meaning that there has been more requests for resources than what is available - so we need to release some.
I had attempted previously to do this inside the resource manager - but I could not figure out how to get events to the client from the RM. So now I am trying to do this client side, but I don't have access to this data (all I need is available memory, pending memory, and total memory available - or something similar).
If you could provide some insight it would be greatly appreciated. I have been having trouble with this for months now.
Thanks,Charles.
> From: daniel@cloudera.com
> To: yarn-dev@hadoop.apache.org
> CC: charliewright@live.ca
> Subject: Re: Question about QueueMetrics/Cluster Metrics
> Date: Sat, 12 Mar 2016 12:05:52 -0800
>
> On 3/12/16 12:00 PM, Charlie Wright wrote:
> > Is there a way to get the root QueueMetrics on the client side?-Charles.
>
> The QueueMetrics objects aren't exposed outside of the RM. What are you
> trying to accomplish? If you just want to access the metrics data, you
> can configure a RollingFileSystemSink to write the metrics into HDFS so
> you can pick them up.
>
> Daniel
RE: Question about QueueMetrics/Cluster Metrics
Posted by Charlie Wright <ch...@live.ca>.
Well, what I have been trying to is send a "Cluster Pressure Event" to the Spark Applications running on a cluster to allow for dynamic re-sizing of Spark applications running on the cluster. It is a 4th year project that I am working on for my university program. We have the Spark side implemented where spark applications are set to have a fair share value for the number of executors along with the already implemented max and min values. This way, a spark app will grow until it has it's max and then throttle back to it's fair share if a Cluster Pressure Event is triggered. The motivation behind this, is to allow for interactive jobs to come onto the cluster - so that they don't have to wait for the other jobs to complete. Currently the only way for a spark application to "shrink down" is if the executors time out for inactivity. We have defined cluster pressure as (pending resources + allocated resources / total resources > 1) meaning that there has been more requests for resources than what is available - so we need to release some.
I had attempted previously to do this inside the resource manager - but I could not figure out how to get events to the client from the RM. So now I am trying to do this client side, but I don't have access to this data (all I need is available memory, pending memory, and total memory available - or something similar).
If you could provide some insight it would be greatly appreciated. I have been having trouble with this for months now.
Thanks,Charles.
> From: daniel@cloudera.com
> To: yarn-dev@hadoop.apache.org
> CC: charliewright@live.ca
> Subject: Re: Question about QueueMetrics/Cluster Metrics
> Date: Sat, 12 Mar 2016 12:05:52 -0800
>
> On 3/12/16 12:00 PM, Charlie Wright wrote:
> > Is there a way to get the root QueueMetrics on the client side?-Charles.
>
> The QueueMetrics objects aren't exposed outside of the RM. What are you
> trying to accomplish? If you just want to access the metrics data, you
> can configure a RollingFileSystemSink to write the metrics into HDFS so
> you can pick them up.
>
> Daniel
Re: Question about QueueMetrics/Cluster Metrics
Posted by Daniel Templeton <da...@cloudera.com>.
On 3/12/16 12:00 PM, Charlie Wright wrote:
> Is there a way to get the root QueueMetrics on the client side?-Charles.
The QueueMetrics objects aren't exposed outside of the RM. What are you
trying to accomplish? If you just want to access the metrics data, you
can configure a RollingFileSystemSink to write the metrics into HDFS so
you can pick them up.
Daniel