You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Matt K <ma...@gmail.com> on 2016/02/03 14:32:57 UTC
spark metrics question
Hi guys,
I'm looking to create a custom sync based on Spark's Metrics System:
https://github.com/apache/spark/blob/9f603fce78fcc997926e9a72dec44d48cbc396fc/core/src/main/scala/org/apache/spark/metrics/MetricsSystem.scala
If I want to collect metrics from the Driver, Master, and Executor nodes,
should the jar with the custom class be installed on Driver, Master, and
Executor nodes?
Also, on Executor nodes, does the MetricsSystem run inside the Executor's
JVM?
Thanks,
-Matt
Re: spark metrics question
Posted by Matt K <ma...@gmail.com>.
Thanks Takeshi, that's exactly what I was looking for.
On Fri, Feb 5, 2016 at 12:32 PM, Takeshi Yamamuro <li...@gmail.com>
wrote:
> How about using `spark.jars` to send jars into a cluster?
>
> On Sat, Feb 6, 2016 at 12:00 AM, Matt K <ma...@gmail.com> wrote:
>
>> Yes. And what I'm trying to figure out if there's a way to package the
>> jar in such a way that I don't have to install it on every Executor node.
>>
>>
>> On Wed, Feb 3, 2016 at 7:46 PM, Yiannis Gkoufas <jo...@gmail.com>
>> wrote:
>>
>>> Hi Matt,
>>>
>>> does the custom class you want to package reports metrics of each
>>> Executor?
>>>
>>> Thanks
>>>
>>> On 3 February 2016 at 15:56, Matt K <ma...@gmail.com> wrote:
>>>
>>>> Thanks for sharing Yiannis, looks very promising!
>>>>
>>>> Do you know if I can package a custom class with my application, or
>>>> does it have to be pre-deployed on all Executor nodes?
>>>>
>>>> On Wed, Feb 3, 2016 at 10:36 AM, Yiannis Gkoufas <jo...@gmail.com>
>>>> wrote:
>>>>
>>>>> Hi Matt,
>>>>>
>>>>> there is some related work I recently did in IBM Research for
>>>>> visualizing the metrics produced.
>>>>> You can read about it here
>>>>> http://www.spark.tc/sparkoscope-enabling-spark-optimization-through-cross-stack-monitoring-and-visualization-2/
>>>>> We recently opensourced it if you are interested to have a deeper look
>>>>> to it: https://github.com/ibm-research-ireland/sparkoscope
>>>>>
>>>>> Thanks,
>>>>> Yiannis
>>>>>
>>>>> On 3 February 2016 at 13:32, Matt K <ma...@gmail.com> wrote:
>>>>>
>>>>>> Hi guys,
>>>>>>
>>>>>> I'm looking to create a custom sync based on Spark's Metrics System:
>>>>>>
>>>>>> https://github.com/apache/spark/blob/9f603fce78fcc997926e9a72dec44d48cbc396fc/core/src/main/scala/org/apache/spark/metrics/MetricsSystem.scala
>>>>>>
>>>>>> If I want to collect metrics from the Driver, Master, and Executor
>>>>>> nodes, should the jar with the custom class be installed on Driver, Master,
>>>>>> and Executor nodes?
>>>>>>
>>>>>> Also, on Executor nodes, does the MetricsSystem run inside the
>>>>>> Executor's JVM?
>>>>>>
>>>>>> Thanks,
>>>>>> -Matt
>>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> www.calcmachine.com - easy online calculator.
>>>>
>>>
>>>
>>
>>
>> --
>> www.calcmachine.com - easy online calculator.
>>
>
>
>
> --
> ---
> Takeshi Yamamuro
>
--
www.calcmachine.com - easy online calculator.
Re: spark metrics question
Posted by Takeshi Yamamuro <li...@gmail.com>.
How about using `spark.jars` to send jars into a cluster?
On Sat, Feb 6, 2016 at 12:00 AM, Matt K <ma...@gmail.com> wrote:
> Yes. And what I'm trying to figure out if there's a way to package the jar
> in such a way that I don't have to install it on every Executor node.
>
>
> On Wed, Feb 3, 2016 at 7:46 PM, Yiannis Gkoufas <jo...@gmail.com>
> wrote:
>
>> Hi Matt,
>>
>> does the custom class you want to package reports metrics of each
>> Executor?
>>
>> Thanks
>>
>> On 3 February 2016 at 15:56, Matt K <ma...@gmail.com> wrote:
>>
>>> Thanks for sharing Yiannis, looks very promising!
>>>
>>> Do you know if I can package a custom class with my application, or does
>>> it have to be pre-deployed on all Executor nodes?
>>>
>>> On Wed, Feb 3, 2016 at 10:36 AM, Yiannis Gkoufas <jo...@gmail.com>
>>> wrote:
>>>
>>>> Hi Matt,
>>>>
>>>> there is some related work I recently did in IBM Research for
>>>> visualizing the metrics produced.
>>>> You can read about it here
>>>> http://www.spark.tc/sparkoscope-enabling-spark-optimization-through-cross-stack-monitoring-and-visualization-2/
>>>> We recently opensourced it if you are interested to have a deeper look
>>>> to it: https://github.com/ibm-research-ireland/sparkoscope
>>>>
>>>> Thanks,
>>>> Yiannis
>>>>
>>>> On 3 February 2016 at 13:32, Matt K <ma...@gmail.com> wrote:
>>>>
>>>>> Hi guys,
>>>>>
>>>>> I'm looking to create a custom sync based on Spark's Metrics System:
>>>>>
>>>>> https://github.com/apache/spark/blob/9f603fce78fcc997926e9a72dec44d48cbc396fc/core/src/main/scala/org/apache/spark/metrics/MetricsSystem.scala
>>>>>
>>>>> If I want to collect metrics from the Driver, Master, and Executor
>>>>> nodes, should the jar with the custom class be installed on Driver, Master,
>>>>> and Executor nodes?
>>>>>
>>>>> Also, on Executor nodes, does the MetricsSystem run inside the
>>>>> Executor's JVM?
>>>>>
>>>>> Thanks,
>>>>> -Matt
>>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> www.calcmachine.com - easy online calculator.
>>>
>>
>>
>
>
> --
> www.calcmachine.com - easy online calculator.
>
--
---
Takeshi Yamamuro
Re: spark metrics question
Posted by Matt K <ma...@gmail.com>.
Yes. And what I'm trying to figure out if there's a way to package the jar
in such a way that I don't have to install it on every Executor node.
On Wed, Feb 3, 2016 at 7:46 PM, Yiannis Gkoufas <jo...@gmail.com>
wrote:
> Hi Matt,
>
> does the custom class you want to package reports metrics of each Executor?
>
> Thanks
>
> On 3 February 2016 at 15:56, Matt K <ma...@gmail.com> wrote:
>
>> Thanks for sharing Yiannis, looks very promising!
>>
>> Do you know if I can package a custom class with my application, or does
>> it have to be pre-deployed on all Executor nodes?
>>
>> On Wed, Feb 3, 2016 at 10:36 AM, Yiannis Gkoufas <jo...@gmail.com>
>> wrote:
>>
>>> Hi Matt,
>>>
>>> there is some related work I recently did in IBM Research for
>>> visualizing the metrics produced.
>>> You can read about it here
>>> http://www.spark.tc/sparkoscope-enabling-spark-optimization-through-cross-stack-monitoring-and-visualization-2/
>>> We recently opensourced it if you are interested to have a deeper look
>>> to it: https://github.com/ibm-research-ireland/sparkoscope
>>>
>>> Thanks,
>>> Yiannis
>>>
>>> On 3 February 2016 at 13:32, Matt K <ma...@gmail.com> wrote:
>>>
>>>> Hi guys,
>>>>
>>>> I'm looking to create a custom sync based on Spark's Metrics System:
>>>>
>>>> https://github.com/apache/spark/blob/9f603fce78fcc997926e9a72dec44d48cbc396fc/core/src/main/scala/org/apache/spark/metrics/MetricsSystem.scala
>>>>
>>>> If I want to collect metrics from the Driver, Master, and Executor
>>>> nodes, should the jar with the custom class be installed on Driver, Master,
>>>> and Executor nodes?
>>>>
>>>> Also, on Executor nodes, does the MetricsSystem run inside the
>>>> Executor's JVM?
>>>>
>>>> Thanks,
>>>> -Matt
>>>>
>>>
>>>
>>
>>
>> --
>> www.calcmachine.com - easy online calculator.
>>
>
>
--
www.calcmachine.com - easy online calculator.
Re: spark metrics question
Posted by Yiannis Gkoufas <jo...@gmail.com>.
Hi Matt,
does the custom class you want to package reports metrics of each Executor?
Thanks
On 3 February 2016 at 15:56, Matt K <ma...@gmail.com> wrote:
> Thanks for sharing Yiannis, looks very promising!
>
> Do you know if I can package a custom class with my application, or does
> it have to be pre-deployed on all Executor nodes?
>
> On Wed, Feb 3, 2016 at 10:36 AM, Yiannis Gkoufas <jo...@gmail.com>
> wrote:
>
>> Hi Matt,
>>
>> there is some related work I recently did in IBM Research for visualizing
>> the metrics produced.
>> You can read about it here
>> http://www.spark.tc/sparkoscope-enabling-spark-optimization-through-cross-stack-monitoring-and-visualization-2/
>> We recently opensourced it if you are interested to have a deeper look to
>> it: https://github.com/ibm-research-ireland/sparkoscope
>>
>> Thanks,
>> Yiannis
>>
>> On 3 February 2016 at 13:32, Matt K <ma...@gmail.com> wrote:
>>
>>> Hi guys,
>>>
>>> I'm looking to create a custom sync based on Spark's Metrics System:
>>>
>>> https://github.com/apache/spark/blob/9f603fce78fcc997926e9a72dec44d48cbc396fc/core/src/main/scala/org/apache/spark/metrics/MetricsSystem.scala
>>>
>>> If I want to collect metrics from the Driver, Master, and Executor
>>> nodes, should the jar with the custom class be installed on Driver, Master,
>>> and Executor nodes?
>>>
>>> Also, on Executor nodes, does the MetricsSystem run inside the
>>> Executor's JVM?
>>>
>>> Thanks,
>>> -Matt
>>>
>>
>>
>
>
> --
> www.calcmachine.com - easy online calculator.
>
Re: spark metrics question
Posted by Matt K <ma...@gmail.com>.
Thanks for sharing Yiannis, looks very promising!
Do you know if I can package a custom class with my application, or does it
have to be pre-deployed on all Executor nodes?
On Wed, Feb 3, 2016 at 10:36 AM, Yiannis Gkoufas <jo...@gmail.com>
wrote:
> Hi Matt,
>
> there is some related work I recently did in IBM Research for visualizing
> the metrics produced.
> You can read about it here
> http://www.spark.tc/sparkoscope-enabling-spark-optimization-through-cross-stack-monitoring-and-visualization-2/
> We recently opensourced it if you are interested to have a deeper look to
> it: https://github.com/ibm-research-ireland/sparkoscope
>
> Thanks,
> Yiannis
>
> On 3 February 2016 at 13:32, Matt K <ma...@gmail.com> wrote:
>
>> Hi guys,
>>
>> I'm looking to create a custom sync based on Spark's Metrics System:
>>
>> https://github.com/apache/spark/blob/9f603fce78fcc997926e9a72dec44d48cbc396fc/core/src/main/scala/org/apache/spark/metrics/MetricsSystem.scala
>>
>> If I want to collect metrics from the Driver, Master, and Executor nodes,
>> should the jar with the custom class be installed on Driver, Master, and
>> Executor nodes?
>>
>> Also, on Executor nodes, does the MetricsSystem run inside the Executor's
>> JVM?
>>
>> Thanks,
>> -Matt
>>
>
>
--
www.calcmachine.com - easy online calculator.
Re: spark metrics question
Posted by Yiannis Gkoufas <jo...@gmail.com>.
Hi Matt,
there is some related work I recently did in IBM Research for visualizing
the metrics produced.
You can read about it here
http://www.spark.tc/sparkoscope-enabling-spark-optimization-through-cross-stack-monitoring-and-visualization-2/
We recently opensourced it if you are interested to have a deeper look to
it: https://github.com/ibm-research-ireland/sparkoscope
Thanks,
Yiannis
On 3 February 2016 at 13:32, Matt K <ma...@gmail.com> wrote:
> Hi guys,
>
> I'm looking to create a custom sync based on Spark's Metrics System:
>
> https://github.com/apache/spark/blob/9f603fce78fcc997926e9a72dec44d48cbc396fc/core/src/main/scala/org/apache/spark/metrics/MetricsSystem.scala
>
> If I want to collect metrics from the Driver, Master, and Executor nodes,
> should the jar with the custom class be installed on Driver, Master, and
> Executor nodes?
>
> Also, on Executor nodes, does the MetricsSystem run inside the Executor's
> JVM?
>
> Thanks,
> -Matt
>