You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by bi...@bitfox.top on 2021/12/23 10:56:44 UTC

measure running time

hello community,

In pyspark how can I measure the running time to the command?
I just want to compare the running time of the RDD API and dataframe 
API, in my this blog:
https://bitfoxtop.wordpress.com/2021/12/23/count-email-addresses-using-sparks-rdd-and-dataframe/

I tried spark.time() it doesn't work.
Thank you.

---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org


Re: measure running time

Posted by Gourav Sengupta <go...@gmail.com>.
Hi,

There are too many blogs out there with absolutely no value. Before writing
another blog, which does not make much sense by doing run time comparisons
between RDD and dataframes (as stated earlier), it may be  useful to first
understand what you are trying to achieve by writing this blog.

Then perhaps based on that you may want to look at different options.


Regards,
Gourav Sengupta



On Fri, Dec 24, 2021 at 10:42 AM <bi...@bitfox.top> wrote:

> As you see below:
>
> $ pip install sparkmeasure
> Collecting sparkmeasure
>    Using cached
>
> https://files.pythonhosted.org/packages/9f/bf/c9810ff2d88513ffc185e65a3ab9df6121ad5b4c78aa8d134a06177f9021/sparkmeasure-0.14.0-py2.py3-none-any.whl
> Installing collected packages: sparkmeasure
> Successfully installed sparkmeasure-0.14.0
>
>
> $ pyspark --packages ch.cern.sparkmeasure:spark-measure_2.12:0.17
> Python 3.6.9 (default, Jan 26 2021, 15:33:00)
> [GCC 8.4.0] on linux
> Type "help", "copyright", "credits" or "license" for more information.
> ..........
> >>> from sparkmeasure import StageMetrics
> Traceback (most recent call last):
>    File "<stdin>", line 1, in <module>
> ModuleNotFoundError: No module named 'sparkmeasure'
>
>
> That doesn't work still.
> I run spark 3.2.0 on an ubuntu system.
>
> Regards.
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: user-unsubscribe@spark.apache.org
>
>

Re: measure running time

Posted by bi...@bitfox.top.
As you see below:

$ pip install sparkmeasure
Collecting sparkmeasure
   Using cached 
https://files.pythonhosted.org/packages/9f/bf/c9810ff2d88513ffc185e65a3ab9df6121ad5b4c78aa8d134a06177f9021/sparkmeasure-0.14.0-py2.py3-none-any.whl
Installing collected packages: sparkmeasure
Successfully installed sparkmeasure-0.14.0


$ pyspark --packages ch.cern.sparkmeasure:spark-measure_2.12:0.17
Python 3.6.9 (default, Jan 26 2021, 15:33:00)
[GCC 8.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
..........
>>> from sparkmeasure import StageMetrics
Traceback (most recent call last):
   File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'sparkmeasure'


That doesn't work still.
I run spark 3.2.0 on an ubuntu system.

Regards.

---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org


Re: measure running time

Posted by bi...@bitfox.top.
Thanks a lot Hollis. It is does due to the pypi version. Now I updated 
it.

$ pip3 -V
pip 9.0.1 from /usr/lib/python3/dist-packages (python 3.6)

$ pip3 install sparkmeasure
Collecting sparkmeasure
   Using cached 
https://files.pythonhosted.org/packages/9f/bf/c9810ff2d88513ffc185e65a3ab9df6121ad5b4c78aa8d134a06177f9021/sparkmeasure-0.14.0-py2.py3-none-any.whl
Installing collected packages: sparkmeasure
Successfully installed sparkmeasure-0.14.0

$ pyspark --packages ch.cern.sparkmeasure:spark-measure_2.12:0.17
Python 3.6.9 (default, Jan 26 2021, 15:33:00)
[GCC 8.4.0] on linux
...
>>> from sparkmeasure import StageMetrics
>>> stagemetrics = StageMetrics(spark)
>>> stagemetrics.runandmeasure(locals(), 'spark.sql("select count(*) from 
>>> range(1000) cross join range(1000) cross join range(100)").show()')
+---------+
| count(1)|
+---------+
|100000000|
+---------+
...


Hope it helps to others who have met the same issue.
Happy holidays. :0

Bitfox


On 2021-12-25 09:48, Hollis wrote:
> ---- Replied mail ----
> 
>  		 From
>  		 Mich Talebzadeh<mi...@gmail.com>
> 
>  		 Date
>  		 12/25/2021 00:25
> 
>  		 To
>  		 Sean Owen<sr...@gmail.com>
> 
>  		 Cc
>  		 user<us...@spark.apache.org>、Luca Canali<lu...@cern.ch>
> 
>  		 Subject
>  		 Re: measure running time
> 
> Hi Sean,
> 
> I have already discussed an issue in my case with Spark 3.1.1 and
> sparkmeasure  with the author Luca Canali on this matter. It has been
> reproduced. I think we ought to wait for a patch.
> 
> HTH,
> 
> Mich
> 
>    view my Linkedin profile [1]
> 
> Disclaimer: Use it at your own risk. Any and all responsibility for
> any loss, damage or destruction of data or any other property which
> may arise from relying on this email's technical content is explicitly
> disclaimed. The author will in no case be liable for any monetary
> damages arising from such loss, damage or destruction.
> 
> On Fri, 24 Dec 2021 at 14:51, Sean Owen <sr...@gmail.com> wrote:
> 
>> You probably did not install it on your cluster, nor included the
>> python package with your app
>> 
>> On Fri, Dec 24, 2021, 4:35 AM <bi...@bitfox.top> wrote:
>> 
>>> but I already installed it:
>>> 
>>> Requirement already satisfied: sparkmeasure in
>>> /usr/local/lib/python2.7/dist-packages
>>> 
>>> so how? thank you.
>>> 
>>> On 2021-12-24 18:15, Hollis wrote:
>>>> Hi bitfox,
>>>> 
>>>> you need pip install sparkmeasure firstly. then can lanch in
>>> pysaprk.
>>>> 
>>>>>>> from sparkmeasure import StageMetrics
>>>>>>> stagemetrics = StageMetrics(spark)
>>>>>>> stagemetrics.runandmeasure(locals(), 'spark.sql("select
>>> count(*)
>>>> from range(1000) cross join range(1000) cross join
>>>> range(100)").show()')
>>>> +---------+
>>>> 
>>>> | count(1)|
>>>> +---------+
>>>> |100000000|
>>>> +---------+
>>>> 
>>>> Regards,
>>>> Hollis
>>>> 
>>>> At 2021-12-24 09:18:19, bitfox@bitfox.top wrote:
>>>>> Hello list,
>>>>> 
>>>>> I run with Spark 3.2.0
>>>>> 
>>>>> After I started pyspark with:
>>>>> $ pyspark --packages
>>> ch.cern.sparkmeasure:spark-measure_2.12:0.17
>>>>> 
>>>>> I can't load from the module sparkmeasure:
>>>>> 
>>>>>>>> from sparkmeasure import StageMetrics
>>>>> Traceback (most recent call last):
>>>>> File "<stdin>", line 1, in <module>
>>>>> ModuleNotFoundError: No module named 'sparkmeasure'
>>>>> 
>>>>> Do you know why? @Luca thanks.
>>>>> 
>>>>> 
>>>>> On 2021-12-24 04:20, bitfox@bitfox.top wrote:
>>>>>> Thanks Gourav and Luca. I will try with the tools you provide
>>> in
>>>> the
>>>>>> Github.
>>>>>> 
>>>>>> On 2021-12-23 23:40, Luca Canali wrote:
>>>>>>> Hi,
>>>>>>> 
>>>>>>> I agree with Gourav that just measuring execution time is a
>>>> simplistic
>>>>>>> approach that may lead you to miss important details, in
>>>> particular
>>>>>>> when running distributed computations.
>>>>>>> 
>>>>>>> WebUI, REST API, and metrics instrumentation in Spark can be
>>> quite
>>>>>>> useful for further drill down. See
>>>>>>> https://spark.apache.org/docs/latest/monitoring.html
>>>>>>> 
>>>>>>> You can also have a look at this tool that takes care of
>>>> automating
>>>>>>> collecting and aggregating some executor task metrics:
>>>>>>> https://github.com/LucaCanali/sparkMeasure
>>>>>>> 
>>>>>>> Best,
>>>>>>> 
>>>>>>> Luca
>>>>>>> 
>>>>>>> From: Gourav Sengupta <go...@gmail.com>
>>>>>>> Sent: Thursday, December 23, 2021 14:23
>>>>>>> To: bitfox@bitfox.top
>>>>>>> Cc: user <us...@spark.apache.org>
>>>>>>> Subject: Re: measure running time
>>>>>>> 
>>>>>>> Hi,
>>>>>>> 
>>>>>>> I do not think that such time comparisons make any sense at
>>> all in
>>>>>>> distributed computation. Just saying that an operation in RDD
>>> and
>>>>>>> Dataframe can be compared based on their start and stop time
>>> may
>>>> not
>>>>>>> provide any valid information.
>>>>>>> 
>>>>>>> You will have to look into the details of timing and the
>>> steps.
>>>> For
>>>>>>> example, please look at the SPARK UI to see how timings are
>>>> calculated
>>>>>>> in distributed computing mode, there are several well written
>>>> papers
>>>>>>> on this.
>>>>>>> 
>>>>>>> Thanks and Regards,
>>>>>>> 
>>>>>>> Gourav Sengupta
>>>>>>> 
>>>>>>> On Thu, Dec 23, 2021 at 10:57 AM <bi...@bitfox.top> wrote:
>>>>>>> 
>>>>>>>> hello community,
>>>>>>>> 
>>>>>>>> In pyspark how can I measure the running time to the
>>> command?
>>>>>>>> I just want to compare the running time of the RDD API and
>>>> dataframe
>>>>>>>> 
>>>>>>>> API, in my this blog:
>>>>>>>> 
>>>>>>> 
>>>> 
>>> 
>> 
> https://bitfoxtop.wordpress.com/2021/12/23/count-email-addresses-using-sparks-rdd-and-dataframe/
>>>>>>>> 
>>>>>>>> I tried spark.time() it doesn't work.
>>>>>>>> Thank you.
>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>> 
>>> 
>> 
> ---------------------------------------------------------------------
>>>>>>>> To unsubscribe e-mail: user-unsubscribe@spark.apache.org
>>>>>> 
>>>>>> 
>>>> 
>>> 
>> 
> ---------------------------------------------------------------------
>>>>>> To unsubscribe e-mail: user-unsubscribe@spark.apache.org
>>>>> 
>>>>> 
>>> 
>> 
> ---------------------------------------------------------------------
>>>>> To unsubscribe e-mail: user-unsubscribe@spark.apache.org
>>> 
>>> 
>> 
> ---------------------------------------------------------------------
>>> To unsubscribe e-mail: user-unsubscribe@spark.apache.org
> 
> 
> Links:
> ------
> [1] https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/

---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org


Re: measure running time

Posted by Hollis <hc...@163.com>.
Hi
I can run this in my pc.
I check the email chian. bitfox install the spark measure with python2 and he launch the pyspark with python3. I think it's the reason.

Regards.
Hollis




---- Replied mail ----
| From | Mich Talebzadeh<mi...@gmail.com> |
| Date | 12/25/2021 00:25 |
| To | Sean Owen<sr...@gmail.com> |
| Cc | user<us...@spark.apache.org>、Luca Canali<lu...@cern.ch> |
| Subject | Re: measure running time |



Hi Sean,




I have already discussed an issue in my case with Spark 3.1.1 and sparkmeasure  with the author Luca Canali on this matter. It has been reproduced. I think we ought to wait for a patch.




HTH,




Mich 







   view my Linkedin profile

 

Disclaimer: Use it at your own risk.Any and all responsibility for any loss, damage or destruction of data or any other property which may arise from relying on this email's technical content is explicitly disclaimed. The author will in no case be liable for any monetary damages arising from such loss, damage or destruction.

 





On Fri, 24 Dec 2021 at 14:51, Sean Owen <sr...@gmail.com> wrote:

You probably did not install it on your cluster, nor included the python package with your app 


On Fri, Dec 24, 2021, 4:35 AM <bi...@bitfox.top> wrote:

but I already installed it:

Requirement already satisfied: sparkmeasure in
/usr/local/lib/python2.7/dist-packages

so how? thank you.

On 2021-12-24 18:15, Hollis wrote:
> Hi bitfox,
>
> you need pip install sparkmeasure firstly. then can lanch in pysaprk.
>
>>>> from sparkmeasure import StageMetrics
>>>> stagemetrics = StageMetrics(spark)
>>>> stagemetrics.runandmeasure(locals(), 'spark.sql("select count(*)
> from range(1000) cross join range(1000) cross join
> range(100)").show()')
> +---------+
>
> | count(1)|
> +---------+
> |100000000|
> +---------+
>
> Regards,
> Hollis
>
> At 2021-12-24 09:18:19, bitfox@bitfox.top wrote:
>> Hello list,
>>
>> I run with Spark 3.2.0
>>
>> After I started pyspark with:
>> $ pyspark --packages ch.cern.sparkmeasure:spark-measure_2.12:0.17
>>
>> I can't load from the module sparkmeasure:
>>
>>>>> from sparkmeasure import StageMetrics
>> Traceback (most recent call last):
>>   File "<stdin>", line 1, in <module>
>> ModuleNotFoundError: No module named 'sparkmeasure'
>>
>> Do you know why? @Luca thanks.
>>
>>
>> On 2021-12-24 04:20, bitfox@bitfox.top wrote:
>>> Thanks Gourav and Luca. I will try with the tools you provide in
> the
>>> Github.
>>>
>>> On 2021-12-23 23:40, Luca Canali wrote:
>>>> Hi,
>>>>
>>>> I agree with Gourav that just measuring execution time is a
> simplistic
>>>> approach that may lead you to miss important details, in
> particular
>>>> when running distributed computations.
>>>>
>>>> WebUI, REST API, and metrics instrumentation in Spark can be quite
>>>> useful for further drill down. See
>>>> https://spark.apache.org/docs/latest/monitoring.html
>>>>
>>>> You can also have a look at this tool that takes care of
> automating
>>>> collecting and aggregating some executor task metrics:
>>>> https://github.com/LucaCanali/sparkMeasure
>>>>
>>>> Best,
>>>>
>>>> Luca
>>>>
>>>> From: Gourav Sengupta <go...@gmail.com>
>>>> Sent: Thursday, December 23, 2021 14:23
>>>> To: bitfox@bitfox.top
>>>> Cc: user <us...@spark.apache.org>
>>>> Subject: Re: measure running time
>>>>
>>>> Hi,
>>>>
>>>> I do not think that such time comparisons make any sense at all in
>>>> distributed computation. Just saying that an operation in RDD and
>>>> Dataframe can be compared based on their start and stop time may
> not
>>>> provide any valid information.
>>>>
>>>> You will have to look into the details of timing and the steps.
> For
>>>> example, please look at the SPARK UI to see how timings are
> calculated
>>>> in distributed computing mode, there are several well written
> papers
>>>> on this.
>>>>
>>>> Thanks and Regards,
>>>>
>>>> Gourav Sengupta
>>>>
>>>> On Thu, Dec 23, 2021 at 10:57 AM <bi...@bitfox.top> wrote:
>>>>
>>>>> hello community,
>>>>>
>>>>> In pyspark how can I measure the running time to the command?
>>>>> I just want to compare the running time of the RDD API and
> dataframe
>>>>>
>>>>> API, in my this blog:
>>>>>
>>>>
> https://bitfoxtop.wordpress.com/2021/12/23/count-email-addresses-using-sparks-rdd-and-dataframe/
>>>>>
>>>>> I tried spark.time() it doesn't work.
>>>>> Thank you.
>>>>>
>>>>>
>>>>
> ---------------------------------------------------------------------
>>>>> To unsubscribe e-mail: user-unsubscribe@spark.apache.org
>>>
>>>
> ---------------------------------------------------------------------
>>> To unsubscribe e-mail: user-unsubscribe@spark.apache.org
>>
>> ---------------------------------------------------------------------
>> To unsubscribe e-mail: user-unsubscribe@spark.apache.org

---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org


Re: measure running time

Posted by Mich Talebzadeh <mi...@gmail.com>.
Hi Sean,


I have already discussed an issue in my case with Spark 3.1.1
and sparkmeasure  with the author Luca Canali on this matter. It has been
reproduced. I think we ought to wait for a patch.


HTH,


Mich



   view my Linkedin profile
<https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>



*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from
such loss, damage or destruction.




On Fri, 24 Dec 2021 at 14:51, Sean Owen <sr...@gmail.com> wrote:

> You probably did not install it on your cluster, nor included the python
> package with your app
>
> On Fri, Dec 24, 2021, 4:35 AM <bi...@bitfox.top> wrote:
>
>> but I already installed it:
>>
>> Requirement already satisfied: sparkmeasure in
>> /usr/local/lib/python2.7/dist-packages
>>
>> so how? thank you.
>>
>> On 2021-12-24 18:15, Hollis wrote:
>> > Hi bitfox,
>> >
>> > you need pip install sparkmeasure firstly. then can lanch in pysaprk.
>> >
>> >>>> from sparkmeasure import StageMetrics
>> >>>> stagemetrics = StageMetrics(spark)
>> >>>> stagemetrics.runandmeasure(locals(), 'spark.sql("select count(*)
>> > from range(1000) cross join range(1000) cross join
>> > range(100)").show()')
>> > +---------+
>> >
>> > | count(1)|
>> > +---------+
>> > |100000000|
>> > +---------+
>> >
>> > Regards,
>> > Hollis
>> >
>> > At 2021-12-24 09:18:19, bitfox@bitfox.top wrote:
>> >> Hello list,
>> >>
>> >> I run with Spark 3.2.0
>> >>
>> >> After I started pyspark with:
>> >> $ pyspark --packages ch.cern.sparkmeasure:spark-measure_2.12:0.17
>> >>
>> >> I can't load from the module sparkmeasure:
>> >>
>> >>>>> from sparkmeasure import StageMetrics
>> >> Traceback (most recent call last):
>> >>   File "<stdin>", line 1, in <module>
>> >> ModuleNotFoundError: No module named 'sparkmeasure'
>> >>
>> >> Do you know why? @Luca thanks.
>> >>
>> >>
>> >> On 2021-12-24 04:20, bitfox@bitfox.top wrote:
>> >>> Thanks Gourav and Luca. I will try with the tools you provide in
>> > the
>> >>> Github.
>> >>>
>> >>> On 2021-12-23 23:40, Luca Canali wrote:
>> >>>> Hi,
>> >>>>
>> >>>> I agree with Gourav that just measuring execution time is a
>> > simplistic
>> >>>> approach that may lead you to miss important details, in
>> > particular
>> >>>> when running distributed computations.
>> >>>>
>> >>>> WebUI, REST API, and metrics instrumentation in Spark can be quite
>> >>>> useful for further drill down. See
>> >>>> https://spark.apache.org/docs/latest/monitoring.html
>> >>>>
>> >>>> You can also have a look at this tool that takes care of
>> > automating
>> >>>> collecting and aggregating some executor task metrics:
>> >>>> https://github.com/LucaCanali/sparkMeasure
>> >>>>
>> >>>> Best,
>> >>>>
>> >>>> Luca
>> >>>>
>> >>>> From: Gourav Sengupta <go...@gmail.com>
>> >>>> Sent: Thursday, December 23, 2021 14:23
>> >>>> To: bitfox@bitfox.top
>> >>>> Cc: user <us...@spark.apache.org>
>> >>>> Subject: Re: measure running time
>> >>>>
>> >>>> Hi,
>> >>>>
>> >>>> I do not think that such time comparisons make any sense at all in
>> >>>> distributed computation. Just saying that an operation in RDD and
>> >>>> Dataframe can be compared based on their start and stop time may
>> > not
>> >>>> provide any valid information.
>> >>>>
>> >>>> You will have to look into the details of timing and the steps.
>> > For
>> >>>> example, please look at the SPARK UI to see how timings are
>> > calculated
>> >>>> in distributed computing mode, there are several well written
>> > papers
>> >>>> on this.
>> >>>>
>> >>>> Thanks and Regards,
>> >>>>
>> >>>> Gourav Sengupta
>> >>>>
>> >>>> On Thu, Dec 23, 2021 at 10:57 AM <bi...@bitfox.top> wrote:
>> >>>>
>> >>>>> hello community,
>> >>>>>
>> >>>>> In pyspark how can I measure the running time to the command?
>> >>>>> I just want to compare the running time of the RDD API and
>> > dataframe
>> >>>>>
>> >>>>> API, in my this blog:
>> >>>>>
>> >>>>
>> >
>> https://bitfoxtop.wordpress.com/2021/12/23/count-email-addresses-using-sparks-rdd-and-dataframe/
>> >>>>>
>> >>>>> I tried spark.time() it doesn't work.
>> >>>>> Thank you.
>> >>>>>
>> >>>>>
>> >>>>
>> > ---------------------------------------------------------------------
>> >>>>> To unsubscribe e-mail: user-unsubscribe@spark.apache.org
>> >>>
>> >>>
>> > ---------------------------------------------------------------------
>> >>> To unsubscribe e-mail: user-unsubscribe@spark.apache.org
>> >>
>> >> ---------------------------------------------------------------------
>> >> To unsubscribe e-mail: user-unsubscribe@spark.apache.org
>>
>> ---------------------------------------------------------------------
>> To unsubscribe e-mail: user-unsubscribe@spark.apache.org
>>
>>

Re: measure running time

Posted by Sean Owen <sr...@gmail.com>.
You probably did not install it on your cluster, nor included the python
package with your app

On Fri, Dec 24, 2021, 4:35 AM <bi...@bitfox.top> wrote:

> but I already installed it:
>
> Requirement already satisfied: sparkmeasure in
> /usr/local/lib/python2.7/dist-packages
>
> so how? thank you.
>
> On 2021-12-24 18:15, Hollis wrote:
> > Hi bitfox,
> >
> > you need pip install sparkmeasure firstly. then can lanch in pysaprk.
> >
> >>>> from sparkmeasure import StageMetrics
> >>>> stagemetrics = StageMetrics(spark)
> >>>> stagemetrics.runandmeasure(locals(), 'spark.sql("select count(*)
> > from range(1000) cross join range(1000) cross join
> > range(100)").show()')
> > +---------+
> >
> > | count(1)|
> > +---------+
> > |100000000|
> > +---------+
> >
> > Regards,
> > Hollis
> >
> > At 2021-12-24 09:18:19, bitfox@bitfox.top wrote:
> >> Hello list,
> >>
> >> I run with Spark 3.2.0
> >>
> >> After I started pyspark with:
> >> $ pyspark --packages ch.cern.sparkmeasure:spark-measure_2.12:0.17
> >>
> >> I can't load from the module sparkmeasure:
> >>
> >>>>> from sparkmeasure import StageMetrics
> >> Traceback (most recent call last):
> >>   File "<stdin>", line 1, in <module>
> >> ModuleNotFoundError: No module named 'sparkmeasure'
> >>
> >> Do you know why? @Luca thanks.
> >>
> >>
> >> On 2021-12-24 04:20, bitfox@bitfox.top wrote:
> >>> Thanks Gourav and Luca. I will try with the tools you provide in
> > the
> >>> Github.
> >>>
> >>> On 2021-12-23 23:40, Luca Canali wrote:
> >>>> Hi,
> >>>>
> >>>> I agree with Gourav that just measuring execution time is a
> > simplistic
> >>>> approach that may lead you to miss important details, in
> > particular
> >>>> when running distributed computations.
> >>>>
> >>>> WebUI, REST API, and metrics instrumentation in Spark can be quite
> >>>> useful for further drill down. See
> >>>> https://spark.apache.org/docs/latest/monitoring.html
> >>>>
> >>>> You can also have a look at this tool that takes care of
> > automating
> >>>> collecting and aggregating some executor task metrics:
> >>>> https://github.com/LucaCanali/sparkMeasure
> >>>>
> >>>> Best,
> >>>>
> >>>> Luca
> >>>>
> >>>> From: Gourav Sengupta <go...@gmail.com>
> >>>> Sent: Thursday, December 23, 2021 14:23
> >>>> To: bitfox@bitfox.top
> >>>> Cc: user <us...@spark.apache.org>
> >>>> Subject: Re: measure running time
> >>>>
> >>>> Hi,
> >>>>
> >>>> I do not think that such time comparisons make any sense at all in
> >>>> distributed computation. Just saying that an operation in RDD and
> >>>> Dataframe can be compared based on their start and stop time may
> > not
> >>>> provide any valid information.
> >>>>
> >>>> You will have to look into the details of timing and the steps.
> > For
> >>>> example, please look at the SPARK UI to see how timings are
> > calculated
> >>>> in distributed computing mode, there are several well written
> > papers
> >>>> on this.
> >>>>
> >>>> Thanks and Regards,
> >>>>
> >>>> Gourav Sengupta
> >>>>
> >>>> On Thu, Dec 23, 2021 at 10:57 AM <bi...@bitfox.top> wrote:
> >>>>
> >>>>> hello community,
> >>>>>
> >>>>> In pyspark how can I measure the running time to the command?
> >>>>> I just want to compare the running time of the RDD API and
> > dataframe
> >>>>>
> >>>>> API, in my this blog:
> >>>>>
> >>>>
> >
> https://bitfoxtop.wordpress.com/2021/12/23/count-email-addresses-using-sparks-rdd-and-dataframe/
> >>>>>
> >>>>> I tried spark.time() it doesn't work.
> >>>>> Thank you.
> >>>>>
> >>>>>
> >>>>
> > ---------------------------------------------------------------------
> >>>>> To unsubscribe e-mail: user-unsubscribe@spark.apache.org
> >>>
> >>>
> > ---------------------------------------------------------------------
> >>> To unsubscribe e-mail: user-unsubscribe@spark.apache.org
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe e-mail: user-unsubscribe@spark.apache.org
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: user-unsubscribe@spark.apache.org
>
>

Re: measure running time

Posted by bi...@bitfox.top.
but I already installed it:

Requirement already satisfied: sparkmeasure in 
/usr/local/lib/python2.7/dist-packages

so how? thank you.

On 2021-12-24 18:15, Hollis wrote:
> Hi bitfox,
> 
> you need pip install sparkmeasure firstly. then can lanch in pysaprk.
> 
>>>> from sparkmeasure import StageMetrics
>>>> stagemetrics = StageMetrics(spark)
>>>> stagemetrics.runandmeasure(locals(), 'spark.sql("select count(*)
> from range(1000) cross join range(1000) cross join
> range(100)").show()')
> +---------+
> 
> | count(1)|
> +---------+
> |100000000|
> +---------+
> 
> Regards,
> Hollis
> 
> At 2021-12-24 09:18:19, bitfox@bitfox.top wrote:
>> Hello list,
>> 
>> I run with Spark 3.2.0
>> 
>> After I started pyspark with:
>> $ pyspark --packages ch.cern.sparkmeasure:spark-measure_2.12:0.17
>> 
>> I can't load from the module sparkmeasure:
>> 
>>>>> from sparkmeasure import StageMetrics
>> Traceback (most recent call last):
>>   File "<stdin>", line 1, in <module>
>> ModuleNotFoundError: No module named 'sparkmeasure'
>> 
>> Do you know why? @Luca thanks.
>> 
>> 
>> On 2021-12-24 04:20, bitfox@bitfox.top wrote:
>>> Thanks Gourav and Luca. I will try with the tools you provide in
> the
>>> Github.
>>> 
>>> On 2021-12-23 23:40, Luca Canali wrote:
>>>> Hi,
>>>> 
>>>> I agree with Gourav that just measuring execution time is a
> simplistic
>>>> approach that may lead you to miss important details, in
> particular
>>>> when running distributed computations.
>>>> 
>>>> WebUI, REST API, and metrics instrumentation in Spark can be quite
>>>> useful for further drill down. See
>>>> https://spark.apache.org/docs/latest/monitoring.html
>>>> 
>>>> You can also have a look at this tool that takes care of
> automating
>>>> collecting and aggregating some executor task metrics:
>>>> https://github.com/LucaCanali/sparkMeasure
>>>> 
>>>> Best,
>>>> 
>>>> Luca
>>>> 
>>>> From: Gourav Sengupta <go...@gmail.com>
>>>> Sent: Thursday, December 23, 2021 14:23
>>>> To: bitfox@bitfox.top
>>>> Cc: user <us...@spark.apache.org>
>>>> Subject: Re: measure running time
>>>> 
>>>> Hi,
>>>> 
>>>> I do not think that such time comparisons make any sense at all in
>>>> distributed computation. Just saying that an operation in RDD and
>>>> Dataframe can be compared based on their start and stop time may
> not
>>>> provide any valid information.
>>>> 
>>>> You will have to look into the details of timing and the steps.
> For
>>>> example, please look at the SPARK UI to see how timings are
> calculated
>>>> in distributed computing mode, there are several well written
> papers
>>>> on this.
>>>> 
>>>> Thanks and Regards,
>>>> 
>>>> Gourav Sengupta
>>>> 
>>>> On Thu, Dec 23, 2021 at 10:57 AM <bi...@bitfox.top> wrote:
>>>> 
>>>>> hello community,
>>>>> 
>>>>> In pyspark how can I measure the running time to the command?
>>>>> I just want to compare the running time of the RDD API and
> dataframe
>>>>> 
>>>>> API, in my this blog:
>>>>> 
>>>> 
> https://bitfoxtop.wordpress.com/2021/12/23/count-email-addresses-using-sparks-rdd-and-dataframe/
>>>>> 
>>>>> I tried spark.time() it doesn't work.
>>>>> Thank you.
>>>>> 
>>>>> 
>>>> 
> ---------------------------------------------------------------------
>>>>> To unsubscribe e-mail: user-unsubscribe@spark.apache.org
>>> 
>>> 
> ---------------------------------------------------------------------
>>> To unsubscribe e-mail: user-unsubscribe@spark.apache.org
>> 
>> ---------------------------------------------------------------------
>> To unsubscribe e-mail: user-unsubscribe@spark.apache.org

---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org


Re:Re: measure running time

Posted by Hollis <hc...@163.com>.
Hi bitfox,


you need pip install sparkmeasure firstly. then can lanch in pysaprk.


| >>> from sparkmeasure import StageMetrics
>>> stagemetrics = StageMetrics(spark)
>>> stagemetrics.runandmeasure(locals(), 'spark.sql("select count(*) from range(1000) cross join range(1000) cross join range(100)").show()')
+---------+                                                                     
| count(1)|
+---------+
|100000000|
+---------+



|


Regards,
Hollis






At 2021-12-24 09:18:19, bitfox@bitfox.top wrote:
>Hello list,
>
>I run with Spark 3.2.0
>
>After I started pyspark with:
>$ pyspark --packages ch.cern.sparkmeasure:spark-measure_2.12:0.17
>
>I can't load from the module sparkmeasure:
>
>>>> from sparkmeasure import StageMetrics
>Traceback (most recent call last):
>   File "<stdin>", line 1, in <module>
>ModuleNotFoundError: No module named 'sparkmeasure'
>
>Do you know why? @Luca thanks.
>
>
>On 2021-12-24 04:20, bitfox@bitfox.top wrote:
>> Thanks Gourav and Luca. I will try with the tools you provide in the 
>> Github.
>> 
>> On 2021-12-23 23:40, Luca Canali wrote:
>>> Hi,
>>> 
>>> I agree with Gourav that just measuring execution time is a simplistic
>>> approach that may lead you to miss important details, in particular
>>> when running distributed computations.
>>> 
>>> WebUI, REST API, and metrics instrumentation in Spark can be quite
>>> useful for further drill down. See
>>> https://spark.apache.org/docs/latest/monitoring.html
>>> 
>>> You can also have a look at this tool that takes care of automating
>>> collecting and aggregating some executor task metrics:
>>> https://github.com/LucaCanali/sparkMeasure
>>> 
>>> Best,
>>> 
>>> Luca
>>> 
>>> From: Gourav Sengupta <go...@gmail.com>
>>> Sent: Thursday, December 23, 2021 14:23
>>> To: bitfox@bitfox.top
>>> Cc: user <us...@spark.apache.org>
>>> Subject: Re: measure running time
>>> 
>>> Hi,
>>> 
>>> I do not think that such time comparisons make any sense at all in
>>> distributed computation. Just saying that an operation in RDD and
>>> Dataframe can be compared based on their start and stop time may not
>>> provide any valid information.
>>> 
>>> You will have to look into the details of timing and the steps. For
>>> example, please look at the SPARK UI to see how timings are calculated
>>> in distributed computing mode, there are several well written papers
>>> on this.
>>> 
>>> Thanks and Regards,
>>> 
>>> Gourav Sengupta
>>> 
>>> On Thu, Dec 23, 2021 at 10:57 AM <bi...@bitfox.top> wrote:
>>> 
>>>> hello community,
>>>> 
>>>> In pyspark how can I measure the running time to the command?
>>>> I just want to compare the running time of the RDD API and dataframe
>>>> 
>>>> API, in my this blog:
>>>> 
>>> https://bitfoxtop.wordpress.com/2021/12/23/count-email-addresses-using-sparks-rdd-and-dataframe/
>>>> 
>>>> I tried spark.time() it doesn't work.
>>>> Thank you.
>>>> 
>>>> 
>>> ---------------------------------------------------------------------
>>>> To unsubscribe e-mail: user-unsubscribe@spark.apache.org
>> 
>> ---------------------------------------------------------------------
>> To unsubscribe e-mail: user-unsubscribe@spark.apache.org
>
>---------------------------------------------------------------------
>To unsubscribe e-mail: user-unsubscribe@spark.apache.org

Re: measure running time

Posted by bi...@bitfox.top.
Hello list,

I run with Spark 3.2.0

After I started pyspark with:
$ pyspark --packages ch.cern.sparkmeasure:spark-measure_2.12:0.17

I can't load from the module sparkmeasure:

>>> from sparkmeasure import StageMetrics
Traceback (most recent call last):
   File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'sparkmeasure'

Do you know why? @Luca thanks.


On 2021-12-24 04:20, bitfox@bitfox.top wrote:
> Thanks Gourav and Luca. I will try with the tools you provide in the 
> Github.
> 
> On 2021-12-23 23:40, Luca Canali wrote:
>> Hi,
>> 
>> I agree with Gourav that just measuring execution time is a simplistic
>> approach that may lead you to miss important details, in particular
>> when running distributed computations.
>> 
>> WebUI, REST API, and metrics instrumentation in Spark can be quite
>> useful for further drill down. See
>> https://spark.apache.org/docs/latest/monitoring.html
>> 
>> You can also have a look at this tool that takes care of automating
>> collecting and aggregating some executor task metrics:
>> https://github.com/LucaCanali/sparkMeasure
>> 
>> Best,
>> 
>> Luca
>> 
>> From: Gourav Sengupta <go...@gmail.com>
>> Sent: Thursday, December 23, 2021 14:23
>> To: bitfox@bitfox.top
>> Cc: user <us...@spark.apache.org>
>> Subject: Re: measure running time
>> 
>> Hi,
>> 
>> I do not think that such time comparisons make any sense at all in
>> distributed computation. Just saying that an operation in RDD and
>> Dataframe can be compared based on their start and stop time may not
>> provide any valid information.
>> 
>> You will have to look into the details of timing and the steps. For
>> example, please look at the SPARK UI to see how timings are calculated
>> in distributed computing mode, there are several well written papers
>> on this.
>> 
>> Thanks and Regards,
>> 
>> Gourav Sengupta
>> 
>> On Thu, Dec 23, 2021 at 10:57 AM <bi...@bitfox.top> wrote:
>> 
>>> hello community,
>>> 
>>> In pyspark how can I measure the running time to the command?
>>> I just want to compare the running time of the RDD API and dataframe
>>> 
>>> API, in my this blog:
>>> 
>> https://bitfoxtop.wordpress.com/2021/12/23/count-email-addresses-using-sparks-rdd-and-dataframe/
>>> 
>>> I tried spark.time() it doesn't work.
>>> Thank you.
>>> 
>>> 
>> ---------------------------------------------------------------------
>>> To unsubscribe e-mail: user-unsubscribe@spark.apache.org
> 
> ---------------------------------------------------------------------
> To unsubscribe e-mail: user-unsubscribe@spark.apache.org

---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org


Re: measure running time

Posted by bi...@bitfox.top.
Thanks Gourav and Luca. I will try with the tools you provide in the 
Github.

On 2021-12-23 23:40, Luca Canali wrote:
> Hi,
> 
> I agree with Gourav that just measuring execution time is a simplistic
> approach that may lead you to miss important details, in particular
> when running distributed computations.
> 
> WebUI, REST API, and metrics instrumentation in Spark can be quite
> useful for further drill down. See
> https://spark.apache.org/docs/latest/monitoring.html
> 
> You can also have a look at this tool that takes care of automating
> collecting and aggregating some executor task metrics:
> https://github.com/LucaCanali/sparkMeasure
> 
> Best,
> 
> Luca
> 
> From: Gourav Sengupta <go...@gmail.com>
> Sent: Thursday, December 23, 2021 14:23
> To: bitfox@bitfox.top
> Cc: user <us...@spark.apache.org>
> Subject: Re: measure running time
> 
> Hi,
> 
> I do not think that such time comparisons make any sense at all in
> distributed computation. Just saying that an operation in RDD and
> Dataframe can be compared based on their start and stop time may not
> provide any valid information.
> 
> You will have to look into the details of timing and the steps. For
> example, please look at the SPARK UI to see how timings are calculated
> in distributed computing mode, there are several well written papers
> on this.
> 
> Thanks and Regards,
> 
> Gourav Sengupta
> 
> On Thu, Dec 23, 2021 at 10:57 AM <bi...@bitfox.top> wrote:
> 
>> hello community,
>> 
>> In pyspark how can I measure the running time to the command?
>> I just want to compare the running time of the RDD API and dataframe
>> 
>> API, in my this blog:
>> 
> https://bitfoxtop.wordpress.com/2021/12/23/count-email-addresses-using-sparks-rdd-and-dataframe/
>> 
>> I tried spark.time() it doesn't work.
>> Thank you.
>> 
>> 
> ---------------------------------------------------------------------
>> To unsubscribe e-mail: user-unsubscribe@spark.apache.org

---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org


Re: measure running time

Posted by Mich Talebzadeh <mi...@gmail.com>.
Thanks Luca,

I am still getting some error


* pyspark --packages ch.cern.sparkmeasure:spark-measure_2.12:0.17*

Python 3.7.3 (default, Mar 27 2019, 22:11:17)

[GCC 7.3.0] :: Anaconda, Inc. on linux

Type "help", "copyright", "credits" or "license" for more information.

:: loading settings :: url =
jar:file:/d4T/hduser/spark-3.1.1-bin-hadoop3.2/jars/ivy-2.4.0.jar!/org/apache/ivy/core/settings/ivysettings.xml

Ivy Default Cache set to: /home/hduser/.ivy2/cache

The jars for the packages stored in: /home/hduser/.ivy2/jars

ch.cern.sparkmeasure#spark-measure_2.12 added as a dependency

:: resolving dependencies ::
org.apache.spark#spark-submit-parent-8175aada-8494-4953-a687-9c95b282c751;1.0

        confs: [default]

        found ch.cern.sparkmeasure#spark-measure_2.12;0.17 in central

        found com.fasterxml.jackson.module#jackson-module-scala_2.12;2.9.9
in central

        found com.fasterxml.jackson.core#jackson-core;2.9.9 in central

        found com.fasterxml.jackson.core#jackson-annotations;2.9.9 in
central

        found com.fasterxml.jackson.core#jackson-databind;2.9.9 in central

        found com.fasterxml.jackson.module#jackson-module-paranamer;2.9.9
in central

        found com.thoughtworks.paranamer#paranamer;2.8 in spark-list

        found org.slf4j#slf4j-api;1.7.26 in central

        found org.influxdb#influxdb-java;2.14 in central

        found com.squareup.retrofit2#retrofit;2.4.0 in central

        found com.squareup.retrofit2#converter-moshi;2.4.0 in central

        found com.squareup.moshi#moshi;1.5.0 in central

        found com.squareup.okio#okio;1.13.0 in central

        found org.msgpack#msgpack-core;0.8.16 in central

        found com.squareup.okhttp3#okhttp;3.11.0 in local-m2-cache

        found com.squareup.okio#okio;1.14.0 in local-m2-cache

        found com.squareup.okhttp3#logging-interceptor;3.11.0 in central

:: resolution report :: resolve 5349ms :: artifacts dl 2ms

        :: modules in use:

        ch.cern.sparkmeasure#spark-measure_2.12;0.17 from central in
[default]

        com.fasterxml.jackson.core#jackson-annotations;2.9.9 from central
in [default]

        com.fasterxml.jackson.core#jackson-core;2.9.9 from central in
[default]

        com.fasterxml.jackson.core#jackson-databind;2.9.9 from central in
[default]

        com.fasterxml.jackson.module#jackson-module-paranamer;2.9.9 from
central in [default]

        com.fasterxml.jackson.module#jackson-module-scala_2.12;2.9.9 from
central in [default]

        com.squareup.moshi#moshi;1.5.0 from central in [default]

        com.squareup.okhttp3#logging-interceptor;3.11.0 from central in
[default]

        com.squareup.okhttp3#okhttp;3.11.0 from local-m2-cache in [default]

        com.squareup.okio#okio;1.14.0 from local-m2-cache in [default]

        com.squareup.retrofit2#converter-moshi;2.4.0 from central in
[default]

        com.squareup.retrofit2#retrofit;2.4.0 from central in [default]

        com.thoughtworks.paranamer#paranamer;2.8 from spark-list in
[default]

        org.influxdb#influxdb-java;2.14 from central in [default]

        org.msgpack#msgpack-core;0.8.16 from central in [default]

        org.slf4j#slf4j-api;1.7.26 from central in [default]

        :: evicted modules:

        com.fasterxml.jackson.core#jackson-annotations;2.9.0 by
[com.fasterxml.jackson.core#jackson-annotations;2.9.9] in [default]

        com.squareup.okhttp3#okhttp;3.10.0 by
[com.squareup.okhttp3#okhttp;3.11.0] in [default]

        com.squareup.okio#okio;1.13.0 by [com.squareup.okio#okio;1.14.0] in
[default]


---------------------------------------------------------------------

        |                  |            modules            ||   artifacts
 |

        |       conf       | number| search|dwnlded|evicted||
number|dwnlded|


---------------------------------------------------------------------

        |      default     |   19  |   11  |   11  |   3   ||   16  |   0
 |


---------------------------------------------------------------------


:: problems summary ::

:::: ERRORS

        unknown resolver null


        SERVER ERROR: Bad Gateway url=
https://dl.bintray.com/spark-packages/maven/com/fasterxml/jackson/jackson-bom/2.9.9/jackson-bom-2.9.9.jar


        SERVER ERROR: Bad Gateway url=
https://dl.bintray.com/spark-packages/maven/com/fasterxml/jackson/jackson-base/2.9.9/jackson-base-2.9.9.jar


        unknown resolver null


I will try to investigate it



Cheers


   view my Linkedin profile
<https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>



*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from
such loss, damage or destruction.




On Thu, 23 Dec 2021 at 19:46, Luca Canali <lu...@cern.ch> wrote:

> Hi Mich,
>
>
>
> With Spark 3.1.1 you need to use spark-measure built with Scala 2.12:
>
>
>
> bin/pyspark --packages ch.cern.sparkmeasure:spark-measure_2.12:0.17
>
>
>
> Best,
>
> Luca
>
>
>
> *From:* Mich Talebzadeh <mi...@gmail.com>
> *Sent:* Thursday, December 23, 2021 19:59
> *To:* Luca Canali <lu...@cern.ch>
> *Cc:* user <us...@spark.apache.org>
> *Subject:* Re: measure running time
>
>
>
> Hi Luca,
>
>
>
> Have you tested this link  https://github.com/LucaCanali/sparkMeasure
>
>
>
> With Spark 3.1.1/PySpark,   I am getting this error
>
>
>
>
>
> pyspark --packages ch.cern.sparkmeasure:spark-measure_2.11:0.17
>
>
>
> :: problems summary ::
>
> :::: ERRORS
>
>         unknown resolver null
>
>
>
>         SERVER ERROR: Bad Gateway url=
> https://dl.bintray.com/spark-packages/maven/com/fasterxml/jackson/jackson-bom/2.9.9/jackson-bom-2.9.9.jar
>
>
>
>         SERVER ERROR: Bad Gateway url=
> https://dl.bintray.com/spark-packages/maven/com/fasterxml/jackson/jackson-base/2.9.9/jackson-base-2.9.9.jar
>
>
>
> Using Python version 3.7.3 (default, Mar 27 2019 22:11:17)
>
> Spark context Web UI available at http://rhes76:4040
>
> Spark context available as 'sc' (master = local[*], app id =
> local-1640285629478).
>
> SparkSession available as 'spark'.
>
>
>
> >>> from sparkmeasure import StageMetrics
>
> >>> stagemetrics = StageMetrics(spark)
>
> Traceback (most recent call last):
>
>   File "<stdin>", line 1, in <module>
>
>   File
> "/home/hduser/anaconda3/envs/pyspark_venv/lib/python3.7/site-packages/sparkmeasure/stagemetrics.py",
> line 15, in __init__
>
>     self.stagemetrics =
> self.sc._jvm.ch.cern.sparkmeasure.StageMetrics(self.sparksession._jsparkSession)
>
>   File "/opt/spark/python/lib/py4j-0.10.9-src.zip/py4j/java_gateway.py",
> line 1569, in __call__
>
>   File "/opt/spark/python/pyspark/sql/utils.py", line 111, in deco
>
>     return f(*a, **kw)
>
>   File "/opt/spark/python/lib/py4j-0.10.9-src.zip/py4j/protocol.py", line
> 328, in get_return_value
>
> py4j.protocol.Py4JJavaError: An error occurred while calling
> None.ch.cern.sparkmeasure.StageMetrics.
>
> : java.lang.NoClassDefFoundError: scala/Product$class
>
>         at ch.cern.sparkmeasure.StageMetrics.<init>(stagemetrics.scala:111)
>
>         at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
>
>         at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
>
>         at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>
>         at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
>
>         at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:247)
>
>         at
> py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
>
>         at py4j.Gateway.invoke(Gateway.java:238)
>
>         at
> py4j.commands.ConstructorCommand.invokeConstructor(ConstructorCommand.java:80)
>
>         at
> py4j.commands.ConstructorCommand.execute(ConstructorCommand.java:69)
>
>         at py4j.GatewayConnection.run(GatewayConnection.java:238)
>
>         at java.lang.Thread.run(Thread.java:748)
>
> Caused by: java.lang.ClassNotFoundException: scala.Product$class
>
>         at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
>
>         at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
>
>         at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>
>         ... 12 more
>
>
>
> Thanks
>
>
>
>
>
>  [image: Image removed by sender.]  view my Linkedin profile
> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>
>
>
> *Disclaimer:* Use it at your own risk. Any and all responsibility for any
> loss, damage or destruction of data or any other property which may arise
> from relying on this email's technical content is explicitly disclaimed.
> The author will in no case be liable for any monetary damages arising from
> such loss, damage or destruction.
>
>
>
>
>
>
>
> On Thu, 23 Dec 2021 at 15:41, Luca Canali <lu...@cern.ch> wrote:
>
> Hi,
>
>
>
> I agree with Gourav that just measuring execution time is a simplistic
> approach that may lead you to miss important details, in particular when
> running distributed computations.
>
> WebUI, REST API, and metrics instrumentation in Spark can be quite useful
> for further drill down. See
> https://spark.apache.org/docs/latest/monitoring.html
>
> You can also have a look at this tool that takes care of automating
> collecting and aggregating some executor task metrics:
> https://github.com/LucaCanali/sparkMeasure
>
>
>
> Best,
>
> Luca
>
>
>
> *From:* Gourav Sengupta <go...@gmail.com>
> *Sent:* Thursday, December 23, 2021 14:23
> *To:* bitfox@bitfox.top
> *Cc:* user <us...@spark.apache.org>
> *Subject:* Re: measure running time
>
>
>
> Hi,
>
>
>
> I do not think that such time comparisons make any sense at all in
> distributed computation. Just saying that an operation in RDD and Dataframe
> can be compared based on their start and stop time may not provide any
> valid information.
>
>
>
> You will have to look into the details of timing and the steps. For
> example, please look at the SPARK UI to see how timings are calculated in
> distributed computing mode, there are several well written papers on this.
>
>
>
>
>
> Thanks and Regards,
>
> Gourav Sengupta
>
>
>
>
>
>
>
>
>
>
>
> On Thu, Dec 23, 2021 at 10:57 AM <bi...@bitfox.top> wrote:
>
> hello community,
>
> In pyspark how can I measure the running time to the command?
> I just want to compare the running time of the RDD API and dataframe
> API, in my this blog:
>
> https://bitfoxtop.wordpress.com/2021/12/23/count-email-addresses-using-sparks-rdd-and-dataframe/
>
> I tried spark.time() it doesn't work.
> Thank you.
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: user-unsubscribe@spark.apache.org
>
>

RE: measure running time

Posted by Luca Canali <lu...@cern.ch>.
Hi Mich,

 

With Spark 3.1.1 you need to use spark-measure built with Scala 2.12:  

 

bin/pyspark --packages ch.cern.sparkmeasure:spark-measure_2.12:0.17

 

Best,

Luca

 

From: Mich Talebzadeh <mi...@gmail.com> 
Sent: Thursday, December 23, 2021 19:59
To: Luca Canali <lu...@cern.ch>
Cc: user <us...@spark.apache.org>
Subject: Re: measure running time

 

Hi Luca,

 

Have you tested this link  https://github.com/LucaCanali/sparkMeasure

 

With Spark 3.1.1/PySpark,   I am getting this error 

 

 

pyspark --packages ch.cern.sparkmeasure:spark-measure_2.11:0.17

 

:: problems summary ::

:::: ERRORS

        unknown resolver null

 

        SERVER ERROR: Bad Gateway url=https://dl.bintray.com/spark-packages/maven/com/fasterxml/jackson/jackson-bom/2.9.9/jackson-bom-2.9.9.jar

 

        SERVER ERROR: Bad Gateway url=https://dl.bintray.com/spark-packages/maven/com/fasterxml/jackson/jackson-base/2.9.9/jackson-base-2.9.9.jar

 

Using Python version 3.7.3 (default, Mar 27 2019 22:11:17)

Spark context Web UI available at http://rhes76:4040

Spark context available as 'sc' (master = local[*], app id = local-1640285629478).

SparkSession available as 'spark'.

 

>>> from sparkmeasure import StageMetrics

>>> stagemetrics = StageMetrics(spark)

Traceback (most recent call last):

  File "<stdin>", line 1, in <module>

  File "/home/hduser/anaconda3/envs/pyspark_venv/lib/python3.7/site-packages/sparkmeasure/stagemetrics.py", line 15, in __init__

    self.stagemetrics = self.sc._jvm.ch.cern.sparkmeasure.StageMetrics(self.sparksession._jsparkSession)

  File "/opt/spark/python/lib/py4j-0.10.9-src.zip/py4j/java_gateway.py", line 1569, in __call__

  File "/opt/spark/python/pyspark/sql/utils.py", line 111, in deco

    return f(*a, **kw)

  File "/opt/spark/python/lib/py4j-0.10.9-src.zip/py4j/protocol.py", line 328, in get_return_value

py4j.protocol.Py4JJavaError: An error occurred while calling None.ch.cern.sparkmeasure.StageMetrics.

: java.lang.NoClassDefFoundError: scala/Product$class

        at ch.cern.sparkmeasure.StageMetrics.<init>(stagemetrics.scala:111)

        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)

        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)

        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)

        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)

        at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:247)

        at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)

        at py4j.Gateway.invoke(Gateway.java:238)

        at py4j.commands.ConstructorCommand.invokeConstructor(ConstructorCommand.java:80)

        at py4j.commands.ConstructorCommand.execute(ConstructorCommand.java:69)

        at py4j.GatewayConnection.run(GatewayConnection.java:238)

        at java.lang.Thread.run(Thread.java:748)

Caused by: java.lang.ClassNotFoundException: scala.Product$class

        at java.net.URLClassLoader.findClass(URLClassLoader.java:382)

        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)

        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)

        ... 12 more

 

Thanks

 

 

   view my Linkedin profile <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/> 

 

Disclaimer: Use it at your own risk. Any and all responsibility for any loss, damage or destruction of data or any other property which may arise from relying on this email's technical content is explicitly disclaimed. The author will in no case be liable for any monetary damages arising from such loss, damage or destruction. 

 

 

 

On Thu, 23 Dec 2021 at 15:41, Luca Canali <luca.canali@cern.ch <ma...@cern.ch> > wrote:

Hi,

 

I agree with Gourav that just measuring execution time is a simplistic approach that may lead you to miss important details, in particular when running distributed computations.

WebUI, REST API, and metrics instrumentation in Spark can be quite useful for further drill down. See https://spark.apache.org/docs/latest/monitoring.html

You can also have a look at this tool that takes care of automating collecting and aggregating some executor task metrics: https://github.com/LucaCanali/sparkMeasure

 

Best,

Luca

 

From: Gourav Sengupta <gourav.sengupta@gmail.com <ma...@gmail.com> > 
Sent: Thursday, December 23, 2021 14:23
To: bitfox@bitfox.top
Cc: user <user@spark.apache.org <ma...@spark.apache.org> >
Subject: Re: measure running time

 

Hi,

 

I do not think that such time comparisons make any sense at all in distributed computation. Just saying that an operation in RDD and Dataframe can be compared based on their start and stop time may not provide any valid information.

 

You will have to look into the details of timing and the steps. For example, please look at the SPARK UI to see how timings are calculated in distributed computing mode, there are several well written papers on this.

 

 

Thanks and Regards,

Gourav Sengupta

 

 

 

 

 

On Thu, Dec 23, 2021 at 10:57 AM <bitfox@bitfox.top <ma...@bitfox.top> > wrote:

hello community,

In pyspark how can I measure the running time to the command?
I just want to compare the running time of the RDD API and dataframe 
API, in my this blog:
https://bitfoxtop.wordpress.com/2021/12/23/count-email-addresses-using-sparks-rdd-and-dataframe/

I tried spark.time() it doesn't work.
Thank you.

---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org <ma...@spark.apache.org> 


Re: measure running time

Posted by Mich Talebzadeh <mi...@gmail.com>.
Hi Luca,

Have you tested this link  https://github.com/LucaCanali/sparkMeasure


With Spark 3.1.1/PySpark,   I am getting this error



pyspark --packages ch.cern.sparkmeasure:spark-measure_2.11:0.17

:: problems summary ::

:::: ERRORS

        unknown resolver null


        SERVER ERROR: Bad Gateway url=
https://dl.bintray.com/spark-packages/maven/com/fasterxml/jackson/jackson-bom/2.9.9/jackson-bom-2.9.9.jar


        SERVER ERROR: Bad Gateway url=
https://dl.bintray.com/spark-packages/maven/com/fasterxml/jackson/jackson-base/2.9.9/jackson-base-2.9.9.jar


Using Python version 3.7.3 (default, Mar 27 2019 22:11:17)

Spark context Web UI available at http://rhes76:4040

Spark context available as 'sc' (master = local[*], app id =
local-1640285629478).

SparkSession available as 'spark'.

>>> from sparkmeasure import StageMetrics
>>> stagemetrics = StageMetrics(spark)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File
"/home/hduser/anaconda3/envs/pyspark_venv/lib/python3.7/site-packages/sparkmeasure/stagemetrics.py",
line 15, in __init__
    self.stagemetrics =
self.sc._jvm.ch.cern.sparkmeasure.StageMetrics(self.sparksession._jsparkSession)
  File "/opt/spark/python/lib/py4j-0.10.9-src.zip/py4j/java_gateway.py",
line 1569, in __call__
  File "/opt/spark/python/pyspark/sql/utils.py", line 111, in deco
    return f(*a, **kw)
  File "/opt/spark/python/lib/py4j-0.10.9-src.zip/py4j/protocol.py", line
328, in get_return_value
py4j.protocol.Py4JJavaError: An error occurred while calling
None.ch.cern.sparkmeasure.StageMetrics.
: java.lang.NoClassDefFoundError: scala/Product$class
        at ch.cern.sparkmeasure.StageMetrics.<init>(stagemetrics.scala:111)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method)
        at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:247)
        at
py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
        at py4j.Gateway.invoke(Gateway.java:238)
        at
py4j.commands.ConstructorCommand.invokeConstructor(ConstructorCommand.java:80)
        at
py4j.commands.ConstructorCommand.execute(ConstructorCommand.java:69)
        at py4j.GatewayConnection.run(GatewayConnection.java:238)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.ClassNotFoundException: scala.Product$class
        at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        ... 12 more

Thanks



   view my Linkedin profile
<https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>



*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from
such loss, damage or destruction.




On Thu, 23 Dec 2021 at 15:41, Luca Canali <lu...@cern.ch> wrote:

> Hi,
>
>
>
> I agree with Gourav that just measuring execution time is a simplistic
> approach that may lead you to miss important details, in particular when
> running distributed computations.
>
> WebUI, REST API, and metrics instrumentation in Spark can be quite useful
> for further drill down. See
> https://spark.apache.org/docs/latest/monitoring.html
>
> You can also have a look at this tool that takes care of automating
> collecting and aggregating some executor task metrics:
> https://github.com/LucaCanali/sparkMeasure
>
>
>
> Best,
>
> Luca
>
>
>
> *From:* Gourav Sengupta <go...@gmail.com>
> *Sent:* Thursday, December 23, 2021 14:23
> *To:* bitfox@bitfox.top
> *Cc:* user <us...@spark.apache.org>
> *Subject:* Re: measure running time
>
>
>
> Hi,
>
>
>
> I do not think that such time comparisons make any sense at all in
> distributed computation. Just saying that an operation in RDD and Dataframe
> can be compared based on their start and stop time may not provide any
> valid information.
>
>
>
> You will have to look into the details of timing and the steps. For
> example, please look at the SPARK UI to see how timings are calculated in
> distributed computing mode, there are several well written papers on this.
>
>
>
>
>
> Thanks and Regards,
>
> Gourav Sengupta
>
>
>
>
>
>
>
>
>
>
>
> On Thu, Dec 23, 2021 at 10:57 AM <bi...@bitfox.top> wrote:
>
> hello community,
>
> In pyspark how can I measure the running time to the command?
> I just want to compare the running time of the RDD API and dataframe
> API, in my this blog:
>
> https://bitfoxtop.wordpress.com/2021/12/23/count-email-addresses-using-sparks-rdd-and-dataframe/
>
> I tried spark.time() it doesn't work.
> Thank you.
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: user-unsubscribe@spark.apache.org
>
>

RE: measure running time

Posted by Luca Canali <lu...@cern.ch>.
Hi,

 

I agree with Gourav that just measuring execution time is a simplistic approach that may lead you to miss important details, in particular when running distributed computations.

WebUI, REST API, and metrics instrumentation in Spark can be quite useful for further drill down. See https://spark.apache.org/docs/latest/monitoring.html

You can also have a look at this tool that takes care of automating collecting and aggregating some executor task metrics: https://github.com/LucaCanali/sparkMeasure

 

Best,

Luca

 

From: Gourav Sengupta <go...@gmail.com> 
Sent: Thursday, December 23, 2021 14:23
To: bitfox@bitfox.top
Cc: user <us...@spark.apache.org>
Subject: Re: measure running time

 

Hi,

 

I do not think that such time comparisons make any sense at all in distributed computation. Just saying that an operation in RDD and Dataframe can be compared based on their start and stop time may not provide any valid information.

 

You will have to look into the details of timing and the steps. For example, please look at the SPARK UI to see how timings are calculated in distributed computing mode, there are several well written papers on this.

 

 

Thanks and Regards,

Gourav Sengupta

 

 

 

 

 

On Thu, Dec 23, 2021 at 10:57 AM <bitfox@bitfox.top <ma...@bitfox.top> > wrote:

hello community,

In pyspark how can I measure the running time to the command?
I just want to compare the running time of the RDD API and dataframe 
API, in my this blog:
https://bitfoxtop.wordpress.com/2021/12/23/count-email-addresses-using-sparks-rdd-and-dataframe/

I tried spark.time() it doesn't work.
Thank you.

---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org <ma...@spark.apache.org> 


Re: measure running time

Posted by Gourav Sengupta <go...@gmail.com>.
Hi,

I do not think that such time comparisons make any sense at all in
distributed computation. Just saying that an operation in RDD and Dataframe
can be compared based on their start and stop time may not provide any
valid information.

You will have to look into the details of timing and the steps. For
example, please look at the SPARK UI to see how timings are calculated in
distributed computing mode, there are several well written papers on this.


Thanks and Regards,
Gourav Sengupta





On Thu, Dec 23, 2021 at 10:57 AM <bi...@bitfox.top> wrote:

> hello community,
>
> In pyspark how can I measure the running time to the command?
> I just want to compare the running time of the RDD API and dataframe
> API, in my this blog:
>
> https://bitfoxtop.wordpress.com/2021/12/23/count-email-addresses-using-sparks-rdd-and-dataframe/
>
> I tried spark.time() it doesn't work.
> Thank you.
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: user-unsubscribe@spark.apache.org
>
>

Re: measure running time

Posted by Mich Talebzadeh <mi...@gmail.com>.
Try this simple thing first

import time
def main():
    start_time = time.time()

   print("\nStarted at");uf.println(lst)
   # your code

   print("\nFinished at");uf.println(lst)
   end_time = time.time()
   time_elapsed = (end_time - start_time)
   print(f"""Elapsed time in seconds is {time_elapsed}""")
   spark_session.stop()


see

https://github.com/michTalebzadeh/spark_on_gke/blob/main/src/RandomDataBigQuery.py

HTH


   view my Linkedin profile
<https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>



*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from
such loss, damage or destruction.




On Thu, 23 Dec 2021 at 10:58, <bi...@bitfox.top> wrote:

> hello community,
>
> In pyspark how can I measure the running time to the command?
> I just want to compare the running time of the RDD API and dataframe
> API, in my this blog:
>
> https://bitfoxtop.wordpress.com/2021/12/23/count-email-addresses-using-sparks-rdd-and-dataframe/
>
> I tried spark.time() it doesn't work.
> Thank you.
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: user-unsubscribe@spark.apache.org
>
>