You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by gsamaras <ge...@gmail.com> on 2016/08/29 21:34:53 UTC
KMeans calls takeSample() twice?
After reading the internal code of Spark about it, I wasn't able to
understand why it calls takeSample() twice? Can someone please explain?
There is a relevant StackOverflow question
<http://stackoverflow.com/questions/38986395/sparkkmeans-calls-takesample-twice>
.
--
View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/KMeans-calls-takeSample-twice-tp18761.html
Sent from the Apache Spark Developers List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
Re: KMeans calls takeSample() twice?
Posted by Georgios Samaras <ge...@gmail.com>.
But as can see in this:
Stackoverflow question
<http://stackoverflow.com/questions/38986395/sparkkmeans-calls-takesample-twice>
this is simply not the case for me. Could it because your data is not big
enough to reproduce?
Or could it be that you are actually reproducing it, but the problem comes
with the weakness of the UI to display it correctly?
On Wed, Aug 31, 2016 at 4:38 AM, Yanbo Liang <yb...@gmail.com> wrote:
> I added println at the start of function takeSample, and found it was
> printed only once for each run of KMeans.
>
> Thanks
> Yanbo
>
> On Tue, Aug 30, 2016 at 10:31 AM, Georgios Samaras <
> georgesamarasdit@gmail.com> wrote:
>
>> Good catch Shivaram. However, the very next line states:
>>
>> // this shouldn't happen often because we use a big multiplier for the
>> initial size
>>
>> which makes me wondering if that is the case, really, since I am
>> experimenting heavily right now and I launched 30~40 jobs, and from a
>> glance on them I can see takeSample() being called twice!
>>
>> George
>>
>>
>> On Tue, Aug 30, 2016 at 10:20 AM, Shivaram Venkataraman <
>> shivaram@eecs.berkeley.edu> wrote:
>>
>>> I think takeSample itself runs multiple jobs if the amount of samples
>>> collected in the first pass is not enough. The comment and code path
>>> at https://github.com/apache/spark/blob/412b0e8969215411b97efd3
>>> d0984dc6cac5d31e0/core/src/main/scala/org/apache/spark/rdd/
>>> RDD.scala#L508
>>> should explain when this happens. Also you can confirm this by
>>> checking if the logWarning shows up in your logs.
>>>
>>> Thanks
>>> Shivaram
>>>
>>> On Tue, Aug 30, 2016 at 9:50 AM, Georgios Samaras
>>> <ge...@gmail.com> wrote:
>>> >
>>> > ---------- Forwarded message ----------
>>> > From: Georgios Samaras <ge...@gmail.com>
>>> > Date: Tue, Aug 30, 2016 at 9:49 AM
>>> > Subject: Re: KMeans calls takeSample() twice?
>>> > To: "Sean Owen [via Apache Spark Developers List]"
>>> > <ml...@n3.nabble.com>
>>> >
>>> >
>>> > I am not sure what you want me to check. Note that I see two
>>> takeSample()s
>>> > being invoked every single time I execute KMeans(). In a current job I
>>> have,
>>> > I did view the details and updated the:
>>> >
>>> > StackOverflow question.
>>> >
>>> >
>>> >
>>> > On Tue, Aug 30, 2016 at 9:25 AM, Sean Owen [via Apache Spark Developers
>>> > List] <ml...@n3.nabble.com> wrote:
>>> >>
>>> >> I'm not sure it's a UI bug; it really does record two different
>>> >> stages, the second of which executes quickly. I am not sure why that
>>> >> would happen off the top of my head. I don't see anything that failed
>>> >> here.
>>> >>
>>> >> Digging into those two stages and what they executed might give a clue
>>> >> to what's really going on there.
>>> >>
>>> >> On Tue, Aug 30, 2016 at 5:18 PM, gsamaras <[hidden email]> wrote:
>>> >> > Yanbo thank you for your reply. So you are saying that this is a
>>> bug in
>>> >> > the
>>> >> > Spark UI in general, and not in the local Spark UI of our cluster,
>>> where
>>> >> > I
>>> >> > work, right?
>>> >> >
>>> >> > George
>>> >>
>>> >> ---------------------------------------------------------------------
>>> >> To unsubscribe e-mail: [hidden email]
>>> >>
>>> >>
>>> >>
>>> >> ________________________________
>>> >> If you reply to this email, your message will be added to the
>>> discussion
>>> >> below:
>>> >>
>>> >> http://apache-spark-developers-list.1001551.n3.nabble.com/KM
>>> eans-calls-takeSample-twice-tp18761p18788.html
>>> >> To unsubscribe from KMeans calls takeSample() twice?, click here.
>>> >> NAML
>>> >
>>> >
>>> >
>>>
>>
>>
>
Re: KMeans calls takeSample() twice?
Posted by Yanbo Liang <yb...@gmail.com>.
I added println at the start of function takeSample, and found it was
printed only once for each run of KMeans.
Thanks
Yanbo
On Tue, Aug 30, 2016 at 10:31 AM, Georgios Samaras <
georgesamarasdit@gmail.com> wrote:
> Good catch Shivaram. However, the very next line states:
>
> // this shouldn't happen often because we use a big multiplier for the
> initial size
>
> which makes me wondering if that is the case, really, since I am
> experimenting heavily right now and I launched 30~40 jobs, and from a
> glance on them I can see takeSample() being called twice!
>
> George
>
>
> On Tue, Aug 30, 2016 at 10:20 AM, Shivaram Venkataraman <
> shivaram@eecs.berkeley.edu> wrote:
>
>> I think takeSample itself runs multiple jobs if the amount of samples
>> collected in the first pass is not enough. The comment and code path
>> at https://github.com/apache/spark/blob/412b0e8969215411b97efd3
>> d0984dc6cac5d31e0/core/src/main/scala/org/apache/spark/rdd/RDD.scala#L508
>> should explain when this happens. Also you can confirm this by
>> checking if the logWarning shows up in your logs.
>>
>> Thanks
>> Shivaram
>>
>> On Tue, Aug 30, 2016 at 9:50 AM, Georgios Samaras
>> <ge...@gmail.com> wrote:
>> >
>> > ---------- Forwarded message ----------
>> > From: Georgios Samaras <ge...@gmail.com>
>> > Date: Tue, Aug 30, 2016 at 9:49 AM
>> > Subject: Re: KMeans calls takeSample() twice?
>> > To: "Sean Owen [via Apache Spark Developers List]"
>> > <ml...@n3.nabble.com>
>> >
>> >
>> > I am not sure what you want me to check. Note that I see two
>> takeSample()s
>> > being invoked every single time I execute KMeans(). In a current job I
>> have,
>> > I did view the details and updated the:
>> >
>> > StackOverflow question.
>> >
>> >
>> >
>> > On Tue, Aug 30, 2016 at 9:25 AM, Sean Owen [via Apache Spark Developers
>> > List] <ml...@n3.nabble.com> wrote:
>> >>
>> >> I'm not sure it's a UI bug; it really does record two different
>> >> stages, the second of which executes quickly. I am not sure why that
>> >> would happen off the top of my head. I don't see anything that failed
>> >> here.
>> >>
>> >> Digging into those two stages and what they executed might give a clue
>> >> to what's really going on there.
>> >>
>> >> On Tue, Aug 30, 2016 at 5:18 PM, gsamaras <[hidden email]> wrote:
>> >> > Yanbo thank you for your reply. So you are saying that this is a bug
>> in
>> >> > the
>> >> > Spark UI in general, and not in the local Spark UI of our cluster,
>> where
>> >> > I
>> >> > work, right?
>> >> >
>> >> > George
>> >>
>> >> ---------------------------------------------------------------------
>> >> To unsubscribe e-mail: [hidden email]
>> >>
>> >>
>> >>
>> >> ________________________________
>> >> If you reply to this email, your message will be added to the
>> discussion
>> >> below:
>> >>
>> >> http://apache-spark-developers-list.1001551.n3.nabble.com/
>> KMeans-calls-takeSample-twice-tp18761p18788.html
>> >> To unsubscribe from KMeans calls takeSample() twice?, click here.
>> >> NAML
>> >
>> >
>> >
>>
>
>
Re: KMeans calls takeSample() twice?
Posted by Georgios Samaras <ge...@gmail.com>.
Good catch Shivaram. However, the very next line states:
// this shouldn't happen often because we use a big multiplier for the
initial size
which makes me wondering if that is the case, really, since I am
experimenting heavily right now and I launched 30~40 jobs, and from a
glance on them I can see takeSample() being called twice!
George
On Tue, Aug 30, 2016 at 10:20 AM, Shivaram Venkataraman <
shivaram@eecs.berkeley.edu> wrote:
> I think takeSample itself runs multiple jobs if the amount of samples
> collected in the first pass is not enough. The comment and code path
> at https://github.com/apache/spark/blob/412b0e8969215411b97efd3d0984dc
> 6cac5d31e0/core/src/main/scala/org/apache/spark/rdd/RDD.scala#L508
> should explain when this happens. Also you can confirm this by
> checking if the logWarning shows up in your logs.
>
> Thanks
> Shivaram
>
> On Tue, Aug 30, 2016 at 9:50 AM, Georgios Samaras
> <ge...@gmail.com> wrote:
> >
> > ---------- Forwarded message ----------
> > From: Georgios Samaras <ge...@gmail.com>
> > Date: Tue, Aug 30, 2016 at 9:49 AM
> > Subject: Re: KMeans calls takeSample() twice?
> > To: "Sean Owen [via Apache Spark Developers List]"
> > <ml...@n3.nabble.com>
> >
> >
> > I am not sure what you want me to check. Note that I see two
> takeSample()s
> > being invoked every single time I execute KMeans(). In a current job I
> have,
> > I did view the details and updated the:
> >
> > StackOverflow question.
> >
> >
> >
> > On Tue, Aug 30, 2016 at 9:25 AM, Sean Owen [via Apache Spark Developers
> > List] <ml...@n3.nabble.com> wrote:
> >>
> >> I'm not sure it's a UI bug; it really does record two different
> >> stages, the second of which executes quickly. I am not sure why that
> >> would happen off the top of my head. I don't see anything that failed
> >> here.
> >>
> >> Digging into those two stages and what they executed might give a clue
> >> to what's really going on there.
> >>
> >> On Tue, Aug 30, 2016 at 5:18 PM, gsamaras <[hidden email]> wrote:
> >> > Yanbo thank you for your reply. So you are saying that this is a bug
> in
> >> > the
> >> > Spark UI in general, and not in the local Spark UI of our cluster,
> where
> >> > I
> >> > work, right?
> >> >
> >> > George
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe e-mail: [hidden email]
> >>
> >>
> >>
> >> ________________________________
> >> If you reply to this email, your message will be added to the discussion
> >> below:
> >>
> >> http://apache-spark-developers-list.1001551.n3.nabble.com/KMeans-calls-
> takeSample-twice-tp18761p18788.html
> >> To unsubscribe from KMeans calls takeSample() twice?, click here.
> >> NAML
> >
> >
> >
>
Re: KMeans calls takeSample() twice?
Posted by Shivaram Venkataraman <sh...@eecs.berkeley.edu>.
I think takeSample itself runs multiple jobs if the amount of samples
collected in the first pass is not enough. The comment and code path
at https://github.com/apache/spark/blob/412b0e8969215411b97efd3d0984dc6cac5d31e0/core/src/main/scala/org/apache/spark/rdd/RDD.scala#L508
should explain when this happens. Also you can confirm this by
checking if the logWarning shows up in your logs.
Thanks
Shivaram
On Tue, Aug 30, 2016 at 9:50 AM, Georgios Samaras
<ge...@gmail.com> wrote:
>
> ---------- Forwarded message ----------
> From: Georgios Samaras <ge...@gmail.com>
> Date: Tue, Aug 30, 2016 at 9:49 AM
> Subject: Re: KMeans calls takeSample() twice?
> To: "Sean Owen [via Apache Spark Developers List]"
> <ml...@n3.nabble.com>
>
>
> I am not sure what you want me to check. Note that I see two takeSample()s
> being invoked every single time I execute KMeans(). In a current job I have,
> I did view the details and updated the:
>
> StackOverflow question.
>
>
>
> On Tue, Aug 30, 2016 at 9:25 AM, Sean Owen [via Apache Spark Developers
> List] <ml...@n3.nabble.com> wrote:
>>
>> I'm not sure it's a UI bug; it really does record two different
>> stages, the second of which executes quickly. I am not sure why that
>> would happen off the top of my head. I don't see anything that failed
>> here.
>>
>> Digging into those two stages and what they executed might give a clue
>> to what's really going on there.
>>
>> On Tue, Aug 30, 2016 at 5:18 PM, gsamaras <[hidden email]> wrote:
>> > Yanbo thank you for your reply. So you are saying that this is a bug in
>> > the
>> > Spark UI in general, and not in the local Spark UI of our cluster, where
>> > I
>> > work, right?
>> >
>> > George
>>
>> ---------------------------------------------------------------------
>> To unsubscribe e-mail: [hidden email]
>>
>>
>>
>> ________________________________
>> If you reply to this email, your message will be added to the discussion
>> below:
>>
>> http://apache-spark-developers-list.1001551.n3.nabble.com/KMeans-calls-takeSample-twice-tp18761p18788.html
>> To unsubscribe from KMeans calls takeSample() twice?, click here.
>> NAML
>
>
>
---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
Fwd: KMeans calls takeSample() twice?
Posted by Georgios Samaras <ge...@gmail.com>.
---------- Forwarded message ----------
From: Georgios Samaras <ge...@gmail.com>
Date: Tue, Aug 30, 2016 at 9:49 AM
Subject: Re: KMeans calls takeSample() twice?
To: "Sean Owen [via Apache Spark Developers List]" <
ml-node+s1001551n18788h58@n3.nabble.com>
I am not sure what you want me to check. Note that I see two takeSample()s
being invoked every single time I execute KMeans(). In a current job I
have, I did view the details and updated the:
StackOverflow question.
<http://stackoverflow.com/questions/38986395/sparkkmeans-calls-takesample-twice>
On Tue, Aug 30, 2016 at 9:25 AM, Sean Owen [via Apache Spark Developers
List] <ml...@n3.nabble.com> wrote:
> I'm not sure it's a UI bug; it really does record two different
> stages, the second of which executes quickly. I am not sure why that
> would happen off the top of my head. I don't see anything that failed
> here.
>
> Digging into those two stages and what they executed might give a clue
> to what's really going on there.
>
> On Tue, Aug 30, 2016 at 5:18 PM, gsamaras <[hidden email]
> <http:///user/SendEmail.jtp?type=node&node=18788&i=0>> wrote:
> > Yanbo thank you for your reply. So you are saying that this is a bug in
> the
> > Spark UI in general, and not in the local Spark UI of our cluster, where
> I
> > work, right?
> >
> > George
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: [hidden email]
> <http:///user/SendEmail.jtp?type=node&node=18788&i=1>
>
>
>
> ------------------------------
> If you reply to this email, your message will be added to the discussion
> below:
> http://apache-spark-developers-list.1001551.n3.nabble.com/
> KMeans-calls-takeSample-twice-tp18761p18788.html
> To unsubscribe from KMeans calls takeSample() twice?, click here
> <http://apache-spark-developers-list.1001551.n3.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=18761&code=Z2VvcmdlU2FtYXJhc0RJVEBnbWFpbC5jb218MTg3NjF8LTE1MTg1NDIwOQ==>
> .
> NAML
> <http://apache-spark-developers-list.1001551.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>
Re: KMeans calls takeSample() twice?
Posted by gsamaras <ge...@gmail.com>.
I am not sure what you want me to check. Note that I see two takeSample()s
being invoked every single time I execute KMeans(). In a current job I
have, I did view the details and updated the:
StackOverflow question.
<http://stackoverflow.com/questions/38986395/sparkkmeans-calls-takesample-twice>
On Tue, Aug 30, 2016 at 9:25 AM, Sean Owen [via Apache Spark Developers
List] <ml...@n3.nabble.com> wrote:
> I'm not sure it's a UI bug; it really does record two different
> stages, the second of which executes quickly. I am not sure why that
> would happen off the top of my head. I don't see anything that failed
> here.
>
> Digging into those two stages and what they executed might give a clue
> to what's really going on there.
>
> On Tue, Aug 30, 2016 at 5:18 PM, gsamaras <[hidden email]
> <http:///user/SendEmail.jtp?type=node&node=18788&i=0>> wrote:
> > Yanbo thank you for your reply. So you are saying that this is a bug in
> the
> > Spark UI in general, and not in the local Spark UI of our cluster, where
> I
> > work, right?
> >
> > George
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: [hidden email]
> <http:///user/SendEmail.jtp?type=node&node=18788&i=1>
>
>
>
> ------------------------------
> If you reply to this email, your message will be added to the discussion
> below:
> http://apache-spark-developers-list.1001551.n3.nabble.com/KMeans-calls-
> takeSample-twice-tp18761p18788.html
> To unsubscribe from KMeans calls takeSample() twice?, click here
> <http://apache-spark-developers-list.1001551.n3.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=18761&code=Z2VvcmdlU2FtYXJhc0RJVEBnbWFpbC5jb218MTg3NjF8LTE1MTg1NDIwOQ==>
> .
> NAML
> <http://apache-spark-developers-list.1001551.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>
--
View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/KMeans-calls-takeSample-twice-tp18761p18789.html
Sent from the Apache Spark Developers List mailing list archive at Nabble.com.
Re: KMeans calls takeSample() twice?
Posted by Sean Owen <so...@cloudera.com>.
I'm not sure it's a UI bug; it really does record two different
stages, the second of which executes quickly. I am not sure why that
would happen off the top of my head. I don't see anything that failed
here.
Digging into those two stages and what they executed might give a clue
to what's really going on there.
On Tue, Aug 30, 2016 at 5:18 PM, gsamaras <ge...@gmail.com> wrote:
> Yanbo thank you for your reply. So you are saying that this is a bug in the
> Spark UI in general, and not in the local Spark UI of our cluster, where I
> work, right?
>
> George
---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
Re: KMeans calls takeSample() twice?
Posted by gsamaras <ge...@gmail.com>.
Yanbo thank you for your reply. So you are saying that this is a bug in the
Spark UI in general, and not in the local Spark UI of our cluster, where I
work, right?
George
On Mon, Aug 29, 2016 at 11:55 PM, Yanbo Liang-2 [via Apache Spark
Developers List] <ml...@n3.nabble.com> wrote:
> I run KMeans with probes and found that takeSample() was called only once
> actually. It looks like this issue was caused by mistake display at Spark
> UI.
>
> Thanks
> Yanbo
>
> On Mon, Aug 29, 2016 at 2:34 PM, gsamaras <[hidden email]
> <http:///user/SendEmail.jtp?type=node&node=18768&i=0>> wrote:
>
>> After reading the internal code of Spark about it, I wasn't able to
>> understand why it calls takeSample() twice? Can someone please explain?
>>
>> There is a relevant StackOverflow question
>> <http://stackoverflow.com/questions/38986395/sparkkmeans-
>> calls-takesample-twice>
>> .
>>
>>
>>
>> --
>> View this message in context: http://apache-spark-developers
>> -list.1001551.n3.nabble.com/KMeans-calls-takeSample-twice-tp18761.html
>> Sent from the Apache Spark Developers List mailing list archive at
>> Nabble.com.
>>
>> ---------------------------------------------------------------------
>> To unsubscribe e-mail: [hidden email]
>> <http:///user/SendEmail.jtp?type=node&node=18768&i=1>
>>
>>
>
>
> ------------------------------
> If you reply to this email, your message will be added to the discussion
> below:
> http://apache-spark-developers-list.1001551.n3.nabble.com/KMeans-calls-
> takeSample-twice-tp18761p18768.html
> To unsubscribe from KMeans calls takeSample() twice?, click here
> <http://apache-spark-developers-list.1001551.n3.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=18761&code=Z2VvcmdlU2FtYXJhc0RJVEBnbWFpbC5jb218MTg3NjF8LTE1MTg1NDIwOQ==>
> .
> NAML
> <http://apache-spark-developers-list.1001551.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>
--
View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/KMeans-calls-takeSample-twice-tp18761p18786.html
Sent from the Apache Spark Developers List mailing list archive at Nabble.com.
Re: KMeans calls takeSample() twice?
Posted by Yanbo Liang <yb...@gmail.com>.
I run KMeans with probes and found that takeSample() was called only once
actually. It looks like this issue was caused by mistake display at Spark
UI.
Thanks
Yanbo
On Mon, Aug 29, 2016 at 2:34 PM, gsamaras <ge...@gmail.com>
wrote:
> After reading the internal code of Spark about it, I wasn't able to
> understand why it calls takeSample() twice? Can someone please explain?
>
> There is a relevant StackOverflow question
> <http://stackoverflow.com/questions/38986395/sparkkmeans-calls-takesample-
> twice>
> .
>
>
>
> --
> View this message in context: http://apache-spark-
> developers-list.1001551.n3.nabble.com/KMeans-calls-
> takeSample-twice-tp18761.html
> Sent from the Apache Spark Developers List mailing list archive at
> Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>
>