You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@spark.apache.org by Pa Rö <pa...@googlemail.com> on 2015/07/20 16:51:00 UTC

k-means iteration not terminate

hi community,

i have write a spark k-means app. now i run it on a cluster.
my job start and at iteration nine or ten the process stop.
in the spark dashbord all time shown "is running", but nothing
happend, no exceptions.

my setting is the following:
1000 input points
k=10
maxIteration=30
a tree node cluster (one node have 16GB RAM und 8cores i7)
i use cloudera live 5.4.4 with spark 1.3

maybe spark need more memory or i have a wrong setting?

best regards,
paul

Re: k-means iteration not terminate

Posted by Pa Rö <pa...@googlemail.com>.

thanks for this information, but i use cloudera live 5.4.4, and that have
only spark 1.3. a newer version is not avaible.
i don't understand this problem, first it compute some iterations and than
it "stop" better "do nothing". i think the problem
is not find in the program code.
maybe you know a other way to fix the problem?

2015-07-21 10:09 GMT+02:00 Akhil Das <ak...@sigmoidanalytics.com>:

> It could be a GC pause or something, you need to check in the stages tab
> and see what is taking time, If you upgrade to Spark 1.4, it has better UI
> and DAG visualization which helps you debug better.
>
> Thanks
> Best Regards
>
> On Mon, Jul 20, 2015 at 8:21 PM, Pa Rö <pa...@googlemail.com>
> wrote:
>
>> hi community,
>>
>> i have write a spark k-means app. now i run it on a cluster.
>> my job start and at iteration nine or ten the process stop.
>> in the spark dashbord all time shown "is running", but nothing
>> happend, no exceptions.
>>
>> my setting is the following:
>> 1000 input points
>> k=10
>> maxIteration=30
>> a tree node cluster (one node have 16GB RAM und 8cores i7)
>> i use cloudera live 5.4.4 with spark 1.3
>>
>> maybe spark need more memory or i have a wrong setting?
>>
>> best regards,
>> paul
>>
>
>

Re: k-means iteration not terminate

Posted by Akhil Das <ak...@sigmoidanalytics.com>.

It could be a GC pause or something, you need to check in the stages tab
and see what is taking time, If you upgrade to Spark 1.4, it has better UI
and DAG visualization which helps you debug better.

Thanks
Best Regards

On Mon, Jul 20, 2015 at 8:21 PM, Pa Rö <pa...@googlemail.com>
wrote:

> hi community,
>
> i have write a spark k-means app. now i run it on a cluster.
> my job start and at iteration nine or ten the process stop.
> in the spark dashbord all time shown "is running", but nothing
> happend, no exceptions.
>
> my setting is the following:
> 1000 input points
> k=10
> maxIteration=30
> a tree node cluster (one node have 16GB RAM und 8cores i7)
> i use cloudera live 5.4.4 with spark 1.3
>
> maybe spark need more memory or i have a wrong setting?
>
> best regards,
> paul
>