SPARK OWLQN Exception: Iteration Stage is so slow

[John Wu <jo...@zamplus.com>]

Hi all,

 

There is a problem we can’t resolve. We implement the OWLQN algorithm in
parallel with SPARK, 

We don’t know why It is very slow in every iteration stage, but the load of
CPU and Memory of each executor are so low that it seems impossible to make
the the every step slow.

 

And  there are so many info log from stdout like this:

 

BlockManagerMasterActor$BlockManagerInfo: Removed taskresult_xxx on
SHXJ-Hx-HBxxx:44126 in memory

 

Thank you.

 

John Wu 

晶赞广告（上海）有限公司 
Zamplus Advertising (Shanghai) Co., Ltd. 
Tel: +8621-6076 0818 Ext. 885
Fax: +8621-6076 0812 
Mobile: +86-13817415695
Room 1105, Tower 3, No. 695, Lingshi Rd., Shanghai 200072, P. R. China 
上海市灵石路695号珠江创业园区3号楼1105室 

Zamplus 
----------------------------------------------------------------------------
-------- 
免责声明：本邮件及其附件可能含有机密信息并受法律保护。如果您错误地收 到此邮
件，请不要转发给任何人，请立即将此邮件所有的副本、及其全部附件彻底删除，并请
告知发件人此邮件被发至错误的收件人。发送人在本邮件 下表达的观点并不一定代表
晶赞广告（上海）有限公司与UMA优质受众营销联盟的观点。晶赞广告（上海）有限公
司与UMA优质受众营销联盟并不 保证本邮件是安全或不受任何计算机病毒影响的，并且
对由于邮件传输而导致的邮件内容错误或缺失不承担任何责任。除非明确说明，本邮件
并不构成 具有约束力的契约。 
----------------------------------------------------------------------------
-------

 

Re: SPARK OWLQN Exception: Iteration Stage is so slow

[Xiangrui Meng <me...@gmail.com>]

Do you mind sharing more details, for example, specs of nodes and data
size? -Xiangrui

2014-07-29 2:51 GMT-07:00 John Wu <jo...@zamplus.com>:
> Hi all,
>
>
>
> There is a problem we can’t resolve. We implement the OWLQN algorithm in
> parallel with SPARK,
>
> We don’t know why It is very slow in every iteration stage, but the load of
> CPU and Memory of each executor are so low that it seems impossible to make
> the the every step slow.
>
>
>
> And  there are so many info log from stdout like this:
>
>
>
> BlockManagerMasterActor$BlockManagerInfo: Removed taskresult_xxx on
> SHXJ-Hx-HBxxx:44126 in memory
>
>
>
> Thank you.
>
>
>
> John Wu
>
> 晶赞广告（上海）有限公司
> Zamplus Advertising (Shanghai) Co., Ltd.
> Tel: +8621-6076 0818 Ext. 885
> Fax: +8621-6076 0812
> Mobile: +86-13817415695
> Room 1105, Tower 3, No. 695, Lingshi Rd., Shanghai 200072, P. R. China
> 上海市灵石路695号珠江创业园区3号楼1105室
>
> Zamplus
> ------------------------------------------------------------------------------------
> 免责声明：本邮件及其附件可能含有机密信息并受法律保护。如果您错误地收
> 到此邮件，请不要转发给任何人，请立即将此邮件所有的副本、及其全部附件彻底删除，并请告知发件人此邮件被发至错误的收件人。发送人在本邮件
> 下表达的观点并不一定代表晶赞广告（上海）有限公司与UMA优质受众营销联盟的观点。晶赞广告（上海）有限公司与UMA优质受众营销联盟并不
> 保证本邮件是安全或不受任何计算机病毒影响的，并且对由于邮件传输而导致的邮件内容错误或缺失不承担任何责任。除非明确说明，本邮件并不构成 具有约束力的契约。
> -----------------------------------------------------------------------------------
>
>