You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@kylin.apache.org by ro...@stratebi.com on 2018/05/07 12:33:32 UTC

Increase of time between steps after Kylin 2.3.1 update

Hi,

 

Some days ago we have updated Kylin 2.2 to Kylin 2.3.1. We performed the
update process with successful. However, we have experimented an increase of
time wasted between steps during building time. For example, the time
between step 7 “Build N-Dimension Cuboid:level 1” and “…level2” is now about
30 seconds, a lot of time if we take into account that in these steps no
processing is executed since we use the "layer" algorithm (Screenshot 1).
This time is much longer than when we used Kylin 2.2, were time between
these skipped steps was about 5 seconds. (Screenshot 2)

 

Have you experienced someone a similar behavior? What could be the cause?

 

Environment:

·         EMR 5.7 (S3 storage for Hive and HBase)

·         Kylin 2.3.1 (updated from 2.2 last Friday). Installed on separated
EC2 instance.

 

Thanks in advance,

 

Screenshot 1:

 

 



 

 

Screenshot 2:

 

 



Roberto Tardío Olmos

Senior Big Data & Business Intelligence Consultant

Avenida de Brasil, 17, Planta 16.28020 Madrid

Fijo: 91.788.34.10




 

 <http://bigdata.stratebi.com/> http://bigdata.stratebi.com/ 

 

 <http://www.stratebi.com> http://www.stratebi.com 

 


RE: Increase of time between steps after Kylin 2.3.1 update

Posted by ro...@stratebi.com.
Thanks ShaoFeng,

 

I have created a new JIRA 

 

https://issues.apache.org/jira/browse/KYLIN-3371

 

From: ShaoFeng Shi [mailto:shaofengshi@apache.org] 
Sent: martes, 8 de mayo de 2018 15:49
To: user <us...@kylin.apache.org>
Subject: Re: Increase of time between steps after Kylin 2.3.1 update

 

Hi Roberto,

 

I have no idea about this time increasing. It seems like some intervals got increased. Maybe other guys have the idea. Would you like to fire a JIRA for tracking this? Thanks!

 

 

2018-05-07 21:52 GMT+08:00 <roberto.tardio@stratebi.com <ma...@stratebi.com> >:

Checking the log I verify that more time is spent between one step and another in our installation of Kylin 2.3.1 than when we used Kylin 2.2.  In addition, JOBS are in PENDING status longer in the job monitor.

 

Kylin 2.2

 

2018-05-04 14:25:40,797 INFO  [Scheduler 542798956 Job cb843ecc-5062-49ec-a881-215097e4501a-518] execution.ExecutableManager:421 : job id:cb843ecc-5062-49ec-a881-215097e4501a from RUNNING to READY

2018-05-04 14:25:45,496 INFO  [Scheduler 542798956 Job cb843ecc-5062-49ec-a881-215097e4501a-518] execution.AbstractExecutable:111 : Executing AbstractExecutable (BUILD CUBE - Captacion_Resumido_Cubo_v3 - 20180504000000_20180505000000 - GMT+02:00 2018-05-04 14:21:34)

 

 

Kylin 2.3.1

 

2018-05-07 15:25:26,338 INFO  [Scheduler 1942482192 Job 6d616841-1c4d-46ca-a1b9-a18047fbada5-1006] execution.ExecutableManager:411 : job id:6d616841-1c4d-46ca-a1b9-a18047fbada5 from RUNNING to READY

2018-05-07 15:25:48,365 INFO  [Scheduler 1942482192 Job 6d616841-1c4d-46ca-a1b9-a18047fbada5-1006] execution.AbstractExecutable:147 : Executing AbstractExecutable (BUILD CUBE - Captacion_Resumido_Cubo_v3 - 20180507000000_20180508000000 - GMT+02:00 2018-05-07 15:19:41)

 

Regards,

 

From: roberto.tardio@stratebi.com <ma...@stratebi.com>  [mailto:roberto.tardio@stratebi.com <ma...@stratebi.com> ] 
Sent: lunes, 7 de mayo de 2018 14:34
To: user@kylin.apache.org <ma...@kylin.apache.org> 
Subject: Increase of time between steps after Kylin 2.3.1 update

 

Hi,

 

Some days ago we have updated Kylin 2.2 to Kylin 2.3.1. We performed the update process with successful. However, we have experimented an increase of time wasted between steps during building time. For example, the time between step 7 “Build N-Dimension Cuboid:level 1” and “…level2” is now about 30 seconds, a lot of time if we take into account that in these steps no processing is executed since we use the "layer" algorithm (Screenshot 1). This time is much longer than when we used Kylin 2.2, were time between these skipped steps was about 5 seconds. (Screenshot 2)

 

Have you experienced someone a similar behavior? What could be the cause?

 

Environment:

·         EMR 5.7 (S3 storage for Hive and HBase)

·         Kylin 2.3.1 (updated from 2.2 last Friday). Installed on separated EC2 instance.

 

Thanks in advance,

 

Screenshot 1:

 

 



 

 

Screenshot 2:

 

 



Roberto Tardío Olmos

Senior Big Data & Business Intelligence Consultant

Avenida de Brasil, 17 <https://maps.google.com/?q=Avenida+de+Brasil,+17&entry=gmail&source=g> , Planta 16.28020 Madrid

Fijo: 91.788.34.10




 

http://bigdata.stratebi.com/ 

 

http://www.stratebi.com 

 





 

-- 

Best regards,

 

Shaofeng Shi 史少锋

 


Re: Increase of time between steps after Kylin 2.3.1 update

Posted by ShaoFeng Shi <sh...@apache.org>.
Hi Roberto,

I have no idea about this time increasing. It seems like some intervals got
increased. Maybe other guys have the idea. Would you like to fire a JIRA
for tracking this? Thanks!


2018-05-07 21:52 GMT+08:00 <ro...@stratebi.com>:

> Checking the log I verify that more time is spent between one step and
> another in our installation of Kylin 2.3.1 than when we used Kylin 2.2.  In
> addition, JOBS are in PENDING status longer in the job monitor.
>
>
>
> Kylin 2.2
>
>
>
> 2018-05-04 14:25:40,797 INFO  [Scheduler 542798956 Job
> cb843ecc-5062-49ec-a881-215097e4501a-518] execution.ExecutableManager:421
> : job id:cb843ecc-5062-49ec-a881-215097e4501a from RUNNING to READY
>
> 2018-05-04 14:25:45,496 INFO  [Scheduler 542798956 Job
> cb843ecc-5062-49ec-a881-215097e4501a-518] execution.AbstractExecutable:111
> : Executing AbstractExecutable (BUILD CUBE - Captacion_Resumido_Cubo_v3 -
> 20180504000000_20180505000000 - GMT+02:00 2018-05-04 14:21:34)
>
>
>
>
>
> Kylin 2.3.1
>
>
>
> 2018-05-07 15:25:26,338 INFO  [Scheduler 1942482192 Job
> 6d616841-1c4d-46ca-a1b9-a18047fbada5-1006] execution.ExecutableManager:411
> : job id:6d616841-1c4d-46ca-a1b9-a18047fbada5 from RUNNING to READY
>
> 2018-05-07 15:25:48,365 INFO  [Scheduler 1942482192 Job
> 6d616841-1c4d-46ca-a1b9-a18047fbada5-1006] execution.AbstractExecutable:147
> : Executing AbstractExecutable (BUILD CUBE - Captacion_Resumido_Cubo_v3 -
> 20180507000000_20180508000000 - GMT+02:00 2018-05-07 15:19:41)
>
>
>
> Regards,
>
>
>
> *From:* roberto.tardio@stratebi.com [mailto:roberto.tardio@stratebi.com]
> *Sent:* lunes, 7 de mayo de 2018 14:34
> *To:* user@kylin.apache.org
> *Subject:* Increase of time between steps after Kylin 2.3.1 update
>
>
>
> Hi,
>
>
>
> Some days ago we have updated Kylin 2.2 to Kylin 2.3.1. We performed the
> update process with successful. However, we have experimented an increase
> of time wasted between steps during building time. For example, the time
> between step 7 “Build N-Dimension Cuboid:level 1” and “…level2” is now
> about 30 seconds, a lot of time if we take into account that in these steps
> no processing is executed since we use the "layer" algorithm (Screenshot
> 1). This time is much longer than when we used Kylin 2.2, were time between
> these skipped steps was about 5 seconds. (Screenshot 2)
>
>
>
> Have you experienced someone a similar behavior? What could be the cause?
>
>
>
> Environment:
>
> ·         EMR 5.7 (S3 storage for Hive and HBase)
>
> ·         Kylin 2.3.1 (updated from 2.2 last Friday). Installed on
> separated EC2 instance.
>
>
>
> Thanks in advance,
>
>
>
> Screenshot 1:
>
>
>
>
>
> [image: cid:image003.png@01D3E60C.42049930]
>
>
>
>
>
> Screenshot 2:
>
>
>
>
>
> *Roberto Tardío Olmos*
>
> *Senior Big Data & Business Intelligence Consultant*
>
> Avenida de Brasil, 17
> <https://maps.google.com/?q=Avenida+de+Brasil,+17&entry=gmail&source=g>,
> Planta 16.28020 Madrid
>
> Fijo: 91.788.34.10
>
>
> [image:
> http://www.stratebi.com/image/layout_set_logo?img_id=21615&t=1486381163544]
>
>
>
> http://bigdata.stratebi.com/
>
>
>
> http://www.stratebi.com
>
>
>



-- 
Best regards,

Shaofeng Shi 史少锋

RE: Increase of time between steps after Kylin 2.3.1 update

Posted by ro...@stratebi.com.
Checking the log I verify that more time is spent between one step and
another in our installation of Kylin 2.3.1 than when we used Kylin 2.2.  In
addition, JOBS are in PENDING status longer in the job monitor.

 

Kylin 2.2

 

2018-05-04 14:25:40,797 INFO  [Scheduler 542798956 Job
cb843ecc-5062-49ec-a881-215097e4501a-518] execution.ExecutableManager:421 :
job id:cb843ecc-5062-49ec-a881-215097e4501a from RUNNING to READY

2018-05-04 14:25:45,496 INFO  [Scheduler 542798956 Job
cb843ecc-5062-49ec-a881-215097e4501a-518] execution.AbstractExecutable:111 :
Executing AbstractExecutable (BUILD CUBE - Captacion_Resumido_Cubo_v3 -
20180504000000_20180505000000 - GMT+02:00 2018-05-04 14:21:34)

 

 

Kylin 2.3.1

 

2018-05-07 15:25:26,338 INFO  [Scheduler 1942482192 Job
6d616841-1c4d-46ca-a1b9-a18047fbada5-1006] execution.ExecutableManager:411 :
job id:6d616841-1c4d-46ca-a1b9-a18047fbada5 from RUNNING to READY

2018-05-07 15:25:48,365 INFO  [Scheduler 1942482192 Job
6d616841-1c4d-46ca-a1b9-a18047fbada5-1006] execution.AbstractExecutable:147
: Executing AbstractExecutable (BUILD CUBE - Captacion_Resumido_Cubo_v3 -
20180507000000_20180508000000 - GMT+02:00 2018-05-07 15:19:41)

 

Regards,

 

From: roberto.tardio@stratebi.com [mailto:roberto.tardio@stratebi.com] 
Sent: lunes, 7 de mayo de 2018 14:34
To: user@kylin.apache.org
Subject: Increase of time between steps after Kylin 2.3.1 update

 

Hi,

 

Some days ago we have updated Kylin 2.2 to Kylin 2.3.1. We performed the
update process with successful. However, we have experimented an increase of
time wasted between steps during building time. For example, the time
between step 7 “Build N-Dimension Cuboid:level 1” and “…level2” is now about
30 seconds, a lot of time if we take into account that in these steps no
processing is executed since we use the "layer" algorithm (Screenshot 1).
This time is much longer than when we used Kylin 2.2, were time between
these skipped steps was about 5 seconds. (Screenshot 2)

 

Have you experienced someone a similar behavior? What could be the cause?

 

Environment:

·         EMR 5.7 (S3 storage for Hive and HBase)

·         Kylin 2.3.1 (updated from 2.2 last Friday). Installed on separated
EC2 instance.

 

Thanks in advance,

 

Screenshot 1:

 

 



 

 

Screenshot 2:

 

 



Roberto Tardío Olmos

Senior Big Data & Business Intelligence Consultant

Avenida de Brasil, 17, Planta 16.28020 Madrid

Fijo: 91.788.34.10




 

 <http://bigdata.stratebi.com/> http://bigdata.stratebi.com/ 

 

 <http://www.stratebi.com> http://www.stratebi.com