You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kylin.apache.org by 康凯森 <ka...@qq.com> on 2017/04/17 11:39:19 UTC
回复：cube building taking 8 hours just for 6 thousands records
what's the slowest step in your cube job?
please refer to http://kylin.apache.org/docs20/howto/howto_optimize_build.html
------------------ 原始邮件 ------------------
发件人: "suresh m";<su...@gmail.com>;
发送时间: 2017年4月17日(星期一) 晚上7:05
收件人: "dev"<de...@kylin.apache.org>;
主题: cube building taking 8 hours just for 6 thousands records
Hi All,
One of my cube taking hours to build even though its having less
records(6000), please provide me some tuning techniques to improve
performance in Kylin.
Regards.
Suresh
Re: cube building taking 8 hours just for 6 thousands records
Posted by Li Yang <li...@apache.org>.
Then it boils down to MR job tuning. You shall identify the slow MR job and
take action from there. There are many ways to diagnose and tune MR job.
Try google 'tune mapreduce job'.
On Fri, Apr 28, 2017 at 6:07 PM, suresh m <su...@gmail.com> wrote:
> Hi Thanks for your timely reply.
>
> I set kylin.cube.algorithm=layer , but still facing same issue.
>
> Please let me know any other tuning to improve performance.
>
> Regards,
>
> Suresh
>
> On Tue, Apr 18, 2017 at 7:23 PM, Li Yang <li...@apache.org> wrote:
>
> > > #24 Step Name: Build Cube
> >
> > It is in-mem cubing taking effect and seems it does not fit your cluster.
> > Try set below in kylin.properties to use layer cubing instead.
> >
> > kylin.cube.algorithm=layer
> >
> >
> > Cheers
> > Yang
> >
> > On Tue, Apr 18, 2017 at 9:13 PM, Alberto Ramón <
> a.ramonportoles@gmail.com>
> > wrote:
> >
> > > Is this your configuration? (check this properties):
> > >
> > > -
> > >
> > > yarn.nodemanager.resource.memory-mb =150/3
> > > -
> > >
> > > yarn.nodemanager.resource.cpu-vcores= 36/3
> > >
> > > And Check the YARN UI, during build phase (you can see how many Mappers
> > and
> > > reduce, RAM and CPU are you using)
> > >
> > > 2017-04-18 13:45 GMT+01:00 suresh m <su...@gmail.com>:
> > >
> > > > sorry it is for all 3 nodes.
> > > >
> > > > each node 55gb ram and 12 cpu are there.
> > > >
> > > > On Tue, Apr 18, 2017 at 6:09 PM, Alberto Ramón <
> > > a.ramonportoles@gmail.com>
> > > > wrote:
> > > >
> > > > > Are these 150 GB and 36 CPUs configured in?
> > > > >
> > > > > -
> > > > >
> > > > > yarn.nodemanager.resource.memory-mb
> > > > > -
> > > > >
> > > > > yarn.nodemanager.resource.cpu-vcores
> > > > >
> > > > > During Build process, you can open YARN UI how many map-reduce are
> > you
> > > > > using.
> > > > >
> > > > > 2017-04-18 13:21 GMT+01:00 suresh m <su...@gmail.com>:
> > > > >
> > > > > > Hi,
> > > > > >
> > > > > > I can see 150gb ram and 36 cpu in our cluster.
> > > > > >
> > > > > > On Tue, Apr 18, 2017 at 12:48 PM, Alberto Ramón <
> > > > > a.ramonportoles@gmail.com
> > > > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Can you check how many ram and cpu is asigned to yarn?
> > > > > > >
> > > > > > > El 18 abr. 2017 7:40 a. m., "suresh m" <suresh.m8687@gmail.com
> >
> > > > > > escribió:
> > > > > > >
> > > > > > > > Hi This is the time consuming step in entire build process.
> > > > > > > >
> > > > > > > > #24 Step Name: Build Cube
> > > > > > > >
> > > > > > > > Please help me to get it resolved.
> > > > > > > >
> > > > > > > > Regards,
> > > > > > > > Suresh
> > > > > > > >
> > > > > > > > On Mon, Apr 17, 2017 at 5:09 PM, 康凯森 <ka...@qq.com>
> > wrote:
> > > > > > > >
> > > > > > > > > what's the slowest step in your cube job?
> > > > > > > > > please refer to http://kylin.apache.org/
> > > > > > docs20/howto/howto_optimize_
> > > > > > > > > build.html
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > ------------------ 原始邮件 ------------------
> > > > > > > > > 发件人: "suresh m";<su...@gmail.com>;
> > > > > > > > > 发送时间: 2017年4月17日(星期一) 晚上7:05
> > > > > > > > > 收件人: "dev"<de...@kylin.apache.org>;
> > > > > > > > >
> > > > > > > > > 主题: cube building taking 8 hours just for 6 thousands
> records
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Hi All,
> > > > > > > > >
> > > > > > > > > One of my cube taking hours to build even though its having
> > > less
> > > > > > > > > records(6000), please provide me some tuning techniques to
> > > > improve
> > > > > > > > > performance in Kylin.
> > > > > > > > >
> > > > > > > > > Regards.
> > > > > > > > > Suresh
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: cube building taking 8 hours just for 6 thousands records
Posted by suresh m <su...@gmail.com>.
Hi Thanks for your timely reply.
I set kylin.cube.algorithm=layer , but still facing same issue.
Please let me know any other tuning to improve performance.
Regards,
Suresh
On Tue, Apr 18, 2017 at 7:23 PM, Li Yang <li...@apache.org> wrote:
> > #24 Step Name: Build Cube
>
> It is in-mem cubing taking effect and seems it does not fit your cluster.
> Try set below in kylin.properties to use layer cubing instead.
>
> kylin.cube.algorithm=layer
>
>
> Cheers
> Yang
>
> On Tue, Apr 18, 2017 at 9:13 PM, Alberto Ramón <a....@gmail.com>
> wrote:
>
> > Is this your configuration? (check this properties):
> >
> > -
> >
> > yarn.nodemanager.resource.memory-mb =150/3
> > -
> >
> > yarn.nodemanager.resource.cpu-vcores= 36/3
> >
> > And Check the YARN UI, during build phase (you can see how many Mappers
> and
> > reduce, RAM and CPU are you using)
> >
> > 2017-04-18 13:45 GMT+01:00 suresh m <su...@gmail.com>:
> >
> > > sorry it is for all 3 nodes.
> > >
> > > each node 55gb ram and 12 cpu are there.
> > >
> > > On Tue, Apr 18, 2017 at 6:09 PM, Alberto Ramón <
> > a.ramonportoles@gmail.com>
> > > wrote:
> > >
> > > > Are these 150 GB and 36 CPUs configured in?
> > > >
> > > > -
> > > >
> > > > yarn.nodemanager.resource.memory-mb
> > > > -
> > > >
> > > > yarn.nodemanager.resource.cpu-vcores
> > > >
> > > > During Build process, you can open YARN UI how many map-reduce are
> you
> > > > using.
> > > >
> > > > 2017-04-18 13:21 GMT+01:00 suresh m <su...@gmail.com>:
> > > >
> > > > > Hi,
> > > > >
> > > > > I can see 150gb ram and 36 cpu in our cluster.
> > > > >
> > > > > On Tue, Apr 18, 2017 at 12:48 PM, Alberto Ramón <
> > > > a.ramonportoles@gmail.com
> > > > > >
> > > > > wrote:
> > > > >
> > > > > > Can you check how many ram and cpu is asigned to yarn?
> > > > > >
> > > > > > El 18 abr. 2017 7:40 a. m., "suresh m" <su...@gmail.com>
> > > > > escribió:
> > > > > >
> > > > > > > Hi This is the time consuming step in entire build process.
> > > > > > >
> > > > > > > #24 Step Name: Build Cube
> > > > > > >
> > > > > > > Please help me to get it resolved.
> > > > > > >
> > > > > > > Regards,
> > > > > > > Suresh
> > > > > > >
> > > > > > > On Mon, Apr 17, 2017 at 5:09 PM, 康凯森 <ka...@qq.com>
> wrote:
> > > > > > >
> > > > > > > > what's the slowest step in your cube job?
> > > > > > > > please refer to http://kylin.apache.org/
> > > > > docs20/howto/howto_optimize_
> > > > > > > > build.html
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > ------------------ 原始邮件 ------------------
> > > > > > > > 发件人: "suresh m";<su...@gmail.com>;
> > > > > > > > 发送时间: 2017年4月17日(星期一) 晚上7:05
> > > > > > > > 收件人: "dev"<de...@kylin.apache.org>;
> > > > > > > >
> > > > > > > > 主题: cube building taking 8 hours just for 6 thousands records
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > Hi All,
> > > > > > > >
> > > > > > > > One of my cube taking hours to build even though its having
> > less
> > > > > > > > records(6000), please provide me some tuning techniques to
> > > improve
> > > > > > > > performance in Kylin.
> > > > > > > >
> > > > > > > > Regards.
> > > > > > > > Suresh
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: cube building taking 8 hours just for 6 thousands records
Posted by Li Yang <li...@apache.org>.
> #24 Step Name: Build Cube
It is in-mem cubing taking effect and seems it does not fit your cluster.
Try set below in kylin.properties to use layer cubing instead.
kylin.cube.algorithm=layer
Cheers
Yang
On Tue, Apr 18, 2017 at 9:13 PM, Alberto Ramón <a....@gmail.com>
wrote:
> Is this your configuration? (check this properties):
>
> -
>
> yarn.nodemanager.resource.memory-mb =150/3
> -
>
> yarn.nodemanager.resource.cpu-vcores= 36/3
>
> And Check the YARN UI, during build phase (you can see how many Mappers and
> reduce, RAM and CPU are you using)
>
> 2017-04-18 13:45 GMT+01:00 suresh m <su...@gmail.com>:
>
> > sorry it is for all 3 nodes.
> >
> > each node 55gb ram and 12 cpu are there.
> >
> > On Tue, Apr 18, 2017 at 6:09 PM, Alberto Ramón <
> a.ramonportoles@gmail.com>
> > wrote:
> >
> > > Are these 150 GB and 36 CPUs configured in?
> > >
> > > -
> > >
> > > yarn.nodemanager.resource.memory-mb
> > > -
> > >
> > > yarn.nodemanager.resource.cpu-vcores
> > >
> > > During Build process, you can open YARN UI how many map-reduce are you
> > > using.
> > >
> > > 2017-04-18 13:21 GMT+01:00 suresh m <su...@gmail.com>:
> > >
> > > > Hi,
> > > >
> > > > I can see 150gb ram and 36 cpu in our cluster.
> > > >
> > > > On Tue, Apr 18, 2017 at 12:48 PM, Alberto Ramón <
> > > a.ramonportoles@gmail.com
> > > > >
> > > > wrote:
> > > >
> > > > > Can you check how many ram and cpu is asigned to yarn?
> > > > >
> > > > > El 18 abr. 2017 7:40 a. m., "suresh m" <su...@gmail.com>
> > > > escribió:
> > > > >
> > > > > > Hi This is the time consuming step in entire build process.
> > > > > >
> > > > > > #24 Step Name: Build Cube
> > > > > >
> > > > > > Please help me to get it resolved.
> > > > > >
> > > > > > Regards,
> > > > > > Suresh
> > > > > >
> > > > > > On Mon, Apr 17, 2017 at 5:09 PM, 康凯森 <ka...@qq.com> wrote:
> > > > > >
> > > > > > > what's the slowest step in your cube job?
> > > > > > > please refer to http://kylin.apache.org/
> > > > docs20/howto/howto_optimize_
> > > > > > > build.html
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > ------------------ 原始邮件 ------------------
> > > > > > > 发件人: "suresh m";<su...@gmail.com>;
> > > > > > > 发送时间: 2017年4月17日(星期一) 晚上7:05
> > > > > > > 收件人: "dev"<de...@kylin.apache.org>;
> > > > > > >
> > > > > > > 主题: cube building taking 8 hours just for 6 thousands records
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > Hi All,
> > > > > > >
> > > > > > > One of my cube taking hours to build even though its having
> less
> > > > > > > records(6000), please provide me some tuning techniques to
> > improve
> > > > > > > performance in Kylin.
> > > > > > >
> > > > > > > Regards.
> > > > > > > Suresh
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: cube building taking 8 hours just for 6 thousands records
Posted by Alberto Ramón <a....@gmail.com>.
Is this your configuration? (check this properties):
-
yarn.nodemanager.resource.memory-mb =150/3
-
yarn.nodemanager.resource.cpu-vcores= 36/3
And Check the YARN UI, during build phase (you can see how many Mappers and
reduce, RAM and CPU are you using)
2017-04-18 13:45 GMT+01:00 suresh m <su...@gmail.com>:
> sorry it is for all 3 nodes.
>
> each node 55gb ram and 12 cpu are there.
>
> On Tue, Apr 18, 2017 at 6:09 PM, Alberto Ramón <a....@gmail.com>
> wrote:
>
> > Are these 150 GB and 36 CPUs configured in?
> >
> > -
> >
> > yarn.nodemanager.resource.memory-mb
> > -
> >
> > yarn.nodemanager.resource.cpu-vcores
> >
> > During Build process, you can open YARN UI how many map-reduce are you
> > using.
> >
> > 2017-04-18 13:21 GMT+01:00 suresh m <su...@gmail.com>:
> >
> > > Hi,
> > >
> > > I can see 150gb ram and 36 cpu in our cluster.
> > >
> > > On Tue, Apr 18, 2017 at 12:48 PM, Alberto Ramón <
> > a.ramonportoles@gmail.com
> > > >
> > > wrote:
> > >
> > > > Can you check how many ram and cpu is asigned to yarn?
> > > >
> > > > El 18 abr. 2017 7:40 a. m., "suresh m" <su...@gmail.com>
> > > escribió:
> > > >
> > > > > Hi This is the time consuming step in entire build process.
> > > > >
> > > > > #24 Step Name: Build Cube
> > > > >
> > > > > Please help me to get it resolved.
> > > > >
> > > > > Regards,
> > > > > Suresh
> > > > >
> > > > > On Mon, Apr 17, 2017 at 5:09 PM, 康凯森 <ka...@qq.com> wrote:
> > > > >
> > > > > > what's the slowest step in your cube job?
> > > > > > please refer to http://kylin.apache.org/
> > > docs20/howto/howto_optimize_
> > > > > > build.html
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > ------------------ 原始邮件 ------------------
> > > > > > 发件人: "suresh m";<su...@gmail.com>;
> > > > > > 发送时间: 2017年4月17日(星期一) 晚上7:05
> > > > > > 收件人: "dev"<de...@kylin.apache.org>;
> > > > > >
> > > > > > 主题: cube building taking 8 hours just for 6 thousands records
> > > > > >
> > > > > >
> > > > > >
> > > > > > Hi All,
> > > > > >
> > > > > > One of my cube taking hours to build even though its having less
> > > > > > records(6000), please provide me some tuning techniques to
> improve
> > > > > > performance in Kylin.
> > > > > >
> > > > > > Regards.
> > > > > > Suresh
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: cube building taking 8 hours just for 6 thousands records
Posted by suresh m <su...@gmail.com>.
sorry it is for all 3 nodes.
each node 55gb ram and 12 cpu are there.
On Tue, Apr 18, 2017 at 6:09 PM, Alberto Ramón <a....@gmail.com>
wrote:
> Are these 150 GB and 36 CPUs configured in?
>
> -
>
> yarn.nodemanager.resource.memory-mb
> -
>
> yarn.nodemanager.resource.cpu-vcores
>
> During Build process, you can open YARN UI how many map-reduce are you
> using.
>
> 2017-04-18 13:21 GMT+01:00 suresh m <su...@gmail.com>:
>
> > Hi,
> >
> > I can see 150gb ram and 36 cpu in our cluster.
> >
> > On Tue, Apr 18, 2017 at 12:48 PM, Alberto Ramón <
> a.ramonportoles@gmail.com
> > >
> > wrote:
> >
> > > Can you check how many ram and cpu is asigned to yarn?
> > >
> > > El 18 abr. 2017 7:40 a. m., "suresh m" <su...@gmail.com>
> > escribió:
> > >
> > > > Hi This is the time consuming step in entire build process.
> > > >
> > > > #24 Step Name: Build Cube
> > > >
> > > > Please help me to get it resolved.
> > > >
> > > > Regards,
> > > > Suresh
> > > >
> > > > On Mon, Apr 17, 2017 at 5:09 PM, 康凯森 <ka...@qq.com> wrote:
> > > >
> > > > > what's the slowest step in your cube job?
> > > > > please refer to http://kylin.apache.org/
> > docs20/howto/howto_optimize_
> > > > > build.html
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > ------------------ 原始邮件 ------------------
> > > > > 发件人: "suresh m";<su...@gmail.com>;
> > > > > 发送时间: 2017年4月17日(星期一) 晚上7:05
> > > > > 收件人: "dev"<de...@kylin.apache.org>;
> > > > >
> > > > > 主题: cube building taking 8 hours just for 6 thousands records
> > > > >
> > > > >
> > > > >
> > > > > Hi All,
> > > > >
> > > > > One of my cube taking hours to build even though its having less
> > > > > records(6000), please provide me some tuning techniques to improve
> > > > > performance in Kylin.
> > > > >
> > > > > Regards.
> > > > > Suresh
> > > > >
> > > >
> > >
> >
>
Re: cube building taking 8 hours just for 6 thousands records
Posted by Alberto Ramón <a....@gmail.com>.
Are these 150 GB and 36 CPUs configured in?
-
yarn.nodemanager.resource.memory-mb
-
yarn.nodemanager.resource.cpu-vcores
During Build process, you can open YARN UI how many map-reduce are you
using.
2017-04-18 13:21 GMT+01:00 suresh m <su...@gmail.com>:
> Hi,
>
> I can see 150gb ram and 36 cpu in our cluster.
>
> On Tue, Apr 18, 2017 at 12:48 PM, Alberto Ramón <a.ramonportoles@gmail.com
> >
> wrote:
>
> > Can you check how many ram and cpu is asigned to yarn?
> >
> > El 18 abr. 2017 7:40 a. m., "suresh m" <su...@gmail.com>
> escribió:
> >
> > > Hi This is the time consuming step in entire build process.
> > >
> > > #24 Step Name: Build Cube
> > >
> > > Please help me to get it resolved.
> > >
> > > Regards,
> > > Suresh
> > >
> > > On Mon, Apr 17, 2017 at 5:09 PM, 康凯森 <ka...@qq.com> wrote:
> > >
> > > > what's the slowest step in your cube job?
> > > > please refer to http://kylin.apache.org/
> docs20/howto/howto_optimize_
> > > > build.html
> > > >
> > > >
> > > >
> > > >
> > > > ------------------ 原始邮件 ------------------
> > > > 发件人: "suresh m";<su...@gmail.com>;
> > > > 发送时间: 2017年4月17日(星期一) 晚上7:05
> > > > 收件人: "dev"<de...@kylin.apache.org>;
> > > >
> > > > 主题: cube building taking 8 hours just for 6 thousands records
> > > >
> > > >
> > > >
> > > > Hi All,
> > > >
> > > > One of my cube taking hours to build even though its having less
> > > > records(6000), please provide me some tuning techniques to improve
> > > > performance in Kylin.
> > > >
> > > > Regards.
> > > > Suresh
> > > >
> > >
> >
>
Re: cube building taking 8 hours just for 6 thousands records
Posted by suresh m <su...@gmail.com>.
Hi,
I can see 150gb ram and 36 cpu in our cluster.
On Tue, Apr 18, 2017 at 12:48 PM, Alberto Ramón <a....@gmail.com>
wrote:
> Can you check how many ram and cpu is asigned to yarn?
>
> El 18 abr. 2017 7:40 a. m., "suresh m" <su...@gmail.com> escribió:
>
> > Hi This is the time consuming step in entire build process.
> >
> > #24 Step Name: Build Cube
> >
> > Please help me to get it resolved.
> >
> > Regards,
> > Suresh
> >
> > On Mon, Apr 17, 2017 at 5:09 PM, 康凯森 <ka...@qq.com> wrote:
> >
> > > what's the slowest step in your cube job?
> > > please refer to http://kylin.apache.org/docs20/howto/howto_optimize_
> > > build.html
> > >
> > >
> > >
> > >
> > > ------------------ 原始邮件 ------------------
> > > 发件人: "suresh m";<su...@gmail.com>;
> > > 发送时间: 2017年4月17日(星期一) 晚上7:05
> > > 收件人: "dev"<de...@kylin.apache.org>;
> > >
> > > 主题: cube building taking 8 hours just for 6 thousands records
> > >
> > >
> > >
> > > Hi All,
> > >
> > > One of my cube taking hours to build even though its having less
> > > records(6000), please provide me some tuning techniques to improve
> > > performance in Kylin.
> > >
> > > Regards.
> > > Suresh
> > >
> >
>
Re: cube building taking 8 hours just for 6 thousands records
Posted by Alberto Ramón <a....@gmail.com>.
Can you check how many ram and cpu is asigned to yarn?
El 18 abr. 2017 7:40 a. m., "suresh m" <su...@gmail.com> escribió:
> Hi This is the time consuming step in entire build process.
>
> #24 Step Name: Build Cube
>
> Please help me to get it resolved.
>
> Regards,
> Suresh
>
> On Mon, Apr 17, 2017 at 5:09 PM, 康凯森 <ka...@qq.com> wrote:
>
> > what's the slowest step in your cube job?
> > please refer to http://kylin.apache.org/docs20/howto/howto_optimize_
> > build.html
> >
> >
> >
> >
> > ------------------ 原始邮件 ------------------
> > 发件人: "suresh m";<su...@gmail.com>;
> > 发送时间: 2017年4月17日(星期一) 晚上7:05
> > 收件人: "dev"<de...@kylin.apache.org>;
> >
> > 主题: cube building taking 8 hours just for 6 thousands records
> >
> >
> >
> > Hi All,
> >
> > One of my cube taking hours to build even though its having less
> > records(6000), please provide me some tuning techniques to improve
> > performance in Kylin.
> >
> > Regards.
> > Suresh
> >
>
Re: cube building taking 8 hours just for 6 thousands records
Posted by suresh m <su...@gmail.com>.
Hi This is the time consuming step in entire build process.
#24 Step Name: Build Cube
Please help me to get it resolved.
Regards,
Suresh
On Mon, Apr 17, 2017 at 5:09 PM, 康凯森 <ka...@qq.com> wrote:
> what's the slowest step in your cube job?
> please refer to http://kylin.apache.org/docs20/howto/howto_optimize_
> build.html
>
>
>
>
> ------------------ 原始邮件 ------------------
> 发件人: "suresh m";<su...@gmail.com>;
> 发送时间: 2017年4月17日(星期一) 晚上7:05
> 收件人: "dev"<de...@kylin.apache.org>;
>
> 主题: cube building taking 8 hours just for 6 thousands records
>
>
>
> Hi All,
>
> One of my cube taking hours to build even though its having less
> records(6000), please provide me some tuning techniques to improve
> performance in Kylin.
>
> Regards.
> Suresh
>