You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@kylin.apache.org by "Kumar, Manoj H" <ma...@jpmorgan.com> on 2018/02/01 07:39:39 UTC
RE: Optimize Cube Build process
We have 15 nodes & each nodes have 8 cores. RAM= 256 MB.
I don’t think memory is issue here.
Regards,
Manoj
From: Alberto Ramón [mailto:a.ramonportoles@gmail.com]
Sent: Thursday, February 01, 2018 1:33 AM
To: user <us...@kylin.apache.org>
Subject: Re: Optimize Cube Build process
How many nodes do you have?
how many RAM and CPU do you have per node?
On 31 January 2018 at 05:07, Kumar, Manoj H <ma...@jpmorgan.com>> wrote:
It has close to 68 mapper & reducers 500.. It keeps running on this. Pls. advise.
[cid:image001.png@01D39B5D.E7ED9FD0]
Regards,
Manoj
From: Kumar, Manoj H
Sent: Wednesday, January 31, 2018 9:24 AM
To: 'user@kylin.apache.org<ma...@kylin.apache.org>' <us...@kylin.apache.org>>
Subject: Optimize Cube Build process
Hi Folks – I have close to 33 million of fact data to be processed, Data is having lot of unique/Distinct values such Loan_unique_code, Facility_code,card_id such.. Dimension looks up are made of these.
Fact table – 33 millions
Looks up tables having to 3 to 4 millions
Cube build type I have chosen – inmem
Engine – Mapreduce
Cube build step is taking 90 minutes which is seems to be high. What I can do in order to minimize build time? What Parameter I should tweak so that Build time gets reduced. Thanks.
I have Followed the same steps as given below but it doesn’t help in this case
http://kylin.apache.org/docs21/howto/howto_optimize_build.html<https://secureweb.jpmchase.net/readonly/http:/kylin.apache.org/docs21/howto/howto_optimize_build.html>
Regards,
Manoj
This message is confidential and subject to terms at: http://www.jpmorgan.com/emaildisclaimer<http://www.jpmorgan.com/emaildisclaimer> including on confidentiality, legal privilege, viruses and monitoring of electronic messages. If you are not the intended recipient, please delete this message and notify the sender immediately. Any unauthorized use is strictly prohibited.
This message is confidential and subject to terms at: http://www.jpmorgan.com/emaildisclaimer including on confidentiality, legal privilege, viruses and monitoring of electronic messages. If you are not the intended recipient, please delete this message and notify the sender immediately. Any unauthorized use is strictly prohibited.
RE: Optimize Cube Build process
Posted by "Kumar, Manoj H" <ma...@jpmorgan.com>.
For each CPU having 256 MB..
Regards,
Manoj
From: ShaoFeng Shi [mailto:shaofengshi@apache.org]
Sent: Thursday, February 01, 2018 2:21 PM
To: user <us...@kylin.apache.org>
Subject: Re: Optimize Cube Build process
Hi Manoj,
"RAM= 256 MB", is this true?
2018-02-01 16:17 GMT+08:00 Alberto Ramón <a....@gmail.com>>:
How many process are you runing in parallel? In build cube step
On 1 Feb 2018 7:39 a.m., "Kumar, Manoj H" <ma...@jpmorgan.com>> wrote:
We have 15 nodes & each nodes have 8 cores. RAM= 256 MB.
I don’t think memory is issue here.
Regards,
Manoj
From: Alberto Ramón [mailtoa.ramonportoles@gmail.com<ma...@gmail.com>]
Sent: Thursday, February 01, 2018 1:33 AM
To: user <us...@kylin.apache.org>>
Subject: Re: Optimize Cube Build process
How many nodes do you have?
how many RAM and CPU do you have per node?
On 31 January 2018 at 05:07, Kumar, Manoj H <ma...@jpmorgan.com>> wrote:
It has close to 68 mapper & reducers 500.. It keeps running on this. Pls. advise.
Regards,
Manoj
From: Kumar, Manoj H
Sent: Wednesday, January 31, 2018 9:24 AM
To: 'user@kylin.apache.org<ma...@kylin.apache.org>' <us...@kylin.apache.org>>
Subject: Optimize Cube Build process
Hi Folks – I have close to 33 million of fact data to be processed, Data is having lot of unique/Distinct values such Loan_unique_code, Facility_code,card_id such.. Dimension looks up are made of these.
Fact table – 33 millions
Looks up tables having to 3 to 4 millions
Cube build type I have chosen – inmem
Engine – Mapreduce
Cube build step is taking 90 minutes which is seems to be high. What I can do in order to minimize build time? What Parameter I should tweak so that Build time gets reduced. Thanks.
I have Followed the same steps as given below but it doesn’t help in this case
http://kylin.apache.org/docs21/howto/howto_optimize_build.html<https://secureweb.jpmchase.net/readonly/http:/kylin.apache.org/docs21/howto/howto_optimize_build.html>
Regards,
Manoj
This message is confidential and subject to terms at: http://www.jpmorgan.com/emaildisclaimer<http://www.jpmorgan.com/emaildisclaimer> including on confidentiality, legal privilege, viruses and monitoring of electronic messages. If you are not the intended recipient, please delete this message and notify the sender immediately. Any unauthorized use is strictly prohibited.
This message is confidential and subject to terms at: http://www.jpmorgan.com/emaildisclaimer<http://www.jpmorgan.com/emaildisclaimer> including on confidentiality, legal privilege, viruses and monitoring of electronic messages. If you are not the intended recipient, please delete this message and notify the sender immediately. Any unauthorized use is strictly prohibited.
--
Best regards,
Shaofeng Shi 史少锋
This message is confidential and subject to terms at: http://www.jpmorgan.com/emaildisclaimer including on confidentiality, legal privilege, viruses and monitoring of electronic messages. If you are not the intended recipient, please delete this message and notify the sender immediately. Any unauthorized use is strictly prohibited.
Re: Optimize Cube Build process
Posted by ShaoFeng Shi <sh...@apache.org>.
Hi Manoj,
"RAM= 256 MB", is this true?
2018-02-01 16:17 GMT+08:00 Alberto Ramón <a....@gmail.com>:
> How many process are you runing in parallel? In build cube step
>
> On 1 Feb 2018 7:39 a.m., "Kumar, Manoj H" <ma...@jpmorgan.com>
> wrote:
>
>> We have 15 nodes & each nodes have 8 cores. RAM= 256 MB.
>>
>>
>>
>> I don’t think memory is issue here.
>>
>>
>>
>> Regards,
>>
>> Manoj
>>
>>
>>
>> *From:* Alberto Ramón [mailtoa.ramonportoles@gmail.com]
>> *Sent:* Thursday, February 01, 2018 1:33 AM
>> *To:* user <us...@kylin.apache.org>
>> *Subject:* Re: Optimize Cube Build process
>>
>>
>>
>> How many nodes do you have?
>>
>> how many RAM and CPU do you have per node?
>>
>>
>>
>> On 31 January 2018 at 05:07, Kumar, Manoj H <ma...@jpmorgan.com>
>> wrote:
>>
>> It has close to 68 mapper & reducers 500.. It keeps running on this. Pls.
>> advise.
>>
>> [image: cid:image001.png@01D39B5D.E7ED9FD0]
>>
>>
>>
>> Regards,
>>
>> Manoj
>>
>>
>>
>> *From:* Kumar, Manoj H
>> *Sent:* Wednesday, January 31, 2018 9:24 AM
>> *To:* 'user@kylin.apache.org' <us...@kylin.apache.org>
>> *Subject:* Optimize Cube Build process
>>
>>
>>
>> Hi Folks – I have close to 33 million of fact data to be processed, Data
>> is having lot of unique/Distinct values such Loan_unique_code,
>> Facility_code,card_id such.. Dimension looks up are made of these.
>>
>>
>>
>> Fact table – 33 millions
>>
>> Looks up tables having to 3 to 4 millions
>>
>> Cube build type I have chosen – inmem
>>
>> Engine – Mapreduce
>>
>>
>>
>> Cube build step is taking 90 minutes which is seems to be high. What I
>> can do in order to minimize build time? What Parameter I should tweak so
>> that Build time gets reduced. Thanks.
>>
>>
>>
>>
>>
>> I have Followed the same steps as given below but it doesn’t help in this
>> case
>>
>>
>>
>> http://kylin.apache.org/docs21/howto/howto_optimize_build.html
>>
>>
>>
>>
>>
>> Regards,
>>
>> Manoj
>>
>>
>>
>> This message is confidential and subject to terms at: http://
>> www.jpmorgan.com/emaildisclaimer including on confidentiality, legal
>> privilege, viruses and monitoring of electronic messages. If you are not
>> the intended recipient, please delete this message and notify the sender
>> immediately. Any unauthorized use is strictly prohibited.
>>
>>
>>
>> This message is confidential and subject to terms at: http://
>> www.jpmorgan.com/emaildisclaimer including on confidentiality, legal
>> privilege, viruses and monitoring of electronic messages. If you are not
>> the intended recipient, please delete this message and notify the sender
>> immediately. Any unauthorized use is strictly prohibited.
>>
>
--
Best regards,
Shaofeng Shi 史少锋
RE: Optimize Cube Build process
Posted by Alberto Ramón <a....@gmail.com>.
How many process are you runing in parallel? In build cube step
On 1 Feb 2018 7:39 a.m., "Kumar, Manoj H" <ma...@jpmorgan.com>
wrote:
> We have 15 nodes & each nodes have 8 cores. RAM= 256 MB.
>
>
>
> I don’t think memory is issue here.
>
>
>
> Regards,
>
> Manoj
>
>
>
> *From:* Alberto Ramón [mailtoa.ramonportoles@gmail.com]
> *Sent:* Thursday, February 01, 2018 1:33 AM
> *To:* user <us...@kylin.apache.org>
> *Subject:* Re: Optimize Cube Build process
>
>
>
> How many nodes do you have?
>
> how many RAM and CPU do you have per node?
>
>
>
> On 31 January 2018 at 05:07, Kumar, Manoj H <ma...@jpmorgan.com>
> wrote:
>
> It has close to 68 mapper & reducers 500.. It keeps running on this. Pls.
> advise.
>
> [image: cid:image001.png@01D39B5D.E7ED9FD0]
>
>
>
> Regards,
>
> Manoj
>
>
>
> *From:* Kumar, Manoj H
> *Sent:* Wednesday, January 31, 2018 9:24 AM
> *To:* 'user@kylin.apache.org' <us...@kylin.apache.org>
> *Subject:* Optimize Cube Build process
>
>
>
> Hi Folks – I have close to 33 million of fact data to be processed, Data
> is having lot of unique/Distinct values such Loan_unique_code,
> Facility_code,card_id such.. Dimension looks up are made of these.
>
>
>
> Fact table – 33 millions
>
> Looks up tables having to 3 to 4 millions
>
> Cube build type I have chosen – inmem
>
> Engine – Mapreduce
>
>
>
> Cube build step is taking 90 minutes which is seems to be high. What I can
> do in order to minimize build time? What Parameter I should tweak so that
> Build time gets reduced. Thanks.
>
>
>
>
>
> I have Followed the same steps as given below but it doesn’t help in this
> case
>
>
>
> http://kylin.apache.org/docs21/howto/howto_optimize_build.html
>
>
>
>
>
> Regards,
>
> Manoj
>
>
>
> This message is confidential and subject to terms at: http://
> www.jpmorgan.com/emaildisclaimer including on confidentiality, legal
> privilege, viruses and monitoring of electronic messages. If you are not
> the intended recipient, please delete this message and notify the sender
> immediately. Any unauthorized use is strictly prohibited.
>
>
>
> This message is confidential and subject to terms at: http://
> www.jpmorgan.com/emaildisclaimer including on confidentiality, legal
> privilege, viruses and monitoring of electronic messages. If you are not
> the intended recipient, please delete this message and notify the sender
> immediately. Any unauthorized use is strictly prohibited.
>