You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@kylin.apache.org by "Bryan Liu (CN)" <Br...@homecredit.cn> on 2019/10/06 22:43:36 UTC
Re: [!!Mass Mail][Probable spam]Re: sometimes need quite a long time
when building cube
Hi Shaofeng
It was in map phase. Thank you
Bryan
在 2019年10月6日，22:03，ShaoFeng Shi <sh...@apache.org>> 写道：
Hi Bryan,
What's the phase of the job in the second screenshot? map phase or reduce phase?
Best regards,
Shaofeng Shi 史少锋
Apache Kylin PMC
Email: shaofengshi@apache.org<ma...@apache.org>
Apache Kylin FAQ: https://kylin.apache.org/docs/gettingstarted/faq.html
Join Kylin user mail group: user-subscribe@kylin.apache.org<ma...@kylin.apache.org>
Join Kylin dev mail group: dev-subscribe@kylin.apache.org<ma...@kylin.apache.org>
Bryan Liu (CN) <Br...@homecredit.cn>> 于2019年9月26日周四 下午3:37写道：
Dears,
I am doing some testing with Kylin now. My Cube based on one source table with about 60~70M rows of data for one month.
Normally we build cube need about 25mins .
But sometimes which need more than 3hours , usually in busy period. When I am checking the MapReduce Jobs for cube building step 3(Extract Fact Table Distinct Columns) , I found some Jobs just take several Seconds. But some Jobs take quit a long time.
Please refer to screenshot as bellow.
I think Hadoop do not have enough resource is one reason. Meanwhile, there should have some problem with Cube building step 2. Seems the data is non-equilibrium.
Could you please give me some advice ? thank you so much .
<image002.jpg>
<image006.jpg>
Re: [!!Mass Mail][Probable spam]Re: sometimes need quite a long time
when building cube
Posted by ShaoFeng Shi <sh...@apache.org>.
Please check the file size of the intermediate hive table first; The file
size should be even after the "Redistribute" step. If not, please check the
columns that it redistributed by (the first three dimensions by default).
Best regards,
Shaofeng Shi 史少锋
Apache Kylin PMC
Email: shaofengshi@apache.org
Apache Kylin FAQ: https://kylin.apache.org/docs/gettingstarted/faq.html
Join Kylin user mail group: user-subscribe@kylin.apache.org
Join Kylin dev mail group: dev-subscribe@kylin.apache.org
Bryan Liu (CN) <Br...@homecredit.cn> 于2019年10月7日周一 上午6:52写道：
> Hi Shaofeng
> It was in map phase. Thank you
>
> Bryan
>
>
> 在 2019年10月6日，22:03，ShaoFeng Shi <sh...@apache.org> 写道：
>
> Hi Bryan,
>
> What's the phase of the job in the second screenshot? map phase or reduce
> phase?
>
> Best regards,
>
> Shaofeng Shi 史少锋
> Apache Kylin PMC
> Email: shaofengshi@apache.org
>
> Apache Kylin FAQ: https://kylin.apache.org/docs/gettingstarted/faq.html
> Join Kylin user mail group: user-subscribe@kylin.apache.org
> Join Kylin dev mail group: dev-subscribe@kylin.apache.org
>
>
>
>
> Bryan Liu (CN) <Br...@homecredit.cn> 于2019年9月26日周四 下午3:37写道：
>
>> Dears,
>>
>>
>>
>> I am doing some testing with Kylin now. My Cube based on one source
>> table with about 60~70M rows of data for one month.
>>
>> Normally we build cube need about 25mins .
>>
>> But sometimes which need more than 3hours , usually in busy period.
>> When I am checking the MapReduce Jobs for cube building step 3(Extract Fact
>> Table Distinct Columns) , I found some Jobs just take several Seconds. But
>> some Jobs take quit a long time.
>>
>>
>>
>> Please refer to screenshot as bellow.
>>
>> I think Hadoop do not have enough resource is one reason. Meanwhile,
>> there should have some problem with Cube building step 2. Seems the data
>> is non-equilibrium.
>>
>> Could you please give me some advice ? thank you so much .
>>
>> <image002.jpg>
>>
>> <image006.jpg>
>>
>>