You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@kylin.apache.org by Sonny Heer <so...@gmail.com> on 2018/03/14 14:58:47 UTC
#3 Step Name: Extract Fact Table Distinct Columns (slow)
Step 3 isn't using our full cluster. How can i increase the
mappers/reducers to use all the slots? Any config to look at in kylin?
Thanks
Re: #3 Step Name: Extract Fact Table Distinct Columns (slow)
Posted by Sonny Heer <so...@gmail.com>.
Maybe this is fixed in 2.0.0? See
https://issues.apache.org/jira/browse/KYLIN-2135
Is this the same issue? It should split dims to higher reducer it seems...
Although we are on 1.6. If this will fix our issue - can i apply this
patch? I may look at what is all there vs 1.6. Any ideas?
On Wed, Mar 14, 2018 at 1:44 PM, Alberto Ramón <a....@gmail.com>
wrote:
> You can monitoring your yarn in step 3
> In any case, step 3 is a sample of Fat table to estimate number of keys
> for each dim
> If this step takes a lot of time, you will need review your cube design
>
> Alb
>
> On 14 March 2018 at 16:54, Sonny Heer <so...@gmail.com> wrote:
>
>> 8 YARN nodes with 11 slots each. each slot is configured to ~2gb. Step
>> #3 in Kylin is launching 19 mappers and 5 reducers. 5 reducers when there
>> are 88 slots.
>>
>> btw: kylin version is 1.6
>>
>> On Wed, Mar 14, 2018 at 9:48 AM, Sonny Heer <so...@gmail.com> wrote:
>>
>>> YARN is properly configured. we use many other m/r and spark programs
>>> that utilize the full slots. It's only when building cubes.
>>>
>>> On Wed, Mar 14, 2018 at 9:46 AM, Alberto Ramón <
>>> a.ramonportoles@gmail.com> wrote:
>>>
>>>> You need check your yarn configuration first
>>>>
>>>> On Wed, 14 Mar 2018, 14:58 Sonny Heer, <so...@gmail.com> wrote:
>>>>
>>>>> Step 3 isn't using our full cluster. How can i increase the
>>>>> mappers/reducers to use all the slots? Any config to look at in kylin?
>>>>>
>>>>> Thanks
>>>>>
>>>>
>>>
>>
>
Re: #3 Step Name: Extract Fact Table Distinct Columns (slow)
Posted by Alberto Ramón <a....@gmail.com>.
You can monitoring your yarn in step 3
In any case, step 3 is a sample of Fat table to estimate number of keys for
each dim
If this step takes a lot of time, you will need review your cube design
Alb
On 14 March 2018 at 16:54, Sonny Heer <so...@gmail.com> wrote:
> 8 YARN nodes with 11 slots each. each slot is configured to ~2gb. Step
> #3 in Kylin is launching 19 mappers and 5 reducers. 5 reducers when there
> are 88 slots.
>
> btw: kylin version is 1.6
>
> On Wed, Mar 14, 2018 at 9:48 AM, Sonny Heer <so...@gmail.com> wrote:
>
>> YARN is properly configured. we use many other m/r and spark programs
>> that utilize the full slots. It's only when building cubes.
>>
>> On Wed, Mar 14, 2018 at 9:46 AM, Alberto Ramón <a.ramonportoles@gmail.com
>> > wrote:
>>
>>> You need check your yarn configuration first
>>>
>>> On Wed, 14 Mar 2018, 14:58 Sonny Heer, <so...@gmail.com> wrote:
>>>
>>>> Step 3 isn't using our full cluster. How can i increase the
>>>> mappers/reducers to use all the slots? Any config to look at in kylin?
>>>>
>>>> Thanks
>>>>
>>>
>>
>
Re: #3 Step Name: Extract Fact Table Distinct Columns (slow)
Posted by Sonny Heer <so...@gmail.com>.
8 YARN nodes with 11 slots each. each slot is configured to ~2gb. Step #3
in Kylin is launching 19 mappers and 5 reducers. 5 reducers when there are
88 slots.
btw: kylin version is 1.6
On Wed, Mar 14, 2018 at 9:48 AM, Sonny Heer <so...@gmail.com> wrote:
> YARN is properly configured. we use many other m/r and spark programs
> that utilize the full slots. It's only when building cubes.
>
> On Wed, Mar 14, 2018 at 9:46 AM, Alberto Ramón <a....@gmail.com>
> wrote:
>
>> You need check your yarn configuration first
>>
>> On Wed, 14 Mar 2018, 14:58 Sonny Heer, <so...@gmail.com> wrote:
>>
>>> Step 3 isn't using our full cluster. How can i increase the
>>> mappers/reducers to use all the slots? Any config to look at in kylin?
>>>
>>> Thanks
>>>
>>
>
Re: #3 Step Name: Extract Fact Table Distinct Columns (slow)
Posted by Sonny Heer <so...@gmail.com>.
YARN is properly configured. we use many other m/r and spark programs that
utilize the full slots. It's only when building cubes.
On Wed, Mar 14, 2018 at 9:46 AM, Alberto Ramón <a....@gmail.com>
wrote:
> You need check your yarn configuration first
>
> On Wed, 14 Mar 2018, 14:58 Sonny Heer, <so...@gmail.com> wrote:
>
>> Step 3 isn't using our full cluster. How can i increase the
>> mappers/reducers to use all the slots? Any config to look at in kylin?
>>
>> Thanks
>>
>
Re: #3 Step Name: Extract Fact Table Distinct Columns (slow)
Posted by Alberto Ramón <a....@gmail.com>.
You need check your yarn configuration first
On Wed, 14 Mar 2018, 14:58 Sonny Heer, <so...@gmail.com> wrote:
> Step 3 isn't using our full cluster. How can i increase the
> mappers/reducers to use all the slots? Any config to look at in kylin?
>
> Thanks
>