You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@kylin.apache.org by Nirav Patel <np...@xactlycorp.com> on 2017/05/11 18:43:56 UTC
How to apply historical Updates to existing cube data
Hi,
Correct me if I am wrong but currently you can not update existing kylin
cube without refreshing entire cube. Does it mean if I am pulling new data
from hive based on lets say customerId, Timestamp for which I already built
cube before I have to rebuild entire cube from scratch? Or can I say
refresh between startTime and endTime which will update cube data for that
timeframe only.
Also Hive data can be partitioned by any keys(columns) not just timestamp.
so why not allow kylin cube updates based on any arbitrary partition
strategy that user have defined on their hive table?
e.g. update part of the cube based on timestamp, customerid, batchid etc.
Thanks,
Nirav
--
[image: What's New with Xactly] <http://www.xactlycorp.com/email-click/>
<https://www.nyse.com/quote/XNYS:XTLY> [image: LinkedIn]
<https://www.linkedin.com/company/xactly-corporation> [image: Twitter]
<https://twitter.com/Xactly> [image: Facebook]
<https://www.facebook.com/XactlyCorp> [image: YouTube]
<http://www.youtube.com/xactlycorporation>
Re: How to apply historical Updates to existing cube data
Posted by Li Yang <li...@apache.org>.
Refreshing a segment of time range in the past is the way to pick up
historic data changes. We don't see this as a common use case though.
History data should not change in most cases.
A new HTable is created to hold the new segment, and the old segment and
its HTable become garbage to be collected.
http://kylin.apache.org/docs20/howto/howto_cleanup_storage.html
Cheers
Yang
On Fri, May 12, 2017 at 6:31 AM, Nirav Patel <np...@xactlycorp.com> wrote:
> First link says you can do incremental based on "range of segments". Is it
> a timestamp/date range that we define during cube creation?
> ANd since data is stored in hbase will kylin just overwrite new data with
> new one with same rowkeys?
>
> Thanks
>
> On Thu, May 11, 2017 at 1:26 PM, Alberto Ramón <a....@gmail.com>
> wrote:
>
>> Q1- Check this previous mailList about late data:
>> http://apache-kylin.74782.x6.nabble.com/Reloading-data-td5669.html
>>
>> You only will need recalculate segments involved
>>
>> Q2- Check Shardin (https://issues.apache.org/jira/browse/KYLIN-1453)
>> Partition by time column is not reoomended (It Will create hotspot in
>> HBase)
>>
>>
>>
>> On 11 May 2017 at 19:43, Nirav Patel <np...@xactlycorp.com> wrote:
>>
>>> Hi,
>>>
>>> Correct me if I am wrong but currently you can not update existing kylin
>>> cube without refreshing entire cube. Does it mean if I am pulling new data
>>> from hive based on lets say customerId, Timestamp for which I already built
>>> cube before I have to rebuild entire cube from scratch? Or can I say
>>> refresh between startTime and endTime which will update cube data for that
>>> timeframe only.
>>>
>>> Also Hive data can be partitioned by any keys(columns) not just
>>> timestamp. so why not allow kylin cube updates based on any arbitrary
>>> partition strategy that user have defined on their hive table?
>>> e.g. update part of the cube based on timestamp, customerid, batchid etc.
>>>
>>> Thanks,
>>> Nirav
>>>
>>>
>>>
>>> [image: What's New with Xactly] <http://www.xactlycorp.com/email-click/>
>>>
>>> <https://www.nyse.com/quote/XNYS:XTLY> [image: LinkedIn]
>>> <https://www.linkedin.com/company/xactly-corporation> [image: Twitter]
>>> <https://twitter.com/Xactly> [image: Facebook]
>>> <https://www.facebook.com/XactlyCorp> [image: YouTube]
>>> <http://www.youtube.com/xactlycorporation>
>>
>>
>>
>
>
>
> [image: What's New with Xactly] <http://www.xactlycorp.com/email-click/>
>
> <https://www.nyse.com/quote/XNYS:XTLY> [image: LinkedIn]
> <https://www.linkedin.com/company/xactly-corporation> [image: Twitter]
> <https://twitter.com/Xactly> [image: Facebook]
> <https://www.facebook.com/XactlyCorp> [image: YouTube]
> <http://www.youtube.com/xactlycorporation>
>
Re: How to apply historical Updates to existing cube data
Posted by Nirav Patel <np...@xactlycorp.com>.
First link says you can do incremental based on "range of segments". Is it
a timestamp/date range that we define during cube creation?
ANd since data is stored in hbase will kylin just overwrite new data with
new one with same rowkeys?
Thanks
On Thu, May 11, 2017 at 1:26 PM, Alberto Ramón <a....@gmail.com>
wrote:
> Q1- Check this previous mailList about late data:
> http://apache-kylin.74782.x6.nabble.com/Reloading-data-td5669.html
>
> You only will need recalculate segments involved
>
> Q2- Check Shardin (https://issues.apache.org/jira/browse/KYLIN-1453)
> Partition by time column is not reoomended (It Will create hotspot in
> HBase)
>
>
>
> On 11 May 2017 at 19:43, Nirav Patel <np...@xactlycorp.com> wrote:
>
>> Hi,
>>
>> Correct me if I am wrong but currently you can not update existing kylin
>> cube without refreshing entire cube. Does it mean if I am pulling new data
>> from hive based on lets say customerId, Timestamp for which I already built
>> cube before I have to rebuild entire cube from scratch? Or can I say
>> refresh between startTime and endTime which will update cube data for that
>> timeframe only.
>>
>> Also Hive data can be partitioned by any keys(columns) not just
>> timestamp. so why not allow kylin cube updates based on any arbitrary
>> partition strategy that user have defined on their hive table?
>> e.g. update part of the cube based on timestamp, customerid, batchid etc.
>>
>> Thanks,
>> Nirav
>>
>>
>>
>> [image: What's New with Xactly] <http://www.xactlycorp.com/email-click/>
>>
>> <https://www.nyse.com/quote/XNYS:XTLY> [image: LinkedIn]
>> <https://www.linkedin.com/company/xactly-corporation> [image: Twitter]
>> <https://twitter.com/Xactly> [image: Facebook]
>> <https://www.facebook.com/XactlyCorp> [image: YouTube]
>> <http://www.youtube.com/xactlycorporation>
>
>
>
--
[image: What's New with Xactly] <http://www.xactlycorp.com/email-click/>
<https://www.nyse.com/quote/XNYS:XTLY> [image: LinkedIn]
<https://www.linkedin.com/company/xactly-corporation> [image: Twitter]
<https://twitter.com/Xactly> [image: Facebook]
<https://www.facebook.com/XactlyCorp> [image: YouTube]
<http://www.youtube.com/xactlycorporation>
Re: How to apply historical Updates to existing cube data
Posted by Alberto Ramón <a....@gmail.com>.
Q1- Check this previous mailList about late data:
http://apache-kylin.74782.x6.nabble.com/Reloading-data-td5669.html
You only will need recalculate segments involved
Q2- Check Shardin (https://issues.apache.org/jira/browse/KYLIN-1453)
Partition by time column is not reoomended (It Will create hotspot in
HBase)
On 11 May 2017 at 19:43, Nirav Patel <np...@xactlycorp.com> wrote:
> Hi,
>
> Correct me if I am wrong but currently you can not update existing kylin
> cube without refreshing entire cube. Does it mean if I am pulling new data
> from hive based on lets say customerId, Timestamp for which I already built
> cube before I have to rebuild entire cube from scratch? Or can I say
> refresh between startTime and endTime which will update cube data for that
> timeframe only.
>
> Also Hive data can be partitioned by any keys(columns) not just timestamp.
> so why not allow kylin cube updates based on any arbitrary partition
> strategy that user have defined on their hive table?
> e.g. update part of the cube based on timestamp, customerid, batchid etc.
>
> Thanks,
> Nirav
>
>
>
> [image: What's New with Xactly] <http://www.xactlycorp.com/email-click/>
>
> <https://www.nyse.com/quote/XNYS:XTLY> [image: LinkedIn]
> <https://www.linkedin.com/company/xactly-corporation> [image: Twitter]
> <https://twitter.com/Xactly> [image: Facebook]
> <https://www.facebook.com/XactlyCorp> [image: YouTube]
> <http://www.youtube.com/xactlycorporation>