You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@kylin.apache.org by ShaoFeng Shi <sh...@apache.org> on 2018/10/02 03:22:11 UTC
Re: Create Views
Hi Hari,
Sorry for the late response. Does it like a lambda architecture? How can
add a view to supporting aggregation pushdown? I'm not clear on this,
please elaborate. Thank you!
hosur narahari <hn...@gmail.com> 于2018年9月30日周日 下午1:50写道：
> Hi ShaoFeng,
>
> Is it possible to provide that support. Because, in many cases cubes are
> used with latest transaction data to get up to date analysis and some kind
> of query engine will be used for merging cube with non-cube data. And most
> of the query engines don't have aggregate pushdown, which makes it unable
> to query from cube. If we provide view, we can solve this problem for all
> query engines irrespective of whether they provide aggregate pushdown or
> not, making kylin more adaptable.
>
> Also it's just a conceptual view, not adding any overhead.
>
> Best Regards,
> Hari
>
> On Fri, Sep 28, 2018 at 7:40 PM ShaoFeng Shi <sh...@apache.org>
> wrote:
>
>> Hi Hari,
>>
>> Creating view on top of kylin tables is not supported in Kylin. Kylin is
>> read-only.
>>
>> hosur narahari <hn...@gmail.com> 于2018年9月28日周五 下午1:09写道：
>>
>>> Hi,
>>>
>>> Is it possible to create views on the lines of rdbms or hive on top of
>>> kylin tables.
>>>
>>> Best Regards,
>>> Hari
>>>
>>
>>
>> --
>> Best regards,
>>
>> Shaofeng Shi 史少锋
>>
>>
--
Best regards,
Shaofeng Shi 史少锋
Re: Create Views
Posted by hosur narahari <hn...@gmail.com>.
Hi ShaoFeng,
Sure I'll do.
Best Regards,
Hari
On Sat, Oct 6, 2018 at 8:26 PM ShaoFeng Shi <sh...@apache.org> wrote:
> Hi Hari,
>
> Would you like to open a JIRA to Kylin with this? Thanks!
>
> hosur narahari <hn...@gmail.com> 于2018年10月6日周六 下午9:25写道：
>
>> Hi ShaoFeng,
>>
>> Exactly. It will solve the problem for all query engines whatsoever.
>>
>> Best Regards,
>> Hari
>>
>> On Sat, Oct 6, 2018 at 6:51 PM ShaoFeng Shi <sh...@apache.org>
>> wrote:
>>
>>> Hi Hari,
>>>
>>> I see now; it is very similar to this integration with spark:
>>> https://kylin.apache.org/docs/tutorial/spark.html
>>>
>>> Spark can register a SQL as a temp table, and then use that temp table
>>> for subsequent analysis. But other engines may not have this function, so a
>>> common view in Kylin may help to fill that gap, am I correct?
>>>
>>>
>>>
>>> hosur narahari <hn...@gmail.com> 于2018年10月2日周二 下午12:48写道：
>>>
>>>> Hi ShaoFeng,
>>>>
>>>> It could be either like lambda architecture or just like merging cube
>>>> data with latest data for which cube has not yet been generated(in case of
>>>> periodic cube generation).
>>>>
>>>> Now coming to aggregate pushdown. I'll give a very simple scenario.
>>>> Consider below query.
>>>>
>>>> *Select sum(price) from kylin_sales*
>>>>
>>>> when we execute it in any query engine like presto, spark, drill etc.,
>>>> it doesn't execute above query but execute *Select price from
>>>> kylin_sales.* After getting all price values, it does *map-reduce* to
>>>> calculate sum. It's good for normal data sources but not for cube, since
>>>> this is already pre-calculated. Even if in future aggregate pushdown is
>>>> provided, it'll be very limited because we can't pushdown everything. For
>>>> example, what do we do when we've joins and then aggregate. And AFAIK,
>>>> unlike indexes(used for predicate pushdown), JDBC driver doesn't provide
>>>> any information on pre aggregated data.
>>>>
>>>> How does views, solve this problem?
>>>>
>>>> Let's say we create view on above query like *Create view sum_view as
>>>> Select sum(price) from kylin_sales;*
>>>>
>>>> In query engine we execute, *Select * from sum_view.* So kylin is
>>>> queried for sum_view, which is basically *Select sum(price) from
>>>> kylin_sales.* In this way, we can solve this problem using views.
>>>>
>>>> Please let me know, if I'm not being clear.
>>>>
>>>> Best Regards,
>>>> Hari
>>>>
>>>> On Tue, Oct 2, 2018 at 8:52 AM ShaoFeng Shi <sh...@apache.org>
>>>> wrote:
>>>>
>>>>> Hi Hari,
>>>>>
>>>>> Sorry for the late response. Does it like a lambda architecture? How
>>>>> can add a view to supporting aggregation pushdown? I'm not clear on this,
>>>>> please elaborate. Thank you!
>>>>>
>>>>> hosur narahari <hn...@gmail.com> 于2018年9月30日周日 下午1:50写道：
>>>>>
>>>>>> Hi ShaoFeng,
>>>>>>
>>>>>> Is it possible to provide that support. Because, in many cases cubes
>>>>>> are used with latest transaction data to get up to date analysis and some
>>>>>> kind of query engine will be used for merging cube with non-cube data. And
>>>>>> most of the query engines don't have aggregate pushdown, which makes it
>>>>>> unable to query from cube. If we provide view, we can solve this problem
>>>>>> for all query engines irrespective of whether they provide aggregate
>>>>>> pushdown or not, making kylin more adaptable.
>>>>>>
>>>>>> Also it's just a conceptual view, not adding any overhead.
>>>>>>
>>>>>> Best Regards,
>>>>>> Hari
>>>>>>
>>>>>> On Fri, Sep 28, 2018 at 7:40 PM ShaoFeng Shi <sh...@apache.org>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi Hari,
>>>>>>>
>>>>>>> Creating view on top of kylin tables is not supported in Kylin.
>>>>>>> Kylin is read-only.
>>>>>>>
>>>>>>> hosur narahari <hn...@gmail.com> 于2018年9月28日周五 下午1:09写道：
>>>>>>>
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> Is it possible to create views on the lines of rdbms or hive on top
>>>>>>>> of kylin tables.
>>>>>>>>
>>>>>>>> Best Regards,
>>>>>>>> Hari
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Best regards,
>>>>>>>
>>>>>>> Shaofeng Shi 史少锋
>>>>>>>
>>>>>>>
>>>>>
>>>>> --
>>>>> Best regards,
>>>>>
>>>>> Shaofeng Shi 史少锋
>>>>>
>>>>>
>>>
>>> --
>>> Best regards,
>>>
>>> Shaofeng Shi 史少锋
>>>
>>>
>
> --
> Best regards,
>
> Shaofeng Shi 史少锋
>
>
Re: Create Views
Posted by ShaoFeng Shi <sh...@apache.org>.
Hi Hari,
Would you like to open a JIRA to Kylin with this? Thanks!
hosur narahari <hn...@gmail.com> 于2018年10月6日周六 下午9:25写道：
> Hi ShaoFeng,
>
> Exactly. It will solve the problem for all query engines whatsoever.
>
> Best Regards,
> Hari
>
> On Sat, Oct 6, 2018 at 6:51 PM ShaoFeng Shi <sh...@apache.org>
> wrote:
>
>> Hi Hari,
>>
>> I see now; it is very similar to this integration with spark:
>> https://kylin.apache.org/docs/tutorial/spark.html
>>
>> Spark can register a SQL as a temp table, and then use that temp table
>> for subsequent analysis. But other engines may not have this function, so a
>> common view in Kylin may help to fill that gap, am I correct?
>>
>>
>>
>> hosur narahari <hn...@gmail.com> 于2018年10月2日周二 下午12:48写道：
>>
>>> Hi ShaoFeng,
>>>
>>> It could be either like lambda architecture or just like merging cube
>>> data with latest data for which cube has not yet been generated(in case of
>>> periodic cube generation).
>>>
>>> Now coming to aggregate pushdown. I'll give a very simple scenario.
>>> Consider below query.
>>>
>>> *Select sum(price) from kylin_sales*
>>>
>>> when we execute it in any query engine like presto, spark, drill etc.,
>>> it doesn't execute above query but execute *Select price from
>>> kylin_sales.* After getting all price values, it does *map-reduce* to
>>> calculate sum. It's good for normal data sources but not for cube, since
>>> this is already pre-calculated. Even if in future aggregate pushdown is
>>> provided, it'll be very limited because we can't pushdown everything. For
>>> example, what do we do when we've joins and then aggregate. And AFAIK,
>>> unlike indexes(used for predicate pushdown), JDBC driver doesn't provide
>>> any information on pre aggregated data.
>>>
>>> How does views, solve this problem?
>>>
>>> Let's say we create view on above query like *Create view sum_view as
>>> Select sum(price) from kylin_sales;*
>>>
>>> In query engine we execute, *Select * from sum_view.* So kylin is
>>> queried for sum_view, which is basically *Select sum(price) from
>>> kylin_sales.* In this way, we can solve this problem using views.
>>>
>>> Please let me know, if I'm not being clear.
>>>
>>> Best Regards,
>>> Hari
>>>
>>> On Tue, Oct 2, 2018 at 8:52 AM ShaoFeng Shi <sh...@apache.org>
>>> wrote:
>>>
>>>> Hi Hari,
>>>>
>>>> Sorry for the late response. Does it like a lambda architecture? How
>>>> can add a view to supporting aggregation pushdown? I'm not clear on this,
>>>> please elaborate. Thank you!
>>>>
>>>> hosur narahari <hn...@gmail.com> 于2018年9月30日周日 下午1:50写道：
>>>>
>>>>> Hi ShaoFeng,
>>>>>
>>>>> Is it possible to provide that support. Because, in many cases cubes
>>>>> are used with latest transaction data to get up to date analysis and some
>>>>> kind of query engine will be used for merging cube with non-cube data. And
>>>>> most of the query engines don't have aggregate pushdown, which makes it
>>>>> unable to query from cube. If we provide view, we can solve this problem
>>>>> for all query engines irrespective of whether they provide aggregate
>>>>> pushdown or not, making kylin more adaptable.
>>>>>
>>>>> Also it's just a conceptual view, not adding any overhead.
>>>>>
>>>>> Best Regards,
>>>>> Hari
>>>>>
>>>>> On Fri, Sep 28, 2018 at 7:40 PM ShaoFeng Shi <sh...@apache.org>
>>>>> wrote:
>>>>>
>>>>>> Hi Hari,
>>>>>>
>>>>>> Creating view on top of kylin tables is not supported in Kylin. Kylin
>>>>>> is read-only.
>>>>>>
>>>>>> hosur narahari <hn...@gmail.com> 于2018年9月28日周五 下午1:09写道：
>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> Is it possible to create views on the lines of rdbms or hive on top
>>>>>>> of kylin tables.
>>>>>>>
>>>>>>> Best Regards,
>>>>>>> Hari
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Best regards,
>>>>>>
>>>>>> Shaofeng Shi 史少锋
>>>>>>
>>>>>>
>>>>
>>>> --
>>>> Best regards,
>>>>
>>>> Shaofeng Shi 史少锋
>>>>
>>>>
>>
>> --
>> Best regards,
>>
>> Shaofeng Shi 史少锋
>>
>>
--
Best regards,
Shaofeng Shi 史少锋
Re: Create Views
Posted by hosur narahari <hn...@gmail.com>.
Hi ShaoFeng,
Exactly. It will solve the problem for all query engines whatsoever.
Best Regards,
Hari
On Sat, Oct 6, 2018 at 6:51 PM ShaoFeng Shi <sh...@apache.org> wrote:
> Hi Hari,
>
> I see now; it is very similar to this integration with spark:
> https://kylin.apache.org/docs/tutorial/spark.html
>
> Spark can register a SQL as a temp table, and then use that temp table for
> subsequent analysis. But other engines may not have this function, so a
> common view in Kylin may help to fill that gap, am I correct?
>
>
>
> hosur narahari <hn...@gmail.com> 于2018年10月2日周二 下午12:48写道：
>
>> Hi ShaoFeng,
>>
>> It could be either like lambda architecture or just like merging cube
>> data with latest data for which cube has not yet been generated(in case of
>> periodic cube generation).
>>
>> Now coming to aggregate pushdown. I'll give a very simple scenario.
>> Consider below query.
>>
>> *Select sum(price) from kylin_sales*
>>
>> when we execute it in any query engine like presto, spark, drill etc., it
>> doesn't execute above query but execute *Select price from kylin_sales.* After
>> getting all price values, it does *map-reduce* to calculate sum. It's
>> good for normal data sources but not for cube, since this is already
>> pre-calculated. Even if in future aggregate pushdown is provided, it'll be
>> very limited because we can't pushdown everything. For example, what do we
>> do when we've joins and then aggregate. And AFAIK, unlike indexes(used for
>> predicate pushdown), JDBC driver doesn't provide any information on pre
>> aggregated data.
>>
>> How does views, solve this problem?
>>
>> Let's say we create view on above query like *Create view sum_view as
>> Select sum(price) from kylin_sales;*
>>
>> In query engine we execute, *Select * from sum_view.* So kylin is
>> queried for sum_view, which is basically *Select sum(price) from
>> kylin_sales.* In this way, we can solve this problem using views.
>>
>> Please let me know, if I'm not being clear.
>>
>> Best Regards,
>> Hari
>>
>> On Tue, Oct 2, 2018 at 8:52 AM ShaoFeng Shi <sh...@apache.org>
>> wrote:
>>
>>> Hi Hari,
>>>
>>> Sorry for the late response. Does it like a lambda architecture? How can
>>> add a view to supporting aggregation pushdown? I'm not clear on this,
>>> please elaborate. Thank you!
>>>
>>> hosur narahari <hn...@gmail.com> 于2018年9月30日周日 下午1:50写道：
>>>
>>>> Hi ShaoFeng,
>>>>
>>>> Is it possible to provide that support. Because, in many cases cubes
>>>> are used with latest transaction data to get up to date analysis and some
>>>> kind of query engine will be used for merging cube with non-cube data. And
>>>> most of the query engines don't have aggregate pushdown, which makes it
>>>> unable to query from cube. If we provide view, we can solve this problem
>>>> for all query engines irrespective of whether they provide aggregate
>>>> pushdown or not, making kylin more adaptable.
>>>>
>>>> Also it's just a conceptual view, not adding any overhead.
>>>>
>>>> Best Regards,
>>>> Hari
>>>>
>>>> On Fri, Sep 28, 2018 at 7:40 PM ShaoFeng Shi <sh...@apache.org>
>>>> wrote:
>>>>
>>>>> Hi Hari,
>>>>>
>>>>> Creating view on top of kylin tables is not supported in Kylin. Kylin
>>>>> is read-only.
>>>>>
>>>>> hosur narahari <hn...@gmail.com> 于2018年9月28日周五 下午1:09写道：
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> Is it possible to create views on the lines of rdbms or hive on top
>>>>>> of kylin tables.
>>>>>>
>>>>>> Best Regards,
>>>>>> Hari
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Best regards,
>>>>>
>>>>> Shaofeng Shi 史少锋
>>>>>
>>>>>
>>>
>>> --
>>> Best regards,
>>>
>>> Shaofeng Shi 史少锋
>>>
>>>
>
> --
> Best regards,
>
> Shaofeng Shi 史少锋
>
>
Re: Create Views
Posted by ShaoFeng Shi <sh...@apache.org>.
Hi Hari,
I see now; it is very similar to this integration with spark:
https://kylin.apache.org/docs/tutorial/spark.html
Spark can register a SQL as a temp table, and then use that temp table for
subsequent analysis. But other engines may not have this function, so a
common view in Kylin may help to fill that gap, am I correct?
hosur narahari <hn...@gmail.com> 于2018年10月2日周二 下午12:48写道：
> Hi ShaoFeng,
>
> It could be either like lambda architecture or just like merging cube data
> with latest data for which cube has not yet been generated(in case of
> periodic cube generation).
>
> Now coming to aggregate pushdown. I'll give a very simple scenario.
> Consider below query.
>
> *Select sum(price) from kylin_sales*
>
> when we execute it in any query engine like presto, spark, drill etc., it
> doesn't execute above query but execute *Select price from kylin_sales.* After
> getting all price values, it does *map-reduce* to calculate sum. It's
> good for normal data sources but not for cube, since this is already
> pre-calculated. Even if in future aggregate pushdown is provided, it'll be
> very limited because we can't pushdown everything. For example, what do we
> do when we've joins and then aggregate. And AFAIK, unlike indexes(used for
> predicate pushdown), JDBC driver doesn't provide any information on pre
> aggregated data.
>
> How does views, solve this problem?
>
> Let's say we create view on above query like *Create view sum_view as
> Select sum(price) from kylin_sales;*
>
> In query engine we execute, *Select * from sum_view.* So kylin is queried
> for sum_view, which is basically *Select sum(price) from kylin_sales.* In
> this way, we can solve this problem using views.
>
> Please let me know, if I'm not being clear.
>
> Best Regards,
> Hari
>
> On Tue, Oct 2, 2018 at 8:52 AM ShaoFeng Shi <sh...@apache.org>
> wrote:
>
>> Hi Hari,
>>
>> Sorry for the late response. Does it like a lambda architecture? How can
>> add a view to supporting aggregation pushdown? I'm not clear on this,
>> please elaborate. Thank you!
>>
>> hosur narahari <hn...@gmail.com> 于2018年9月30日周日 下午1:50写道：
>>
>>> Hi ShaoFeng,
>>>
>>> Is it possible to provide that support. Because, in many cases cubes are
>>> used with latest transaction data to get up to date analysis and some kind
>>> of query engine will be used for merging cube with non-cube data. And most
>>> of the query engines don't have aggregate pushdown, which makes it unable
>>> to query from cube. If we provide view, we can solve this problem for all
>>> query engines irrespective of whether they provide aggregate pushdown or
>>> not, making kylin more adaptable.
>>>
>>> Also it's just a conceptual view, not adding any overhead.
>>>
>>> Best Regards,
>>> Hari
>>>
>>> On Fri, Sep 28, 2018 at 7:40 PM ShaoFeng Shi <sh...@apache.org>
>>> wrote:
>>>
>>>> Hi Hari,
>>>>
>>>> Creating view on top of kylin tables is not supported in Kylin. Kylin
>>>> is read-only.
>>>>
>>>> hosur narahari <hn...@gmail.com> 于2018年9月28日周五 下午1:09写道：
>>>>
>>>>> Hi,
>>>>>
>>>>> Is it possible to create views on the lines of rdbms or hive on top of
>>>>> kylin tables.
>>>>>
>>>>> Best Regards,
>>>>> Hari
>>>>>
>>>>
>>>>
>>>> --
>>>> Best regards,
>>>>
>>>> Shaofeng Shi 史少锋
>>>>
>>>>
>>
>> --
>> Best regards,
>>
>> Shaofeng Shi 史少锋
>>
>>
--
Best regards,
Shaofeng Shi 史少锋
Re: Create Views
Posted by hosur narahari <hn...@gmail.com>.
Hi ShaoFeng,
It could be either like lambda architecture or just like merging cube data
with latest data for which cube has not yet been generated(in case of
periodic cube generation).
Now coming to aggregate pushdown. I'll give a very simple scenario.
Consider below query.
*Select sum(price) from kylin_sales*
when we execute it in any query engine like presto, spark, drill etc., it
doesn't execute above query but execute *Select price from kylin_sales.* After
getting all price values, it does *map-reduce* to calculate sum. It's good
for normal data sources but not for cube, since this is already
pre-calculated. Even if in future aggregate pushdown is provided, it'll be
very limited because we can't pushdown everything. For example, what do we
do when we've joins and then aggregate. And AFAIK, unlike indexes(used for
predicate pushdown), JDBC driver doesn't provide any information on pre
aggregated data.
How does views, solve this problem?
Let's say we create view on above query like *Create view sum_view as
Select sum(price) from kylin_sales;*
In query engine we execute, *Select * from sum_view.* So kylin is queried
for sum_view, which is basically *Select sum(price) from kylin_sales.* In
this way, we can solve this problem using views.
Please let me know, if I'm not being clear.
Best Regards,
Hari
On Tue, Oct 2, 2018 at 8:52 AM ShaoFeng Shi <sh...@apache.org> wrote:
> Hi Hari,
>
> Sorry for the late response. Does it like a lambda architecture? How can
> add a view to supporting aggregation pushdown? I'm not clear on this,
> please elaborate. Thank you!
>
> hosur narahari <hn...@gmail.com> 于2018年9月30日周日 下午1:50写道：
>
>> Hi ShaoFeng,
>>
>> Is it possible to provide that support. Because, in many cases cubes are
>> used with latest transaction data to get up to date analysis and some kind
>> of query engine will be used for merging cube with non-cube data. And most
>> of the query engines don't have aggregate pushdown, which makes it unable
>> to query from cube. If we provide view, we can solve this problem for all
>> query engines irrespective of whether they provide aggregate pushdown or
>> not, making kylin more adaptable.
>>
>> Also it's just a conceptual view, not adding any overhead.
>>
>> Best Regards,
>> Hari
>>
>> On Fri, Sep 28, 2018 at 7:40 PM ShaoFeng Shi <sh...@apache.org>
>> wrote:
>>
>>> Hi Hari,
>>>
>>> Creating view on top of kylin tables is not supported in Kylin. Kylin is
>>> read-only.
>>>
>>> hosur narahari <hn...@gmail.com> 于2018年9月28日周五 下午1:09写道：
>>>
>>>> Hi,
>>>>
>>>> Is it possible to create views on the lines of rdbms or hive on top of
>>>> kylin tables.
>>>>
>>>> Best Regards,
>>>> Hari
>>>>
>>>
>>>
>>> --
>>> Best regards,
>>>
>>> Shaofeng Shi 史少锋
>>>
>>>
>
> --
> Best regards,
>
> Shaofeng Shi 史少锋
>
>