You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@kylin.apache.org by ShaoFeng Shi <sh...@apache.org> on 2017/03/01 09:27:53 UTC
Re: Which version of spark is kylin 2.0 Beta using ?
Kylin embed a spark, which is spark v1.6.3; I added a blog to briefly
introduce this, later will add a doc to instruct on how to config:
https://kylin.apache.org/blog/2017/02/23/by-layer-spark-cubing/
To customize the spark conf, please do that in kylin.properties with prefix
"kylin.engine.spark-conf."
2017-03-01 15:49 GMT+08:00 Mars Xu <xu...@gmail.com>:
> Hi ,
> I have install kylin 2.0 beta on my cluster which has already deploy a
> spark 2.1, but when I startup kylin server and build cube, the Web UI of
> Cube Build Step show it is spark 1.6.3.
>
> Is that kylin already include a spark version ? or some other else I
> need to configure ?
>
>
> Thanks
>
> Mars.
--
Best regards,
Shaofeng Shi 史少锋
Re: Which version of spark is kylin 2.0 Beta using ?
Posted by ShaoFeng Shi <sh...@apache.org>.
Yes it is expected; the child RDD is based on parent RDD; as parent RDD
already be generated so some stages will be skipped (each layer's DAG is a
full DAG from source data), those stages can be skipped. See this
screenshot:
[image: 内嵌图片 1]
2017-03-06 10:01 GMT+08:00 Mars Xu <xu...@gmail.com>:
> A more problem is confusing me now, I read this blog
> http://kylin.apache.org/blog/2017/02/23/by-layer-spark-cubing/ ,and got
> the key point of by-layer spark cubing is "a RDD per layer”, and during
> the cubing, I take a look at Spark DAG, is that a spark job for a layer ?
>
> my cube has 11 dimensions, the first spark job has 2 stages, Is the stage
> 1 get the first layer( 11-D Cuboid) RDD(in your blog is RDD1) ? In second
> job, it has 3 stages, the first stage is skipped , Are the remained 2
> stages get the second layer(10-D cuboids) RDD (RDD2) ? and In every follow
> spark jobs, always skipped some former stages ,just leave last 2 stages to
> implements.
>
> Please forgive my ignorance of this.
>
> Thanks.
>
>
> 在 2017年3月3日，下午6:10，ShaoFeng Shi <sh...@gmail.com> 写道：
>
> I'm afraid not that simple; the embedded spark in Kylin is v1.6.3, and the
> Kylin code is compiled with 1.6.3 API; if you want to switch to 2.1, need
> recompile and re-package. That is doable, you can take a try.
>
> Get Outlook for iOS <https://aka.ms/o0ukef>
>
>
>
>
> On Thu, Mar 2, 2017 at 1:37 PM +0700, "Mars Xu" <xu...@gmail.com>
> wrote:
>
> Hi, Shaofeng,
>>
>> Can I change the spark version to spark 2.1 using
>> “kylin.engine.spark-conf” to config ?
>>
>> 在 2017年3月1日，下午5:27，ShaoFeng Shi <sh...@apache.org> 写道：
>>
>> Kylin embed a spark, which is spark v1.6.3; I added a blog to briefly
>> introduce this, later will add a doc to instruct on how to config:
>>
>> https://kylin.apache.org/blog/2017/02/23/by-layer-spark-cubing/
>>
>> To customize the spark conf, please do that in kylin.properties with
>> prefix "kylin.engine.spark-conf."
>>
>> 2017-03-01 15:49 GMT+08:00 Mars Xu <xu...@gmail.com>:
>>
>>> Hi ,
>>> I have install kylin 2.0 beta on my cluster which has already deploy
>>> a spark 2.1, but when I startup kylin server and build cube, the Web UI of
>>> Cube Build Step show it is spark 1.6.3.
>>>
>>> Is that kylin already include a spark version ? or some other else I
>>> need to configure ?
>>>
>>>
>>> Thanks
>>>
>>> Mars.
>>
>>
>>
>>
>> --
>> Best regards,
>>
>> Shaofeng Shi 史少锋
>>
>>
>>
>
--
Best regards,
Shaofeng Shi 史少锋
Re: Which version of spark is kylin 2.0 Beta using ?
Posted by Mars Xu <xu...@gmail.com>.
A more problem is confusing me now, I read this blog http://kylin.apache.org/blog/2017/02/23/by-layer-spark-cubing/ <http://kylin.apache.org/blog/2017/02/23/by-layer-spark-cubing/> ,and got the key point of by-layer spark cubing is "a RDD per layer”, and during the cubing, I take a look at Spark DAG, is that a spark job for a layer ?
my cube has 11 dimensions, the first spark job has 2 stages, Is the stage 1 get the first layer( 11-D Cuboid) RDD(in your blog is RDD1) ? In second job, it has 3 stages, the first stage is skipped , Are the remained 2 stages get the second layer(10-D cuboids) RDD (RDD2) ? and In every follow spark jobs, always skipped some former stages ,just leave last 2 stages to implements.
Please forgive my ignorance of this.
Thanks.
> 在 2017年3月3日，下午6:10，ShaoFeng Shi <sh...@gmail.com> 写道：
>
> I'm afraid not that simple; the embedded spark in Kylin is v1.6.3, and the Kylin code is compiled with 1.6.3 API; if you want to switch to 2.1, need recompile and re-package. That is doable, you can take a try.
>
> Get Outlook for iOS <https://aka.ms/o0ukef>
>
>
>
> On Thu, Mar 2, 2017 at 1:37 PM +0700, "Mars Xu" <xujiao.mycafe@gmail.com <ma...@gmail.com>> wrote:
>
> Hi, Shaofeng,
>
> Can I change the spark version to spark 2.1 using “kylin.engine.spark-conf” to config ?
>
>> 在 2017年3月1日，下午5:27，ShaoFeng Shi <shaofengshi@apache.org <ma...@apache.org>> 写道：
>>
>> Kylin embed a spark, which is spark v1.6.3; I added a blog to briefly introduce this, later will add a doc to instruct on how to config:
>>
>> https://kylin.apache.org/blog/2017/02/23/by-layer-spark-cubing/ <https://kylin.apache.org/blog/2017/02/23/by-layer-spark-cubing/>
>>
>> To customize the spark conf, please do that in kylin.properties with prefix "kylin.engine.spark-conf."
>>
>> 2017-03-01 15:49 GMT+08:00 Mars Xu <xujiao.mycafe@gmail.com <ma...@gmail.com>>:
>> Hi ,
>> I have install kylin 2.0 beta on my cluster which has already deploy a spark 2.1, but when I startup kylin server and build cube, the Web UI of Cube Build Step show it is spark 1.6.3.
>>
>> Is that kylin already include a spark version ? or some other else I need to configure ?
>>
>>
>> Thanks
>>
>> Mars.
>>
>>
>>
>> --
>> Best regards,
>>
>> Shaofeng Shi 史少锋
>>
>
Re: Which version of spark is kylin 2.0 Beta using ?
Posted by ShaoFeng Shi <sh...@gmail.com>.
I'm afraid not that simple; the embedded spark in Kylin is v1.6.3, and the Kylin code is compiled with 1.6.3 API; if you want to switch to 2.1, need recompile and re-package. That is doable, you can take a try.
Get Outlook for iOS
On Thu, Mar 2, 2017 at 1:37 PM +0700, "Mars Xu" <xu...@gmail.com> wrote:
Hi, Shaofeng,
Can I change the spark version to spark 2.1 using “kylin.engine.spark-conf” to config ?
在 2017年3月1日，下午5:27，ShaoFeng Shi <sh...@apache.org> 写道：
Kylin embed a spark, which is spark v1.6.3; I added a blog to briefly introduce this, later will add a doc to instruct on how to config:
https://kylin.apache.org/blog/2017/02/23/by-layer-spark-cubing/
To customize the spark conf, please do that in kylin.properties with prefix "kylin.engine.spark-conf."
2017-03-01 15:49 GMT+08:00 Mars Xu <xu...@gmail.com>:
Hi ,
I have install kylin 2.0 beta on my cluster which has already deploy a spark 2.1, but when I startup kylin server and build cube, the Web UI of Cube Build Step show it is spark 1.6.3.
Is that kylin already include a spark version ? or some other else I need to configure ?
Thanks
Mars.
--
Best regards,
Shaofeng Shi 史少锋
Re: Which version of spark is kylin 2.0 Beta using ?
Posted by ShaoFeng Shi <sh...@gmail.com>.
I'm afraid not that simple; the embedded spark in Kylin is v1.6.3, and the Kylin code is compiled with 1.6.3 API; if you want to switch to 2.1, need recompile and re-package. That is doable, you can take a try.
Get Outlook for iOS
On Thu, Mar 2, 2017 at 1:37 PM +0700, "Mars Xu" <xu...@gmail.com> wrote:
Hi, Shaofeng,
Can I change the spark version to spark 2.1 using “kylin.engine.spark-conf” to config ?
在 2017年3月1日，下午5:27，ShaoFeng Shi <sh...@apache.org> 写道：
Kylin embed a spark, which is spark v1.6.3; I added a blog to briefly introduce this, later will add a doc to instruct on how to config:
https://kylin.apache.org/blog/2017/02/23/by-layer-spark-cubing/
To customize the spark conf, please do that in kylin.properties with prefix "kylin.engine.spark-conf."
2017-03-01 15:49 GMT+08:00 Mars Xu <xu...@gmail.com>:
Hi ,
I have install kylin 2.0 beta on my cluster which has already deploy a spark 2.1, but when I startup kylin server and build cube, the Web UI of Cube Build Step show it is spark 1.6.3.
Is that kylin already include a spark version ? or some other else I need to configure ?
Thanks
Mars.
--
Best regards,
Shaofeng Shi 史少锋
Re: Which version of spark is kylin 2.0 Beta using ?
Posted by Mars Xu <xu...@gmail.com>.
Hi, Shaofeng,
Can I change the spark version to spark 2.1 using “kylin.engine.spark-conf” to config ?
> 在 2017年3月1日，下午5:27，ShaoFeng Shi <sh...@apache.org> 写道：
>
> Kylin embed a spark, which is spark v1.6.3; I added a blog to briefly introduce this, later will add a doc to instruct on how to config:
>
> https://kylin.apache.org/blog/2017/02/23/by-layer-spark-cubing/ <https://kylin.apache.org/blog/2017/02/23/by-layer-spark-cubing/>
>
> To customize the spark conf, please do that in kylin.properties with prefix "kylin.engine.spark-conf."
>
> 2017-03-01 15:49 GMT+08:00 Mars Xu <xujiao.mycafe@gmail.com <ma...@gmail.com>>:
> Hi ,
> I have install kylin 2.0 beta on my cluster which has already deploy a spark 2.1, but when I startup kylin server and build cube, the Web UI of Cube Build Step show it is spark 1.6.3.
>
> Is that kylin already include a spark version ? or some other else I need to configure ?
>
>
> Thanks
>
> Mars.
>
>
>
> --
> Best regards,
>
> Shaofeng Shi 史少锋
>
Re: Which version of spark is kylin 2.0 Beta using ?—— How can others get data used in your test？
Posted by Mars Xu <xu...@gmail.com>.
Get it, Thanks a lot.
> 在 2017年3月6日，上午9:50，ShaoFeng Shi <sh...@gmail.com> 写道：
>
> Hi Mars,
>
> The cube "airline_cube1" is the cube using spark engine; others are using MR, their logical structure is the same; Please note the properties "kylin.engine.spark.config-override.*" in conf/kylin.properties are renamed to "kylin.engine.spark-conf.*" in the beta release. For your reference.
>
> 2017-03-03 18:05 GMT+08:00 ShaoFeng Shi <shaofengshi@gmail.com <ma...@gmail.com>>:
> hi mars,
>
> The dataset can be download from internet, you can refer this article to get it: http://kyligence.io/2016/11/1729/ <http://kyligence.io/2016/11/1729/>
>
> For the cube model, I can share it later (next week) when back to office.
>
>
>
> Get Outlook for iOS <https://aka.ms/o0ukef>
>
>
>
> On Thu, Mar 2, 2017 at 4:53 PM +0700, "Mars Xu" <xujiao.mycafe@gmail.com <ma...@gmail.com>> wrote:
>
> Hello Shaofeng,
>
> Thanks a lot for providing the by-layer-spark cubing model, I have seen the blog which metioned your test on Spark, My question is can other people get the test data ?
> <PastedGraphic-1.png>
> If many users can use the same data to test kylin 2.0, may be it will improve kylin more efficiently.
>
> Thanks again.
>
> Mars.
>
>
>
>> 在 2017年3月1日，下午5:27，ShaoFeng Shi <shaofengshi@apache.org <ma...@apache.org>> 写道：
>>
>> Kylin embed a spark, which is spark v1.6.3; I added a blog to briefly introduce this, later will add a doc to instruct on how to config:
>>
>> https://kylin.apache.org/blog/2017/02/23/by-layer-spark-cubing/ <https://kylin.apache.org/blog/2017/02/23/by-layer-spark-cubing/>
>>
>> To customize the spark conf, please do that in kylin.properties with prefix "kylin.engine.spark-conf."
>>
>> 2017-03-01 15:49 GMT+08:00 Mars Xu <xujiao.mycafe@gmail.com <ma...@gmail.com>>:
>> Hi ,
>> I have install kylin 2.0 beta on my cluster which has already deploy a spark 2.1, but when I startup kylin server and build cube, the Web UI of Cube Build Step show it is spark 1.6.3.
>>
>> Is that kylin already include a spark version ? or some other else I need to configure ?
>>
>>
>> Thanks
>>
>> Mars.
>>
>>
>>
>> --
>> Best regards,
>>
>> Shaofeng Shi 史少锋
>>
>
>
> <project_2016_10_10_09_00_37.zip>
Re: Which version of spark is kylin 2.0 Beta using ?—— How can others get data used in your test？
Posted by ShaoFeng Shi <sh...@gmail.com>.
Hi Mars,
The cube "airline_cube1" is the cube using spark engine; others are using
MR, their logical structure is the same; Please note the properties
"kylin.engine.spark.config-override.*" in conf/kylin.properties are renamed
to "kylin.engine.spark-conf.*" in the beta release. For your reference.
2017-03-03 18:05 GMT+08:00 ShaoFeng Shi <sh...@gmail.com>:
> hi mars,
>
> The dataset can be download from internet, you can refer this article to
> get it: http://kyligence.io/2016/11/1729/
>
> For the cube model, I can share it later (next week) when back to office.
>
>
>
> Get Outlook for iOS <https://aka.ms/o0ukef>
>
>
>
>
> On Thu, Mar 2, 2017 at 4:53 PM +0700, "Mars Xu" <xu...@gmail.com>
> wrote:
>
> Hello Shaofeng,
>>
>> Thanks a lot for providing the by-layer-spark cubing model, I have
>> seen the blog which metioned your test on Spark, My question is can other
>> people get the test data ?
>> If many users can use the same data to test kylin 2.0, may be it will
>> improve kylin more efficiently.
>>
>> Thanks again.
>>
>> Mars.
>>
>>
>>
>> 在 2017年3月1日，下午5:27，ShaoFeng Shi <sh...@apache.org> 写道：
>>
>> Kylin embed a spark, which is spark v1.6.3; I added a blog to briefly
>> introduce this, later will add a doc to instruct on how to config:
>>
>> https://kylin.apache.org/blog/2017/02/23/by-layer-spark-cubing/
>>
>> To customize the spark conf, please do that in kylin.properties with
>> prefix "kylin.engine.spark-conf."
>>
>> 2017-03-01 15:49 GMT+08:00 Mars Xu <xu...@gmail.com>:
>>
>>> Hi ,
>>> I have install kylin 2.0 beta on my cluster which has already deploy
>>> a spark 2.1, but when I startup kylin server and build cube, the Web UI of
>>> Cube Build Step show it is spark 1.6.3.
>>>
>>> Is that kylin already include a spark version ? or some other else I
>>> need to configure ?
>>>
>>>
>>> Thanks
>>>
>>> Mars.
>>
>>
>>
>>
>> --
>> Best regards,
>>
>> Shaofeng Shi 史少锋
>>
>>
>>
Re: Which version of spark is kylin 2.0 Beta using ?—— How can others get data used in your test？
Posted by ShaoFeng Shi <sh...@gmail.com>.
Hi Mars,
The cube "airline_cube1" is the cube using spark engine; others are using
MR, their logical structure is the same; Please note the properties
"kylin.engine.spark.config-override.*" in conf/kylin.properties are renamed
to "kylin.engine.spark-conf.*" in the beta release. For your reference.
2017-03-03 18:05 GMT+08:00 ShaoFeng Shi <sh...@gmail.com>:
> hi mars,
>
> The dataset can be download from internet, you can refer this article to
> get it: http://kyligence.io/2016/11/1729/
>
> For the cube model, I can share it later (next week) when back to office.
>
>
>
> Get Outlook for iOS <https://aka.ms/o0ukef>
>
>
>
>
> On Thu, Mar 2, 2017 at 4:53 PM +0700, "Mars Xu" <xu...@gmail.com>
> wrote:
>
> Hello Shaofeng,
>>
>> Thanks a lot for providing the by-layer-spark cubing model, I have
>> seen the blog which metioned your test on Spark, My question is can other
>> people get the test data ?
>> If many users can use the same data to test kylin 2.0, may be it will
>> improve kylin more efficiently.
>>
>> Thanks again.
>>
>> Mars.
>>
>>
>>
>> 在 2017年3月1日，下午5:27，ShaoFeng Shi <sh...@apache.org> 写道：
>>
>> Kylin embed a spark, which is spark v1.6.3; I added a blog to briefly
>> introduce this, later will add a doc to instruct on how to config:
>>
>> https://kylin.apache.org/blog/2017/02/23/by-layer-spark-cubing/
>>
>> To customize the spark conf, please do that in kylin.properties with
>> prefix "kylin.engine.spark-conf."
>>
>> 2017-03-01 15:49 GMT+08:00 Mars Xu <xu...@gmail.com>:
>>
>>> Hi ,
>>> I have install kylin 2.0 beta on my cluster which has already deploy
>>> a spark 2.1, but when I startup kylin server and build cube, the Web UI of
>>> Cube Build Step show it is spark 1.6.3.
>>>
>>> Is that kylin already include a spark version ? or some other else I
>>> need to configure ?
>>>
>>>
>>> Thanks
>>>
>>> Mars.
>>
>>
>>
>>
>> --
>> Best regards,
>>
>> Shaofeng Shi 史少锋
>>
>>
>>
Re: Which version of spark is kylin 2.0 Beta using ?—— How can others get data used in your test？
Posted by ShaoFeng Shi <sh...@gmail.com>.
hi mars,
The dataset can be download from internet, you can refer this article to get it: http://kyligence.io/2016/11/1729/
For the cube model, I can share it later (next week) when back to office.
Get Outlook for iOS
On Thu, Mar 2, 2017 at 4:53 PM +0700, "Mars Xu" <xu...@gmail.com> wrote:
Hello Shaofeng,
Thanks a lot for providing the by-layer-spark cubing model, I have seen the blog which metioned your test on Spark, My question is can other people get the test data ? If many users can use the same data to test kylin 2.0, may be it will improve kylin more efficiently.
Thanks again.
Mars.
在 2017年3月1日，下午5:27，ShaoFeng Shi <sh...@apache.org> 写道：
Kylin embed a spark, which is spark v1.6.3; I added a blog to briefly introduce this, later will add a doc to instruct on how to config:
https://kylin.apache.org/blog/2017/02/23/by-layer-spark-cubing/
To customize the spark conf, please do that in kylin.properties with prefix "kylin.engine.spark-conf."
2017-03-01 15:49 GMT+08:00 Mars Xu <xu...@gmail.com>:
Hi ,
I have install kylin 2.0 beta on my cluster which has already deploy a spark 2.1, but when I startup kylin server and build cube, the Web UI of Cube Build Step show it is spark 1.6.3.
Is that kylin already include a spark version ? or some other else I need to configure ?
Thanks
Mars.
--
Best regards,
Shaofeng Shi 史少锋
Re: Which version of spark is kylin 2.0 Beta using ?—— How can others get data used in your test？
Posted by ShaoFeng Shi <sh...@gmail.com>.
hi mars,
The dataset can be download from internet, you can refer this article to get it: http://kyligence.io/2016/11/1729/
For the cube model, I can share it later (next week) when back to office.
Get Outlook for iOS
On Thu, Mar 2, 2017 at 4:53 PM +0700, "Mars Xu" <xu...@gmail.com> wrote:
Hello Shaofeng,
Thanks a lot for providing the by-layer-spark cubing model, I have seen the blog which metioned your test on Spark, My question is can other people get the test data ? If many users can use the same data to test kylin 2.0, may be it will improve kylin more efficiently.
Thanks again.
Mars.
在 2017年3月1日，下午5:27，ShaoFeng Shi <sh...@apache.org> 写道：
Kylin embed a spark, which is spark v1.6.3; I added a blog to briefly introduce this, later will add a doc to instruct on how to config:
https://kylin.apache.org/blog/2017/02/23/by-layer-spark-cubing/
To customize the spark conf, please do that in kylin.properties with prefix "kylin.engine.spark-conf."
2017-03-01 15:49 GMT+08:00 Mars Xu <xu...@gmail.com>:
Hi ,
I have install kylin 2.0 beta on my cluster which has already deploy a spark 2.1, but when I startup kylin server and build cube, the Web UI of Cube Build Step show it is spark 1.6.3.
Is that kylin already include a spark version ? or some other else I need to configure ?
Thanks
Mars.
--
Best regards,
Shaofeng Shi 史少锋
Re: Which version of spark is kylin 2.0 Beta using ?—— How can others get data used in your test？
Posted by Mars Xu <xu...@gmail.com>.
Hello Shaofeng,
Thanks a lot for providing the by-layer-spark cubing model, I have seen the blog which metioned your test on Spark, My question is can other people get the test data ?
If many users can use the same data to test kylin 2.0, may be it will improve kylin more efficiently.
Thanks again.
Mars.
> 在 2017年3月1日，下午5:27，ShaoFeng Shi <sh...@apache.org> 写道：
>
> Kylin embed a spark, which is spark v1.6.3; I added a blog to briefly introduce this, later will add a doc to instruct on how to config:
>
> https://kylin.apache.org/blog/2017/02/23/by-layer-spark-cubing/ <https://kylin.apache.org/blog/2017/02/23/by-layer-spark-cubing/>
>
> To customize the spark conf, please do that in kylin.properties with prefix "kylin.engine.spark-conf."
>
> 2017-03-01 15:49 GMT+08:00 Mars Xu <xujiao.mycafe@gmail.com <ma...@gmail.com>>:
> Hi ,
> I have install kylin 2.0 beta on my cluster which has already deploy a spark 2.1, but when I startup kylin server and build cube, the Web UI of Cube Build Step show it is spark 1.6.3.
>
> Is that kylin already include a spark version ? or some other else I need to configure ?
>
>
> Thanks
>
> Mars.
>
>
>
> --
> Best regards,
>
> Shaofeng Shi 史少锋
>