kylin build cube based on spark

[wangfeng <wa...@gmail.com>]

Hi everyone, I want to know, if Kylin could build cube olnly base on spark. I
means that my environment doesn't support running MR jobs, and spark is
running on Mesos. 

In few days ago, I complied with "Fast Cubing on Spark in Apache Kylin"
(http://kylin.apache.org/blog/2015/09/09/fast-cubing-on-spark/)  and found
that: only in the "#1 Step Name: Create Intermediate Flat Hive Table"  shows
kylin use Spark to create table, and the next steps are all using MR...... 

And I read the source code of Kylin,found that there is a file named with
"engine-spark" while the other named "engine-mr" I guess that kylin could
build cube only through Spark, however, I try my best to deploy the Kylin
but there is nothing to do....

so I want to someone to answer my question, and it would be very kind of you
to tell me how to deal with this problem 

Thank you for helping me....

------------------------------------------------------translate-------------------------------------------------------------------------------------
大家好,我有一个问题想请教大家:我想知道 kylin 在build cube
这个过程中,是否能够完全运行在spark中,而不是依赖或则还需要运行在MR上. 因为在之前我按照官网上面所给出的"Fast Cubing on
Spark in Apache Kylin"
(http://kylin.apache.org/blog/2015/09/09/fast-cubing-on-spark/)这个配置示例进行了配置,但是发现,只有在
"#1 Step Name: Create Intermediate Flat Hive Table"   这个步骤中 是使用的spark
,后面的steps 都是使用的MR..并且我在kylin-1.5.3 的源码中发现了"engine-spark"这个文件夹,以及"engine-mr"
因此猜测是能够完全在build cube中运行在spark 上的,同时根据kylin 的更新日志上面写的,他们能够运行在spark上,只是速度要比MR
稍微慢点...我想请问大家,如果可以完全运行在spark上,那么我应该修改哪些配置?(我猜测kylin_job_conf.xml 这个可能要修改)
请大家帮助我  谢谢大家...

--
View this message in context: http://apache-kylin.74782.x6.nabble.com/kylin-build-cube-based-on-spark-tp5447.html
Sent from the Apache Kylin mailing list archive at Nabble.com.

Re: kylin build cube based on spark

[Luke Han <lu...@gmail.com>]

That part is for experimental only which we tried to compare performance
between Spark & MR cubing.

For real world, we would like to recommend to leverage MR cubing for
today's releases, the Spark one not updated in recent versions.

Thanks.


Best Regards!
---------------------

Luke Han

On Sat, Jul 30, 2016 at 10:25 AM, wangfeng <wa...@gmail.com>
wrote:

> btw my Kylin version is :apache-kylin-1.5.0-HBase1.1.3
>
> 我的 kylin版本是:apache-kylin-1.5.0-HBase1.1.3
>
> --
> View this message in context:
> http://apache-kylin.74782.x6.nabble.com/kylin-build-cube-based-on-spark-tp5447p5448.html
> Sent from the Apache Kylin mailing list archive at Nabble.com.
>

Re: kylin build cube based on spark

[wangfeng <wa...@gmail.com>]

btw my Kylin version is :apache-kylin-1.5.0-HBase1.1.3

我的 kylin版本是:apache-kylin-1.5.0-HBase1.1.3

--
View this message in context: http://apache-kylin.74782.x6.nabble.com/kylin-build-cube-based-on-spark-tp5447p5448.html
Sent from the Apache Kylin mailing list archive at Nabble.com.