You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@kylin.apache.org by 谢巍盛 <xi...@bestpay.com.cn> on 2018/02/06 13:10:54 UTC
Unable to build cube with Spark
Hi,
we follow the step in
http://kylin.apache.org/docs21/tutorial/cube_spark.html try to build
cube with Spark, but couldnt get it worked. The cube engine on the webUI
didnt offer another option other than MR, it didnt show Spark(beta) as
the tutorial shows. Anyone knows what the problem might be? Or is there
any way to debug this issue? Thanks.
Best,
Vincent
Re: Unable to build cube with Spark
Posted by ShaoFeng Shi <sh...@apache.org>.
OK, thanks for the update!
2018-02-07 15:50 GMT+08:00 谢巍盛 <xi...@bestpay.com.cn>:
> 1. we were unable to build cube, the issue was later rooted in missing a
> spark library, which is weird, because we didnt set up kylin with spark by
> that time, we just wanna try it on MR first.
> 2. When we trying to set up kylin with Spark following the tutorial on
> kylin's official web, we cannot find the 'Spark' option in cube engine,
> this issue is still under investigation and we dont have much clue on this.
>
>
> On 2/7/18 3:28 PM, ShaoFeng Shi wrote:
>
> CDH is also supported. But the vendors may have customization on Spark, so
> sometimes you may need to use their Spark release or copying additional
> jars to the Spark that Kylin ships.
>
> The information you provided is limited; In the first post you said there
> is no Spark option in the cube wizard, in the second you fix the issue by
> adding spark-network jar, I don't know how they are related, as the first
> seems to be the GUI issue.
>
> 2018-02-07 9:51 GMT+08:00 谢巍盛 <xi...@bestpay.com.cn>:
>
>> Okay, if that's the case, I assume setting up kylin on hadoop with apache
>> version instead of CDH/HDP wont have this issue.
>>
>> Thanks.
>>
>> On 2/7/18 9:44 AM, Ted Yu wrote:
>>
>> Since you used a vendor's Kylin package, please consider asking on their
>> mailing list.
>>
>> -------- Original message --------
>> From: 谢巍盛 <xi...@bestpay.com.cn> <xi...@bestpay.com.cn>
>> Date: 2/6/18 5:36 PM (GMT-08:00)
>> To: user@kylin.apache.org
>> Subject: Re: Unable to build cube with Spark
>>
>> Hi Shaofeng,
>>
>> The kylin used on our end is *apache*-*kylin*-*2.2.0*-*bin*-*cdh57*
>> data source is on Hive.
>>
>> We managed to fix this issue by adding spark-network-common_2.10-1.6.0-cdh5.9.2.jar
>> under $SPARK_HOME/lib/
>>
>> Still, it seems weird to us, we didnt setup Spark for kylin, somehow
>> kylin will look for this library while building cube. Any clue?
>>
>> Best,
>> Vincent
>>
>> On 2/7/18 9:08 AM, ShaoFeng Shi wrote:
>>
>> Hi Weisheng,
>>
>> Two questions:
>> What's your Kylin version? and what's the data source of this cube, Hive
>> or Kafka?
>>
>> 2018-02-06 21:10 GMT+08:00 谢巍盛 <xi...@bestpay.com.cn>:
>>
>>> Hi,
>>>
>>> we follow the step in http://kylin.apache.org/docs21
>>> /tutorial/cube_spark.html try to build cube with Spark, but couldnt get
>>> it worked. The cube engine on the webUI didnt offer another option other
>>> than MR, it didnt show Spark(beta) as the tutorial shows. Anyone knows what
>>> the problem might be? Or is there any way to debug this issue? Thanks.
>>>
>>> Best,
>>>
>>> Vincent
>>>
>>>
>>
>>
>> --
>> Best regards,
>>
>> Shaofeng Shi 史少锋
>>
>>
>>
>>
>
>
> --
> Best regards,
>
> Shaofeng Shi 史少锋
>
>
>
--
Best regards,
Shaofeng Shi 史少锋
Re: Unable to build cube with Spark
Posted by 谢巍盛 <xi...@bestpay.com.cn>.
1. we were unable to build cube, the issue was later rooted in missing a
spark library, which is weird, because we didnt set up kylin with spark
by that time, we just wanna try it on MR first.
2. When we trying to set up kylin with Spark following the tutorial on
kylin's official web, we cannot find the 'Spark' option in cube engine,
this issue is still under investigation and we dont have much clue on this.
On 2/7/18 3:28 PM, ShaoFeng Shi wrote:
> CDH is also supported. But the vendors may have customization on
> Spark, so sometimes you may need to use their Spark release or copying
> additional jars to the Spark that Kylin ships.
>
> The information you provided is limited; In the first post you said
> there is no Spark option in the cube wizard, in the second you fix the
> issue by adding spark-network jar, I don't know how they are related,
> as the first seems to be the GUI issue.
>
> 2018-02-07 9:51 GMT+08:00 谢巍盛 <xieweisheng@bestpay.com.cn
> <ma...@bestpay.com.cn>>:
>
> Okay, if that's the case, I assume setting up kylin on hadoop with
> apache version instead of CDH/HDP wont have this issue.
>
> Thanks.
>
>
> On 2/7/18 9:44 AM, Ted Yu wrote:
>> Since you used a vendor's Kylin package, please consider asking
>> on their mailing list.
>>
>> -------- Original message --------
>> From: 谢巍盛 <xi...@bestpay.com.cn>
>> <ma...@bestpay.com.cn>
>> Date: 2/6/18 5:36 PM (GMT-08:00)
>> To: user@kylin.apache.org <ma...@kylin.apache.org>
>> Subject: Re: Unable to build cube with Spark
>>
>> Hi Shaofeng,
>>
>> The kylin used on our end is /apache/-/kylin/-/2.2.0/-/bin/-/cdh57/
>>
>> data source is on Hive.
>>
>> We managed to fix this issue by adding
>> spark-network-common_2.10-1.6.0-cdh5.9.2.jar under $SPARK_HOME/lib/
>>
>> Still, it seems weird to us, we didnt setup Spark for kylin,
>> somehow kylin will look for this library while building cube. Any
>> clue?
>>
>> Best,
>> Vincent
>>
>> On 2/7/18 9:08 AM, ShaoFeng Shi wrote:
>>> Hi Weisheng,
>>>
>>> Two questions:
>>> What's your Kylin version? and what's the data source of this
>>> cube, Hive or Kafka?
>>>
>>> 2018-02-06 21:10 GMT+08:00 谢巍盛 <xieweisheng@bestpay.com.cn
>>> <ma...@bestpay.com.cn>>:
>>>
>>> Hi,
>>>
>>> we follow the step in
>>> http://kylin.apache.org/docs21/tutorial/cube_spark.html
>>> <http://kylin.apache.org/docs21/tutorial/cube_spark.html>
>>> try to build cube with Spark, but couldnt get it worked. The
>>> cube engine on the webUI didnt offer another option other
>>> than MR, it didnt show Spark(beta) as the tutorial shows.
>>> Anyone knows what the problem might be? Or is there any way
>>> to debug this issue? Thanks.
>>>
>>> Best,
>>>
>>> Vincent
>>>
>>>
>>>
>>>
>>> --
>>> Best regards,
>>>
>>> Shaofeng Shi 史少锋
>>>
>>
>
>
>
>
> --
> Best regards,
>
> Shaofeng Shi 史少锋
>
Re: Unable to build cube with Spark
Posted by ShaoFeng Shi <sh...@apache.org>.
CDH is also supported. But the vendors may have customization on Spark, so
sometimes you may need to use their Spark release or copying additional
jars to the Spark that Kylin ships.
The information you provided is limited; In the first post you said there
is no Spark option in the cube wizard, in the second you fix the issue by
adding spark-network jar, I don't know how they are related, as the first
seems to be the GUI issue.
2018-02-07 9:51 GMT+08:00 谢巍盛 <xi...@bestpay.com.cn>:
> Okay, if that's the case, I assume setting up kylin on hadoop with apache
> version instead of CDH/HDP wont have this issue.
>
> Thanks.
>
> On 2/7/18 9:44 AM, Ted Yu wrote:
>
> Since you used a vendor's Kylin package, please consider asking on their
> mailing list.
>
> -------- Original message --------
> From: 谢巍盛 <xi...@bestpay.com.cn> <xi...@bestpay.com.cn>
> Date: 2/6/18 5:36 PM (GMT-08:00)
> To: user@kylin.apache.org
> Subject: Re: Unable to build cube with Spark
>
> Hi Shaofeng,
>
> The kylin used on our end is *apache*-*kylin*-*2.2.0*-*bin*-*cdh57*
> data source is on Hive.
>
> We managed to fix this issue by adding spark-network-common_2.10-1.6.0-cdh5.9.2.jar
> under $SPARK_HOME/lib/
>
> Still, it seems weird to us, we didnt setup Spark for kylin, somehow kylin
> will look for this library while building cube. Any clue?
>
> Best,
> Vincent
>
> On 2/7/18 9:08 AM, ShaoFeng Shi wrote:
>
> Hi Weisheng,
>
> Two questions:
> What's your Kylin version? and what's the data source of this cube, Hive
> or Kafka?
>
> 2018-02-06 21:10 GMT+08:00 谢巍盛 <xi...@bestpay.com.cn>:
>
>> Hi,
>>
>> we follow the step in http://kylin.apache.org/docs21
>> /tutorial/cube_spark.html try to build cube with Spark, but couldnt get
>> it worked. The cube engine on the webUI didnt offer another option other
>> than MR, it didnt show Spark(beta) as the tutorial shows. Anyone knows what
>> the problem might be? Or is there any way to debug this issue? Thanks.
>>
>> Best,
>>
>> Vincent
>>
>>
>
>
> --
> Best regards,
>
> Shaofeng Shi 史少锋
>
>
>
>
--
Best regards,
Shaofeng Shi 史少锋
Re: Unable to build cube with Spark
Posted by 谢巍盛 <xi...@bestpay.com.cn>.
Okay, if that's the case, I assume setting up kylin on hadoop with
apache version instead of CDH/HDP wont have this issue.
Thanks.
On 2/7/18 9:44 AM, Ted Yu wrote:
> Since you used a vendor's Kylin package, please consider asking on
> their mailing list.
>
> -------- Original message --------
> From: 谢巍盛 <xi...@bestpay.com.cn>
> Date: 2/6/18 5:36 PM (GMT-08:00)
> To: user@kylin.apache.org
> Subject: Re: Unable to build cube with Spark
>
> Hi Shaofeng,
>
> The kylin used on our end is /apache/-/kylin/-/2.2.0/-/bin/-/cdh57/
>
> data source is on Hive.
>
> We managed to fix this issue by adding
> spark-network-common_2.10-1.6.0-cdh5.9.2.jar under $SPARK_HOME/lib/
>
> Still, it seems weird to us, we didnt setup Spark for kylin, somehow
> kylin will look for this library while building cube. Any clue?
>
> Best,
> Vincent
>
> On 2/7/18 9:08 AM, ShaoFeng Shi wrote:
>> Hi Weisheng,
>>
>> Two questions:
>> What's your Kylin version? and what's the data source of this cube,
>> Hive or Kafka?
>>
>> 2018-02-06 21:10 GMT+08:00 谢巍盛 <xieweisheng@bestpay.com.cn
>> <ma...@bestpay.com.cn>>:
>>
>> Hi,
>>
>> we follow the step in
>> http://kylin.apache.org/docs21/tutorial/cube_spark.html
>> <http://kylin.apache.org/docs21/tutorial/cube_spark.html> try to
>> build cube with Spark, but couldnt get it worked. The cube engine
>> on the webUI didnt offer another option other than MR, it didnt
>> show Spark(beta) as the tutorial shows. Anyone knows what the
>> problem might be? Or is there any way to debug this issue? Thanks.
>>
>> Best,
>>
>> Vincent
>>
>>
>>
>>
>> --
>> Best regards,
>>
>> Shaofeng Shi 史少锋
>>
>
Re: Unable to build cube with Spark
Posted by Ted Yu <yu...@gmail.com>.
Since you used a vendor's Kylin package, please consider asking on their mailing list.
-------- Original message --------From: 谢巍盛 <xi...@bestpay.com.cn> Date: 2/6/18 5:36 PM (GMT-08:00) To: user@kylin.apache.org Subject: Re: Unable to build cube with Spark
Hi Shaofeng,
The kylin used on our end is apache-kylin-2.2.0-bin-cdh57
data source is on Hive.
We managed to fix this issue by adding
spark-network-common_2.10-1.6.0-cdh5.9.2.jar under $SPARK_HOME/lib/
Still, it seems weird to us, we didnt setup Spark for kylin, somehow
kylin will look for this library while building cube. Any clue?
Best,
Vincent
On 2/7/18 9:08 AM, ShaoFeng Shi wrote:
Hi Weisheng,
Two questions:
What's your Kylin version? and what's the data source of
this cube, Hive or Kafka?
2018-02-06 21:10 GMT+08:00 谢巍盛 <xi...@bestpay.com.cn>:
Hi,
we follow the step in http://kylin.apache.org/docs21/tutorial/cube_spark.html
try to build cube with Spark, but couldnt get it worked. The
cube engine on the webUI didnt offer another option other
than MR, it didnt show Spark(beta) as the tutorial shows.
Anyone knows what the problem might be? Or is there any way
to debug this issue? Thanks.
Best,
Vincent
--
Best regards,
Shaofeng Shi 史少锋
Re: Unable to build cube with Spark
Posted by 谢巍盛 <xi...@bestpay.com.cn>.
Hi Shaofeng,
The kylin used on our end is /apache/-/kylin/-/2.2.0/-/bin/-/cdh57/
data source is on Hive.
We managed to fix this issue by adding
spark-network-common_2.10-1.6.0-cdh5.9.2.jar under $SPARK_HOME/lib/
Still, it seems weird to us, we didnt setup Spark for kylin, somehow
kylin will look for this library while building cube. Any clue?
Best,
Vincent
On 2/7/18 9:08 AM, ShaoFeng Shi wrote:
> Hi Weisheng,
>
> Two questions:
> What's your Kylin version? and what's the data source of this cube,
> Hive or Kafka?
>
> 2018-02-06 21:10 GMT+08:00 谢巍盛 <xieweisheng@bestpay.com.cn
> <ma...@bestpay.com.cn>>:
>
> Hi,
>
> we follow the step in
> http://kylin.apache.org/docs21/tutorial/cube_spark.html
> <http://kylin.apache.org/docs21/tutorial/cube_spark.html> try to
> build cube with Spark, but couldnt get it worked. The cube engine
> on the webUI didnt offer another option other than MR, it didnt
> show Spark(beta) as the tutorial shows. Anyone knows what the
> problem might be? Or is there any way to debug this issue? Thanks.
>
> Best,
>
> Vincent
>
>
>
>
> --
> Best regards,
>
> Shaofeng Shi 史少锋
>
Re: Unable to build cube with Spark
Posted by ShaoFeng Shi <sh...@apache.org>.
Hi Weisheng,
Two questions:
What's your Kylin version? and what's the data source of this cube, Hive or
Kafka?
2018-02-06 21:10 GMT+08:00 谢巍盛 <xi...@bestpay.com.cn>:
> Hi,
>
> we follow the step in http://kylin.apache.org/docs21
> /tutorial/cube_spark.html try to build cube with Spark, but couldnt get
> it worked. The cube engine on the webUI didnt offer another option other
> than MR, it didnt show Spark(beta) as the tutorial shows. Anyone knows what
> the problem might be? Or is there any way to debug this issue? Thanks.
>
> Best,
>
> Vincent
>
>
--
Best regards,
Shaofeng Shi 史少锋