You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@kylin.apache.org by ShaoFeng Shi <sh...@apache.org> on 2018/09/18 09:34:39 UTC
[Discuss] Exclude Spark from Kylin's binary package
Hello,
Today Kylin's binary package ships an Apache Spark binary, the total
package is 297 MB in v2.5.0; I noticed that if the package is above 300 MB,
it would be rejected by Apache SVN. Now in the whole package, Spark takes
about 200 MB. Later when we upgrade to a higher version, the package can be
bigger. Without spark, the package is around 100 MB.
Since we don't have customization to Spark, we can provide a
download-spark.sh script for the user to easily download. If the user
already has that in local, he just needs to copy or create a symbolic link
in Kylin folder.
This will reduce 2/3 package size, and save time/bandwidth for distributing
Kylin. If you see a problem this change may bring, please complain.
--
Best regards,
Shaofeng Shi 史少锋
Re: [Discuss] Exclude Spark from Kylin's binary package
Posted by ShaoFeng Shi <sh...@apache.org>.
Thanks for billy's comment; A JIRA is created for this change:
https://issues.apache.org/jira/browse/KYLIN-3571
Billy Liu <bi...@apache.org> 于2018年9月18日周二 下午10:35写道：
> I like the idea to remove the built-in spark distribution. I think
> Kylin could work well with all spark 2.1+
>
> With Warm regards
>
> Billy Liu
>
> ShaoFeng Shi <sh...@apache.org> 于2018年9月18日周二 下午5:35写道：
> >
> > Hello,
> >
> > Today Kylin's binary package ships an Apache Spark binary, the total
> package is 297 MB in v2.5.0; I noticed that if the package is above 300 MB,
> it would be rejected by Apache SVN. Now in the whole package, Spark takes
> about 200 MB. Later when we upgrade to a higher version, the package can be
> bigger. Without spark, the package is around 100 MB.
> >
> > Since we don't have customization to Spark, we can provide a
> download-spark.sh script for the user to easily download. If the user
> already has that in local, he just needs to copy or create a symbolic link
> in Kylin folder.
> >
> > This will reduce 2/3 package size, and save time/bandwidth for
> distributing Kylin. If you see a problem this change may bring, please
> complain.
> >
> > --
> > Best regards,
> >
> > Shaofeng Shi 史少锋
> >
>
--
Best regards,
Shaofeng Shi 史少锋
Re: [Discuss] Exclude Spark from Kylin's binary package
Posted by ShaoFeng Shi <sh...@apache.org>.
Thanks for billy's comment; A JIRA is created for this change:
https://issues.apache.org/jira/browse/KYLIN-3571
Billy Liu <bi...@apache.org> 于2018年9月18日周二 下午10:35写道：
> I like the idea to remove the built-in spark distribution. I think
> Kylin could work well with all spark 2.1+
>
> With Warm regards
>
> Billy Liu
>
> ShaoFeng Shi <sh...@apache.org> 于2018年9月18日周二 下午5:35写道：
> >
> > Hello,
> >
> > Today Kylin's binary package ships an Apache Spark binary, the total
> package is 297 MB in v2.5.0; I noticed that if the package is above 300 MB,
> it would be rejected by Apache SVN. Now in the whole package, Spark takes
> about 200 MB. Later when we upgrade to a higher version, the package can be
> bigger. Without spark, the package is around 100 MB.
> >
> > Since we don't have customization to Spark, we can provide a
> download-spark.sh script for the user to easily download. If the user
> already has that in local, he just needs to copy or create a symbolic link
> in Kylin folder.
> >
> > This will reduce 2/3 package size, and save time/bandwidth for
> distributing Kylin. If you see a problem this change may bring, please
> complain.
> >
> > --
> > Best regards,
> >
> > Shaofeng Shi 史少锋
> >
>
--
Best regards,
Shaofeng Shi 史少锋
Re: [Discuss] Exclude Spark from Kylin's binary package
Posted by Billy Liu <bi...@apache.org>.
I like the idea to remove the built-in spark distribution. I think
Kylin could work well with all spark 2.1+
With Warm regards
Billy Liu
ShaoFeng Shi <sh...@apache.org> 于2018年9月18日周二 下午5:35写道：
>
> Hello,
>
> Today Kylin's binary package ships an Apache Spark binary, the total package is 297 MB in v2.5.0; I noticed that if the package is above 300 MB, it would be rejected by Apache SVN. Now in the whole package, Spark takes about 200 MB. Later when we upgrade to a higher version, the package can be bigger. Without spark, the package is around 100 MB.
>
> Since we don't have customization to Spark, we can provide a download-spark.sh script for the user to easily download. If the user already has that in local, he just needs to copy or create a symbolic link in Kylin folder.
>
> This will reduce 2/3 package size, and save time/bandwidth for distributing Kylin. If you see a problem this change may bring, please complain.
>
> --
> Best regards,
>
> Shaofeng Shi 史少锋
>
Re: [Discuss] Exclude Spark from Kylin's binary package
Posted by Billy Liu <bi...@apache.org>.
I like the idea to remove the built-in spark distribution. I think
Kylin could work well with all spark 2.1+
With Warm regards
Billy Liu
ShaoFeng Shi <sh...@apache.org> 于2018年9月18日周二 下午5:35写道：
>
> Hello,
>
> Today Kylin's binary package ships an Apache Spark binary, the total package is 297 MB in v2.5.0; I noticed that if the package is above 300 MB, it would be rejected by Apache SVN. Now in the whole package, Spark takes about 200 MB. Later when we upgrade to a higher version, the package can be bigger. Without spark, the package is around 100 MB.
>
> Since we don't have customization to Spark, we can provide a download-spark.sh script for the user to easily download. If the user already has that in local, he just needs to copy or create a symbolic link in Kylin folder.
>
> This will reduce 2/3 package size, and save time/bandwidth for distributing Kylin. If you see a problem this change may bring, please complain.
>
> --
> Best regards,
>
> Shaofeng Shi 史少锋
>