You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by Zoltan Haindrich <ki...@rxd.hu> on 2017/03/16 22:42:50 UTC
spark-without-hive assembly for hive build/development purposes
Hello,
Hive needs a spark assembly to execute the HoS tests.
Until now\u2026this assembly have been downloaded from an S3 bucket - because
this is not the best solution available, it sometimes causes troubles
and inconveniences...
We had a discussion about improving this; and the best option would be
to download spark-without-hive assembly from the maven repository...but
that opens up a few questions:
1) which project should publish it: Hive or Spark?
2) what should be the group-id? this artifact is only needed for
Hive...but it contains Spark code! :) so its in some kind of grey zone...
3) how will we be able to get a spark-without-hive artifact for 2.0.0 -
since that version is already released?
for more details:
https://issues.apache.org/jira/browse/HIVE-14735
What do you guys think?
cheers,
Zoltan
---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
Re: spark-without-hive assembly for hive build/development purposes
Posted by Zoltan Haindrich <ki...@rxd.hu>.
Hive uses spark-submit to launch the spark jobs - that's why the
assembly is needed.
On 03/16/2017 11:54 PM, Reynold Xin wrote:
> Why do you need an assembly? Is there something preventing Hive from
> depending on normal jars like all other applications?
>
> On Thu, Mar 16, 2017 at 3:42 PM, Zoltan Haindrich <kirk@rxd.hu
> <ma...@rxd.hu>> wrote:
>
> Hello,
>
> Hive needs a spark assembly to execute the HoS tests.
> Until now\u2026this assembly have been downloaded from an S3 bucket -
> because this is not the best solution available, it sometimes
> causes troubles and inconveniences...
>
> We had a discussion about improving this; and the best option
> would be to download spark-without-hive assembly from the maven
> repository...but that opens up a few questions:
>
> 1) which project should publish it: Hive or Spark?
> 2) what should be the group-id? this artifact is only needed for
> Hive...but it contains Spark code! :) so its in some kind of grey
> zone...
> 3) how will we be able to get a spark-without-hive artifact for
> 2.0.0 - since that version is already released?
>
> for more details:
> https://issues.apache.org/jira/browse/HIVE-14735
> <https://issues.apache.org/jira/browse/HIVE-14735>
>
> What do you guys think?
>
> cheers,
> Zoltan
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
> <ma...@spark.apache.org>
>
>
Re: spark-without-hive assembly for hive build/development purposes
Posted by Reynold Xin <rx...@databricks.com>.
Why do you need an assembly? Is there something preventing Hive from
depending on normal jars like all other applications?
On Thu, Mar 16, 2017 at 3:42 PM, Zoltan Haindrich <ki...@rxd.hu> wrote:
> Hello,
>
> Hive needs a spark assembly to execute the HoS tests.
> Until now…this assembly have been downloaded from an S3 bucket - because
> this is not the best solution available, it sometimes causes troubles and
> inconveniences...
>
> We had a discussion about improving this; and the best option would be to
> download spark-without-hive assembly from the maven repository...but that
> opens up a few questions:
>
> 1) which project should publish it: Hive or Spark?
> 2) what should be the group-id? this artifact is only needed for
> Hive...but it contains Spark code! :) so its in some kind of grey zone...
> 3) how will we be able to get a spark-without-hive artifact for 2.0.0 -
> since that version is already released?
>
> for more details:
> https://issues.apache.org/jira/browse/HIVE-14735
>
> What do you guys think?
>
> cheers,
> Zoltan
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>
>
Re: spark-without-hive assembly for hive build/development purposes
Posted by Marcelo Vanzin <va...@cloudera.com>.
The solution to this is being tracked in
https://issues.apache.org/jira/browse/HIVE-15302, although I haven't
seen activity in a while.
On Thu, Mar 16, 2017 at 3:42 PM, Zoltan Haindrich <ki...@rxd.hu> wrote:
> Hello,
>
> Hive needs a spark assembly to execute the HoS tests.
> Until now…this assembly have been downloaded from an S3 bucket - because
> this is not the best solution available, it sometimes causes troubles and
> inconveniences...
>
> We had a discussion about improving this; and the best option would be to
> download spark-without-hive assembly from the maven repository...but that
> opens up a few questions:
>
> 1) which project should publish it: Hive or Spark?
> 2) what should be the group-id? this artifact is only needed for Hive...but
> it contains Spark code! :) so its in some kind of grey zone...
> 3) how will we be able to get a spark-without-hive artifact for 2.0.0 -
> since that version is already released?
>
> for more details:
> https://issues.apache.org/jira/browse/HIVE-14735
>
> What do you guys think?
>
> cheers,
> Zoltan
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>
--
Marcelo
---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org