You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by "Cheng, Hao" <ha...@intel.com> on 2014/10/28 05:54:45 UTC
Support Hive 0.13 .1 in Spark SQL
Hi, all
I have some PRs blocked by hive upgrading (e.g. https://github.com/apache/spark/pull/2570), the problem is some internal hive method signature changed, it's hard to make the compatible in code level (sql/hive) when switching back/forth the Hive versions.
I guess the motivation of the upgrading is to support the Metastore with different Hive versions. So, how about just keep the metastore related hive jars upgrading or utilize the HCatalog directly? And of course we can either leaving hive-exec.jar hive-cli.jar etc as 0.12 or upgrade to 0.13.1, but not support them both.
Sorry if I missed some discussion of Hive upgrading.
Cheng Hao
Re: Support Hive 0.13 .1 in Spark SQL
Posted by Patrick Wendell <pw...@gmail.com>.
Hey Cheng,
Right now we aren't using stable API's to communicate with the Hive
Metastore. We didn't want to drop support for Hive 0.12 so right now
we are using a shim layer to support compiling for 0.12 and 0.13. This
is very costly to maintain.
If Hive has a stable meta-data API for talking to a Metastore, we
should use that (is HCatalog sufficient for this purpose?). Ideally we
would be able to talk to multiple versions of the Hive metastore and
we can keep a single internal version of Hive for our use of Serde's,
etc.
I've created SPARK-4114 for this:
https://issues.apache.org/jira/browse/SPARK-4114
This is a very important issue for Spark SQL, so I'd welcome comments
on that JIRA from anyone who is familiar with Hive/HCatalog internals.
- Patrick
On Mon, Oct 27, 2014 at 9:54 PM, Cheng, Hao <ha...@intel.com> wrote:
> Hi, all
>
> I have some PRs blocked by hive upgrading (e.g.
> https://github.com/apache/spark/pull/2570), the problem is some internal
> hive method signature changed, it's hard to make the compatible in code
> level (sql/hive) when switching back/forth the Hive versions.
>
>
>
> I guess the motivation of the upgrading is to support the Metastore with
> different Hive versions. So, how about just keep the metastore related hive
> jars upgrading or utilize the HCatalog directly? And of course we can either
> leaving hive-exec.jar hive-cli.jar etc as 0.12 or upgrade to 0.13.1, but not
> support them both.
>
>
>
> Sorry if I missed some discussion of Hive upgrading.
>
>
>
> Cheng Hao
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org
Re: Support Hive 0.13 .1 in Spark SQL
Posted by Patrick Wendell <pw...@gmail.com>.
Hey Cheng,
Right now we aren't using stable API's to communicate with the Hive
Metastore. We didn't want to drop support for Hive 0.12 so right now
we are using a shim layer to support compiling for 0.12 and 0.13. This
is very costly to maintain.
If Hive has a stable meta-data API for talking to a Metastore, we
should use that (is HCatalog sufficient for this purpose?). Ideally we
would be able to talk to multiple versions of the Hive metastore and
we can keep a single internal version of Hive for our use of Serde's,
etc.
I've created SPARK-4114 for this:
https://issues.apache.org/jira/browse/SPARK-4114
This is a very important issue for Spark SQL, so I'd welcome comments
on that JIRA from anyone who is familiar with Hive/HCatalog internals.
- Patrick
On Mon, Oct 27, 2014 at 9:54 PM, Cheng, Hao <ha...@intel.com> wrote:
> Hi, all
>
> I have some PRs blocked by hive upgrading (e.g.
> https://github.com/apache/spark/pull/2570), the problem is some internal
> hive method signature changed, it's hard to make the compatible in code
> level (sql/hive) when switching back/forth the Hive versions.
>
>
>
> I guess the motivation of the upgrading is to support the Metastore with
> different Hive versions. So, how about just keep the metastore related hive
> jars upgrading or utilize the HCatalog directly? And of course we can either
> leaving hive-exec.jar hive-cli.jar etc as 0.12 or upgrade to 0.13.1, but not
> support them both.
>
>
>
> Sorry if I missed some discussion of Hive upgrading.
>
>
>
> Cheng Hao
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org