regarding Hive Thrift Metastore Server

[Shangzhong zhu <sh...@gmail.com>]

Hi all,

Our current setting of Hive is:

Hive Thrift server -> MySQL metastore.

All requests to MySQL metastore is going through the Thrift server.
However, we have been seeing issues with this setting. Thrift server once a
while gets stuck with TTransport timeout errors or even OOM.

Seems removing the Hive Thrift server, and let all clients directly hit
MySQL is a better option.

Just want to check with the community that, is everyone directly using
MySQL metastore without Thrift server, and is it the setting that Hive
developer team recommend.

Thanks,
Shanzhong

Re: regarding Hive Thrift Metastore Server

[Shangzhong zhu <sh...@gmail.com>]

We are using Hive 0.9.0.

Besides the security issues, any other issues you see when directly hitting
mysql?

We are frequently seeing Hive jobs failed (both CLI and oozie Hive jobs)
due to Meta Thrift server not responding. Any insight into this?

Thanks,
Shanzhong


On Wed, Jul 24, 2013 at 2:23 PM, Thejas Nair <th...@hortonworks.com> wrote:

> Without a metastore server, you would need to make access to the mysql
> db possible from all hive clients. This is inherently less secure,
> because anybody who can run hive cli can modify the mysql db, and
> there is not metastore server doing the authorization checks.
> If you don't care about security, metastore server is not that
> necessary. In the hortonworks distribution (HDP), we have metastore
> server running by default, and that is the common use case.
>
> What version of hive are you using ? Try disabling the file system
> cache in hive-site.xml in metastore (fs.hdfs.impl.disable.cache=true
> and fs.file.impl.disable.cache=true)
>
> On Wed, Jul 24, 2013 at 1:56 PM, Shangzhong zhu <sh...@gmail.com>
> wrote:
> > Hi all,
> >
> > Our current setting of Hive is:
> >
> > Hive Thrift server -> MySQL metastore.
> >
> > All requests to MySQL metastore is going through the Thrift server.
> However,
> > we have been seeing issues with this setting. Thrift server once a while
> > gets stuck with TTransport timeout errors or even OOM.
> >
> > Seems removing the Hive Thrift server, and let all clients directly hit
> > MySQL is a better option.
> >
> > Just want to check with the community that, is everyone directly using
> MySQL
> > metastore without Thrift server, and is it the setting that Hive
> developer
> > team recommend.
> >
> > Thanks,
> > Shanzhong
>

Re: regarding Hive Thrift Metastore Server

[Thejas Nair <th...@hortonworks.com>]

Without a metastore server, you would need to make access to the mysql
db possible from all hive clients. This is inherently less secure,
because anybody who can run hive cli can modify the mysql db, and
there is not metastore server doing the authorization checks.
If you don't care about security, metastore server is not that
necessary. In the hortonworks distribution (HDP), we have metastore
server running by default, and that is the common use case.

What version of hive are you using ? Try disabling the file system
cache in hive-site.xml in metastore (fs.hdfs.impl.disable.cache=true
and fs.file.impl.disable.cache=true)

On Wed, Jul 24, 2013 at 1:56 PM, Shangzhong zhu <sh...@gmail.com> wrote:
> Hi all,
>
> Our current setting of Hive is:
>
> Hive Thrift server -> MySQL metastore.
>
> All requests to MySQL metastore is going through the Thrift server. However,
> we have been seeing issues with this setting. Thrift server once a while
> gets stuck with TTransport timeout errors or even OOM.
>
> Seems removing the Hive Thrift server, and let all clients directly hit
> MySQL is a better option.
>
> Just want to check with the community that, is everyone directly using MySQL
> metastore without Thrift server, and is it the setting that Hive developer
> team recommend.
>
> Thanks,
> Shanzhong