hive external metastore connection timeout

[jamborta <ja...@gmail.com>]

Hi all,

I am setting up a spark standalone server with an external hive metastore
(using mysql), there is an issue that after 5 minutes inactivity, if I try
to reconnect to the metastore (i.e. by executing a new query), it hangs for
about 10 mins then times out. My guess is that datanucleus does not close
the existing connections from the pool, but still tries to create new ones
for some reason. 

Tried different type of connection pools, didn't help either.

thanks,





--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/hive-external-metastore-connection-timeout-tp23052.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org

Re: hive external metastore connection timeout

[Yana Kadiyska <ya...@gmail.com>]

I have not run into this particular issue but I'm not using latest bits in
production. However, testing your theory should be easy -- MySQL is just a
database, so you should be able to use a regular mysql client and see how
many connections are active. You can then compare to the maximum allowed
connections, and/or up the maximum allowed and see if you still hit the
ceiling. Hopefully someone will have a more direct answer for you...

also, not sure what query you're executing to reconnect, but try something
inexpensive like "show tables" -- I had connection timeout issues with the
actual query when pulling data...that was helped by setting
"hive.metastore.client.socket.timeout" higher...

On Wed, May 27, 2015 at 11:03 AM, jamborta <ja...@gmail.com> wrote:

> Hi all,
>
> I am setting up a spark standalone server with an external hive metastore
> (using mysql), there is an issue that after 5 minutes inactivity, if I try
> to reconnect to the metastore (i.e. by executing a new query), it hangs for
> about 10 mins then times out. My guess is that datanucleus does not close
> the existing connections from the pool, but still tries to create new ones
> for some reason.
>
> Tried different type of connection pools, didn't help either.
>
> thanks,
>
>
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/hive-external-metastore-connection-timeout-tp23052.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>