[jira] [Created] (SENTRY-1965) sentry server database database Destroy connection exception (Galera cluster)

["Steven Hancz (JIRA)" <ji...@apache.org>]

Steven Hancz created SENTRY-1965:
------------------------------------

             Summary: sentry server database database Destroy connection exception (Galera cluster)
                 Key: SENTRY-1965
                 URL: https://issues.apache.org/jira/browse/SENTRY-1965
             Project: Sentry
          Issue Type: Bug
          Components: Sentry
    Affects Versions: 1.5.1
         Environment: Sentry 1.5.1 
CDH 5.9.1
Mysql 5.6.35
Galera 25.18
            Reporter: Steven Hancz


We have implemented an HA solution for the Sentry server database.
Basically instead of using a single MySQL server we have a Galera cluster that is accessed via a DNS load balanced VIP. So that if one MySQL server stops working the VIP will detect and send the DB request to the surviving node. A similar set up is working for the HIVE metastore.
However we noticed that Sentry just like spark uses the BoneCP connection pool to connect to the database. There are some hard codded configuration options in the bonecp-default-config.xml that are causing issues with Sentry.
idleConnectionTestPeriodInMinutes  default 240 minutes
idleMaxAgeInMinutes                        default 60 minutes

Based on this BonceCP will test each idle connection every 240 minutes
but an idle connection is closed after 60 minutes (second parameter) so the connection testing will never take place as the connection will be closed after 60 minutes. The test takes place every 240 minutes.

However in an HA configuration with a VIP you can set the connection time out and how often to test for target availability. 

We had the exact same problem for hive there the work around was to include a second configuration file for BoneCP called bonecp-config.xml. This was added to the hive server jar.  The second config file (bonecp-config.xml) contains

idleConnectionTestPeriodInMinutes  1
idleMaxAgeInMinutes 5

So that every connection is tested every minute and an idle connection is closed after 5 minutes. But since we test it every minute they will be kept alive.

So the question is how to enable a similar setting for Sentry ?
With default boneCP configuration and Galera cluster in the back end Sentry is returning the following error:

Sep 26, 7:30:39.900 AM	ERROR	com.jolbox.bonecp.ConnectionTesterThread	
Destroy connection exception
com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException: Communications link failure during rollback(). Transaction resolution unknown.
	at sun.reflect.GeneratedConstructorAccessor41.newInstance(Unknown Source)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
	at com.mysql.jdbc.Util.handleNewInstance(Util.java:404)
	at com.mysql.jdbc.Util.getInstance(Util.java:387)
	at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:917)
	at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:896)
	at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:885)
	at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:860)
	at com.mysql.jdbc.ConnectionImpl.rollback(ConnectionImpl.java:4634)
	at com.mysql.jdbc.ConnectionImpl.realClose(ConnectionImpl.java:4263)
	at com.mysql.jdbc.ConnectionImpl.close(ConnectionImpl.java:1519)
	at com.jolbox.bonecp.ConnectionHandle.internalClose(ConnectionHandle.java:396)
	at com.jolbox.bonecp.ConnectionTesterThread.closeConnection(ConnectionTesterThread.java:155)
	at com.jolbox.bonecp.ConnectionTesterThread.run(ConnectionTesterThread.java:95)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)

From our research it appears that sentry server is using BoneCP class but in more than one location. Changing the parameters in BoneCP for Sentry alone is does not appear to be sufficient. Trace file shows that parameters are not changed and time outs are default boncecp parameter. Where else do we have to change boneCP config? 

Regards,




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)