You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@sentry.apache.org by "kalyan kumar kalvagadda (JIRA)" <ji...@apache.org> on 2017/04/07 16:59:41 UTC
[jira] [Commented] (SENTRY-1629) Current MAuthzPathsMapping table
definition may cause error 'Duplicate entry XX for key PRIMARY'
[ https://issues.apache.org/jira/browse/SENTRY-1629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15961113#comment-15961113 ]
kalyan kumar kalvagadda commented on SENTRY-1629:
-------------------------------------------------
Submitting code changes to address this issue. Here is the snapshot of the changes
1. Change the JDO definition to have Path as separate entity.
2. SQL changes needed for the handle the new JDO definition.
3. application changes to use new JDO definition.
4. Updated the unit test cases to test the case where an authz object is associated with more than one path.
Testing done:
1. Tested the sql changes with derby, mysql, oracle, postgres and db2.
> Current MAuthzPathsMapping table definition may cause error 'Duplicate entry XX for key PRIMARY'
> ------------------------------------------------------------------------------------------------
>
> Key: SENTRY-1629
> URL: https://issues.apache.org/jira/browse/SENTRY-1629
> Project: Sentry
> Issue Type: Sub-task
> Components: Hdfs Plugin
> Affects Versions: sentry-ha-redesign
> Reporter: Hao Hao
> Assignee: kalyan kumar kalvagadda
> Priority: Critical
> Fix For: sentry-ha-redesign
>
>
> In MAuthzPathsMapping table defines 1-N mapping of authzObjName to HMSPaths. However, current JDO metadata/SQL definition may cause duplicate entry error.
> {noformat} 2017-02-13 07:03:06,909 ERROR org.apache.sentry.provider.db.service.persistent.TransactionManager: The transaction has reached max retry numbe, rAdd request failed : INSERT INTO `MAUTHZPATHSMAPPING_PATHS` (`AUTHZ_OBJ_ID_OID`,`PATHS`) VALUES (?,?)
> javax.jdo.JDODataStoreException: Add request failed : INSERT INTO `MAUTHZPATHSMAPPING_PATHS` (`AUTHZ_OBJ_ID_OID`,`PATHS`) VALUES (?,?)
> FailedObject:MSentryPathsUpdate [authzObj=default.web_logs], paths=[[user/admin/2015_11_21, user/admin/2015_11_20, user/hive/warehouse/web_logs, user/admin/2015_11_18, user/admin/2015_11_19]], createTimeMs=[1486969386906]
> at org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:449)
> at org.datanucleus.api.jdo.JDOPersistenceManager.jdoMakePersistent(JDOPersistenceManager.java:732)
> at org.datanucleus.api.jdo.JDOPersistenceManager.makePersistent(JDOPersistenceManager.java:752)
> at org.apache.sentry.provider.db.service.persistent.SentryStore.createAuthzPathsMappingCore(SentryStore.java:2200)
> at org.apache.sentry.provider.db.service.persistent.SentryStore.access$2500(SentryStore.java:88)
> at org.apache.sentry.provider.db.service.persistent.SentryStore$31.execute(SentryStore.java:2174)
> at org.apache.sentry.provider.db.service.persistent.TransactionManager.executeTransaction(TransactionManager.java:114)
> at org.apache.sentry.provider.db.service.persistent.TransactionManager.executeTransactionWithRetry(TransactionManager.java:183)
> at org.apache.sentry.provider.db.service.persistent.SentryStore.persistFullPathsImage(SentryStore.java:2170)
> at org.apache.sentry.service.thrift.HMSFollower.run(HMSFollower.java:263)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> NestedThrowablesStackTrace:
> org.datanucleus.store.rdbms.exceptions.MappedDatastoreException: INSERT INTO `MAUTHZPATHSMAPPING_PATHS` (`AUTHZ_OBJ_ID_OID`,`PATHS`) VALUES (?,?)
> at org.datanucleus.store.rdbms.scostore.JoinSetStore.doInternalAdd(JoinSetStore.java:754)
> at org.datanucleus.store.rdbms.scostore.JoinSetStore.internalAdd(JoinSetStore.java:502)
> at org.datanucleus.store.rdbms.scostore.JoinSetStore.addAll(JoinSetStore.java:433)
> at org.datanucleus.store.rdbms.mapping.java.CollectionMapping.postInsert(CollectionMapping.java:136)
> at org.datanucleus.store.rdbms.request.InsertRequest.execute(InsertRequest.java:519)
> at org.datanucleus.store.rdbms.RDBMSPersistenceHandler.insertTable(RDBMSPersistenceHandler.java:167)
> at org.datanucleus.store.rdbms.RDBMSPersistenceHandler.insertObject(RDBMSPersistenceHandler.java:143)
> at org.datanucleus.state.JDOStateManager.internalMakePersistent(JDOStateManager.java:3784)
> at org.datanucleus.state.JDOStateManager.makePersistent(JDOStateManager.java:3760)
> at org.datanucleus.ExecutionContextImpl.persistObjectInternal(ExecutionContextImpl.java:2219)
> at org.datanucleus.ExecutionContextImpl.persistObjectWork(ExecutionContextImpl.java:2065)
> at org.datanucleus.ExecutionContextImpl.persistObject(ExecutionContextImpl.java:1913)
> at org.datanucleus.ExecutionContextThreadedImpl.persistObject(ExecutionContextThreadedImpl.java:217)
> at org.datanucleus.api.jdo.JDOPersistenceManager.jdoMakePersistent(JDOPersistenceManager.java:727)
> at org.datanucleus.api.jdo.JDOPersistenceManager.makePersistent(JDOPersistenceManager.java:752)
> at org.apache.sentry.provider.db.service.persistent.SentryStore.createAuthzPathsMappingCore(SentryStore.java:2200)
> at org.apache.sentry.provider.db.service.persistent.SentryStore.access$2500(SentryStore.java:88)
> at org.apache.sentry.provider.db.service.persistent.SentryStore$31.execute(SentryStore.java:2174)
> at org.apache.sentry.provider.db.service.persistent.TransactionManager.executeTransaction(TransactionManager.java:114)
> at org.apache.sentry.provider.db.service.persistent.TransactionManager.executeTransactionWithRetry(TransactionManager.java:183)
> at org.apache.sentry.provider.db.service.persistent.SentryStore.persistFullPathsImage(SentryStore.java:2170)
> at org.apache.sentry.service.thrift.HMSFollower.run(HMSFollower.java:263)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.sql.BatchUpdateException: Duplicate entry '102' for key 'PRIMARY'
> at com.mysql.jdbc.PreparedStatement.executeBatchSerially(PreparedStatement.java:1815)
> at com.mysql.jdbc.PreparedStatement.executeBatch(PreparedStatement.java:1277)
> at com.jolbox.bonecp.StatementHandle.executeBatch(StatementHandle.java:469)
> at org.datanucleus.store.rdbms.ParamLoggingPreparedStatement.executeBatch(ParamLoggingPreparedStatement.java:372)
> at org.datanucleus.store.rdbms.SQLController.processConnectionStatement(SQLController.java:683)
> at org.datanucleus.store.rdbms.SQLController.executeStatementUpdate(SQLController.java:409)
> at org.datanucleus.store.rdbms.scostore.JoinSetStore.doInternalAdd(JoinSetStore.java:733)
> ... 28 more
> Caused by: com.mysql.jdbc.exceptions.jdbc4.MySQLIntegrityConstraintViolationException: Duplicate entry '102' for key 'PRIMARY'
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
> at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
> at com.mysql.jdbc.Util.handleNewInstance(Util.java:377)
> at com.mysql.jdbc.Util.getInstance(Util.java:360)
> at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:971)
> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3887)
> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3823)
> at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2435)
> at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2582)
> at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2530)
> at com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:1907)
> at com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2141)
> at com.mysql.jdbc.PreparedStatement.executeBatchSerially(PreparedStatement.java:1773)
> ... 34 more
> {noformat}
> The possible fix could be either changing JDO metadata definition of 1-N mapping with adding a new HMSPaths table. Or change the SQL scripts.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)