You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@sentry.apache.org by "Alexander Kolbasov (JIRA)" <ji...@apache.org> on 2017/06/04 07:58:04 UTC
[jira] [Updated] (SENTRY-1780) FullUpdateInitializer does not kill
the threads whenever getFullHMSSnapshot throws an exception
[ https://issues.apache.org/jira/browse/SENTRY-1780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Alexander Kolbasov updated SENTRY-1780:
---------------------------------------
Status: Patch Available (was: In Progress)
> FullUpdateInitializer does not kill the threads whenever getFullHMSSnapshot throws an exception
> -----------------------------------------------------------------------------------------------
>
> Key: SENTRY-1780
> URL: https://issues.apache.org/jira/browse/SENTRY-1780
> Project: Sentry
> Issue Type: Sub-task
> Components: Sentry
> Affects Versions: sentry-ha-redesign
> Reporter: Sergio Peña
> Assignee: Alexander Kolbasov
> Attachments: SENTRY-1780.01-sentry-ha-redesign.patch
>
>
> Errors may happen inside the FullUpdateInitializer.getFullHMSSnapshot() causing to throw an exception. When
> this happens, the current threads running are not killed and continue consuming resources while they're alive.
> We should kill those threads. The information collected by them won't be used anymore.
> I noticed this issue on the logs:
> {noformat}
> ### HMSFollower 1st run ######
> 2017-05-23 13:57:56,976 (pool-5-thread-1) [DEBUG - org.apache.sentry.service.thrift.HMSFollower.run(HMSFollower.java:208)] HMSFollower run is called.
> ### Starting to get a snapshot (2 databases found (db2, default) and submitted for execution) ###
> 2017-05-23 13:57:56,978 (pool-5-thread-1) [DEBUG - org.apache.sentry.hdfs.FullUpdateInitializer.getFullHMSSnapshot(FullUpdateInitializer.java:396)] get full snapshot
> 2017-05-23 13:57:56,979 (pool-5-thread-1) [DEBUG - org.apache.sentry.hdfs.FullUpdateInitializer.getFullHMSSnapshot(FullUpdateInitializer.java:401)] submit database: db2
> 2017-05-23 13:57:56,979 (pool-5-thread-1) [DEBUG - org.apache.sentry.hdfs.FullUpdateInitializer.getFullHMSSnapshot(FullUpdateInitializer.java:401)] submit database: default
> ### Thread 1 started: db2 ###
> 2017-05-23 13:57:56,979 (pool-150-thread-1) [DEBUG - org.apache.sentry.hdfs.FullUpdateInitializer$DbTask.doTask(FullUpdateInitializer.java:346)] begin thread: db2
> ### Waiting for db2 thread-1 ###
> 2017-05-23 13:57:56,979 (pool-5-thread-1) [DEBUG - org.apache.sentry.hdfs.FullUpdateInitializer.getFullHMSSnapshot(FullUpdateInitializer.java:416)] Waiting for call java.util.concurrent.FutureTa$
> k@3b30d168
> ### Thread 2 started: default ###
> 2017-05-23 13:57:56,979 (pool-150-thread-2) [DEBUG - org.apache.sentry.hdfs.FullUpdateInitializer$DbTask.doTask(FullUpdateInitializer.java:346)] begin thread: default
> ### Thread 1 failed ###
> 2017-05-23 13:57:57,007 (pool-150-thread-1) [DEBUG - org.apache.sentry.hdfs.FullUpdateInitializer$BaseTask$RetryStrategy.exec(FullUpdateInitializer.java:212)] Failed to execute task on 1 attempt$
> . Sleeping for 1000 ms. Exception: org.apache.thrift.TApplicationException: get_database failed: out of sequence response
> 2017-05-23 13:57:58,011 (pool-150-thread-1) [ERROR - org.apache.sentry.hdfs.FullUpdateInitializer$BaseTask$RetryStrategy.exec(FullUpdateInitializer.java:232)] Task did not complete successfully $
> fter 1 tries
> ### Thread 1 failed, and throwing an exception from getFullHMSSnapshot() ###
> 2017-05-23 13:57:58,011 (pool-5-thread-1) [DEBUG - org.apache.sentry.hdfs.FullUpdateInitializer.getFullHMSSnapshot(FullUpdateInitializer.java:418)] Call ended org.apache.sentry.hdfs.FullUpdateIn$
> tializer$CallResult@7cf642e8
> 2017-05-23 13:57:58,012 (pool-5-thread-1) [DEBUG - org.apache.sentry.hdfs.FullUpdateInitializer.getFullHMSSnapshot(FullUpdateInitializer.java:421)] throwing exception
> ### HMSFollower 2nd run ######
> 2017-05-23 13:57:59,012 (pool-5-thread-1) [DEBUG - org.apache.sentry.service.thrift.HMSFollower.run(HMSFollower.java:208)] HMSFollower run is called.
> ### Thread 2 from 1st HMSFollower finished with a failure ####
> 2017-05-23 13:58:14,722 (pool-150-thread-2) [DEBUG - org.apache.sentry.hdfs.FullUpdateInitializer$BaseTask$RetryStrategy.exec(FullUpdateInitializer.java:212)] Failed to execute task on 1 attempts
> . Sleeping for 1000 ms. Exception: org.apache.thrift.transport.TTransportException: java.net.SocketException: Socket closed
> 2017-05-23 13:58:14,725 (pool-150-thread-2) [ERROR - org.apache.sentry.hdfs.FullUpdateInitializer$BaseTask$RetryStrategy.exec(FullUpdateInitializer.java:232)] Task did not complete successfully a
> fter 2 tries
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)