You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ranger.apache.org by "Zhou Yifan (Jira)" <ji...@apache.org> on 2022/06/14 02:26:00 UTC
[jira] [Created] (RANGER-3787) Non-daemon threads started by ElasticSearchAuditDestination cause Spark application hanging
Zhou Yifan created RANGER-3787:
----------------------------------
Summary: Non-daemon threads started by ElasticSearchAuditDestination cause Spark application hanging
Key: RANGER-3787
URL: https://issues.apache.org/jira/browse/RANGER-3787
Project: Ranger
Issue Type: Improvement
Components: audit
Affects Versions: 3.0.0
Reporter: Zhou Yifan
When using kyuubi-spark-authz plugin (which extends RangerBasePlugin) in Spark application to save audit log to ElasticSearch, I found that if Spark application was submitted in local or client mode, it hanged forever even after main thread exited.
Here is my `ranger-spark-audit.xml`:
{code:java}
<configuration>
<property>
<name>xasecure.audit.destination.elasticsearch</name>
<value>enabled</value>
</property> <property>
<name>xasecure.audit.destination.elasticsearch.urls</name>
<value>es-master-1,es-master-2,es-master-3</value>
</property> <property>
<name>xasecure.audit.destination.elasticsearch.port</name>
<value>9200</value>
</property>
</configuration> {code}
In `jstack` output, I found 2 kinds of non-daemon threads:
{code:java}
"I/O dispatcher 1" #64 prio=5 os_prio=31 tid=0x00007f8717d89800 nid=0x14303 runnable [0x00007000054bc000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198)
at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x00000007808a7b58> (a sun.nio.ch.Util$3)
- locked <0x00000007808a7b48> (a java.util.Collections$UnmodifiableSet)
- locked <0x000000078088b318> (a sun.nio.ch.KQueueSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:255)
at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:591)
at java.lang.Thread.run(Thread.java:750)"pool-22-thread-1" #63 prio=5 os_prio=31 tid=0x00007f86c77d1800 nid=0xb503 runnable [0x00007000053b9000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198)
at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x0000000780832dc0> (a sun.nio.ch.Util$3)
- locked <0x0000000780832db0> (a java.util.Collections$UnmodifiableSet)
- locked <0x0000000780832b60> (a sun.nio.ch.KQueueSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute(AbstractMultiworkerIOReactor.java:343)
at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.execute(PoolingNHttpClientConnectionManager.java:221)
at org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase$1.run(CloseableHttpAsyncClientBase.java:64)
at java.lang.Thread.run(Thread.java:750) {code}
If I turn off `xasecure.audit.destination.elasticsearch`, Spark application exits normally.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)