You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ranger.apache.org by "Zhou Yifan (Jira)" <ji...@apache.org> on 2022/06/14 02:39:00 UTC
[jira] [Updated] (RANGER-3787) Non-daemon threads started by ElasticSearchAuditDestination cause Spark application hanging
[ https://issues.apache.org/jira/browse/RANGER-3787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Zhou Yifan updated RANGER-3787:
-------------------------------
Description:
When using [kyuubi-spark-authz plugin|https://github.com/apache/incubator-kyuubi/tree/master/extensions/spark/kyuubi-spark-authz] (which extends RangerBasePlugin) in Spark application to save audit log to ElasticSearch, I found that if Spark application was submitted in local or client mode, it hanged forever even after main thread exited.
Here is my `ranger-spark-audit.xml`:
{code:java}
<configuration>
<property>
<name>xasecure.audit.destination.elasticsearch</name>
<value>enabled</value>
</property> <property>
<name>xasecure.audit.destination.elasticsearch.urls</name>
<value>es-master-1,es-master-2,es-master-3</value>
</property> <property>
<name>xasecure.audit.destination.elasticsearch.port</name>
<value>9200</value>
</property>
</configuration> {code}
In `jstack` output, I found 2 kinds of non-daemon threads:
{code:java}
"I/O dispatcher 1" #64 prio=5 os_prio=31 tid=0x00007f8717d89800 nid=0x14303 runnable [0x00007000054bc000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198)
at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x00000007808a7b58> (a sun.nio.ch.Util$3)
- locked <0x00000007808a7b48> (a java.util.Collections$UnmodifiableSet)
- locked <0x000000078088b318> (a sun.nio.ch.KQueueSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:255)
at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:591)
at java.lang.Thread.run(Thread.java:750)"pool-22-thread-1" #63 prio=5 os_prio=31 tid=0x00007f86c77d1800 nid=0xb503 runnable [0x00007000053b9000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198)
at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x0000000780832dc0> (a sun.nio.ch.Util$3)
- locked <0x0000000780832db0> (a java.util.Collections$UnmodifiableSet)
- locked <0x0000000780832b60> (a sun.nio.ch.KQueueSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute(AbstractMultiworkerIOReactor.java:343)
at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.execute(PoolingNHttpClientConnectionManager.java:221)
at org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase$1.run(CloseableHttpAsyncClientBase.java:64)
at java.lang.Thread.run(Thread.java:750) {code}
If I turn off `xasecure.audit.destination.elasticsearch`, Spark application exits normally.
was:
When using kyuubi-spark-authz plugin (which extends RangerBasePlugin) in Spark application to save audit log to ElasticSearch, I found that if Spark application was submitted in local or client mode, it hanged forever even after main thread exited.
Here is my `ranger-spark-audit.xml`:
{code:java}
<configuration>
<property>
<name>xasecure.audit.destination.elasticsearch</name>
<value>enabled</value>
</property> <property>
<name>xasecure.audit.destination.elasticsearch.urls</name>
<value>es-master-1,es-master-2,es-master-3</value>
</property> <property>
<name>xasecure.audit.destination.elasticsearch.port</name>
<value>9200</value>
</property>
</configuration> {code}
In `jstack` output, I found 2 kinds of non-daemon threads:
{code:java}
"I/O dispatcher 1" #64 prio=5 os_prio=31 tid=0x00007f8717d89800 nid=0x14303 runnable [0x00007000054bc000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198)
at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x00000007808a7b58> (a sun.nio.ch.Util$3)
- locked <0x00000007808a7b48> (a java.util.Collections$UnmodifiableSet)
- locked <0x000000078088b318> (a sun.nio.ch.KQueueSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:255)
at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:591)
at java.lang.Thread.run(Thread.java:750)"pool-22-thread-1" #63 prio=5 os_prio=31 tid=0x00007f86c77d1800 nid=0xb503 runnable [0x00007000053b9000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198)
at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x0000000780832dc0> (a sun.nio.ch.Util$3)
- locked <0x0000000780832db0> (a java.util.Collections$UnmodifiableSet)
- locked <0x0000000780832b60> (a sun.nio.ch.KQueueSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute(AbstractMultiworkerIOReactor.java:343)
at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.execute(PoolingNHttpClientConnectionManager.java:221)
at org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase$1.run(CloseableHttpAsyncClientBase.java:64)
at java.lang.Thread.run(Thread.java:750) {code}
If I turn off `xasecure.audit.destination.elasticsearch`, Spark application exits normally.
> Non-daemon threads started by ElasticSearchAuditDestination cause Spark application hanging
> -------------------------------------------------------------------------------------------
>
> Key: RANGER-3787
> URL: https://issues.apache.org/jira/browse/RANGER-3787
> Project: Ranger
> Issue Type: Improvement
> Components: audit
> Affects Versions: 3.0.0
> Reporter: Zhou Yifan
> Priority: Major
>
> When using [kyuubi-spark-authz plugin|https://github.com/apache/incubator-kyuubi/tree/master/extensions/spark/kyuubi-spark-authz] (which extends RangerBasePlugin) in Spark application to save audit log to ElasticSearch, I found that if Spark application was submitted in local or client mode, it hanged forever even after main thread exited.
> Here is my `ranger-spark-audit.xml`:
>
> {code:java}
> <configuration>
> <property>
> <name>xasecure.audit.destination.elasticsearch</name>
> <value>enabled</value>
> </property> <property>
> <name>xasecure.audit.destination.elasticsearch.urls</name>
> <value>es-master-1,es-master-2,es-master-3</value>
> </property> <property>
> <name>xasecure.audit.destination.elasticsearch.port</name>
> <value>9200</value>
> </property>
> </configuration> {code}
> In `jstack` output, I found 2 kinds of non-daemon threads:
> {code:java}
> "I/O dispatcher 1" #64 prio=5 os_prio=31 tid=0x00007f8717d89800 nid=0x14303 runnable [0x00007000054bc000]
> java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
> at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198)
> at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x00000007808a7b58> (a sun.nio.ch.Util$3)
> - locked <0x00000007808a7b48> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000078088b318> (a sun.nio.ch.KQueueSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:255)
> at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
> at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:591)
> at java.lang.Thread.run(Thread.java:750)"pool-22-thread-1" #63 prio=5 os_prio=31 tid=0x00007f86c77d1800 nid=0xb503 runnable [0x00007000053b9000]
> java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
> at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198)
> at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x0000000780832dc0> (a sun.nio.ch.Util$3)
> - locked <0x0000000780832db0> (a java.util.Collections$UnmodifiableSet)
> - locked <0x0000000780832b60> (a sun.nio.ch.KQueueSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute(AbstractMultiworkerIOReactor.java:343)
> at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.execute(PoolingNHttpClientConnectionManager.java:221)
> at org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase$1.run(CloseableHttpAsyncClientBase.java:64)
> at java.lang.Thread.run(Thread.java:750) {code}
> If I turn off `xasecure.audit.destination.elasticsearch`, Spark application exits normally.
>
>
--
This message was sent by Atlassian Jira
(v8.20.7#820007)