You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ranger.apache.org by "Bhavik Patel (Jira)" <ji...@apache.org> on 2022/10/20 11:51:00 UTC
[jira] [Commented] (RANGER-3787) Non-daemon threads started by ElasticSearchAuditDestination cause Spark application hanging
[ https://issues.apache.org/jira/browse/RANGER-3787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17621043#comment-17621043 ]
Bhavik Patel commented on RANGER-3787:
--------------------------------------
[~zhouyifan279] As this is applicable for Range Spark Sql plugin, so let's wait for sometime to merge Spark Sql plugin Jira https://issues.apache.org/jira/browse/RANGER-2128
> Non-daemon threads started by ElasticSearchAuditDestination cause Spark application hanging
> -------------------------------------------------------------------------------------------
>
> Key: RANGER-3787
> URL: https://issues.apache.org/jira/browse/RANGER-3787
> Project: Ranger
> Issue Type: Improvement
> Components: audit
> Affects Versions: 3.0.0
> Reporter: Zhou Yifan
> Priority: Major
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> When using [kyuubi-spark-authz plugin|https://github.com/apache/incubator-kyuubi/tree/master/extensions/spark/kyuubi-spark-authz] (which extends RangerBasePlugin) to save audit log to ElasticSearch in Spark application, I found that if Spark application was submitted in local or client mode, it hanged forever even after main thread exited. But if I turn off `xasecure.audit.destination.elasticsearch`, Spark application exits normally.
>
> Here is my `ranger-spark-audit.xml`:
> {code:java}
> <configuration>
> <property>
> <name>xasecure.audit.destination.elasticsearch</name>
> <value>enabled</value>
> </property> <property>
> <name>xasecure.audit.destination.elasticsearch.urls</name>
> <value>es-master-1,es-master-2,es-master-3</value>
> </property> <property>
> <name>xasecure.audit.destination.elasticsearch.port</name>
> <value>9200</value>
> </property>
> </configuration> {code}
> In `jstack` output, I found 2 kinds of non-daemon threads started by ElasticSearch RestHighLevelClient:
> {code:java}
> "I/O dispatcher 1" #64 prio=5 os_prio=31 tid=0x00007f8717d89800 nid=0x14303 runnable [0x00007000054bc000]
> java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
> at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198)
> at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x00000007808a7b58> (a sun.nio.ch.Util$3)
> - locked <0x00000007808a7b48> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000078088b318> (a sun.nio.ch.KQueueSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:255)
> at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
> at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:591)
> at java.lang.Thread.run(Thread.java:750)"pool-22-thread-1" #63 prio=5 os_prio=31 tid=0x00007f86c77d1800 nid=0xb503 runnable [0x00007000053b9000]
> java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
> at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198)
> at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x0000000780832dc0> (a sun.nio.ch.Util$3)
> - locked <0x0000000780832db0> (a java.util.Collections$UnmodifiableSet)
> - locked <0x0000000780832b60> (a sun.nio.ch.KQueueSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute(AbstractMultiworkerIOReactor.java:343)
> at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.execute(PoolingNHttpClientConnectionManager.java:221)
> at org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase$1.run(CloseableHttpAsyncClientBase.java:64)
> at java.lang.Thread.run(Thread.java:750) {code}
>
> h3. Reproduce steps
> Build kyuubi-spark-authz plugin:
> {code:java}
> git clone git@github.com:zhouyifan279/incubator-kyuubi.git
> cd incubator-kyuubi
> ./build/mvn clean package -pl :kyuubi-spark-authz_2.12 -Pspark-3.1-ranger-3.0 -DskipTests {code}
> Setup plugin in Spark:
> # Dowload and untar spark distribution: [https://www.apache.org/dyn/closer.lua/spark/spark-3.1.3/spark-3.1.3-bin-hadoop3.2.tgz]
> # Put `incubator-kyuubi/extensions/spark/kyuubi-spark-authz/target/kyuubi-spark-authz_2.12-1.6.0-SNAPSHOT-1-with-ranger-3.0.jar` under `spark-3.1.3-bin-hadoop3.2/jars`
> # Put `ranger-spark-security.xml` and `ranger-spark-audit.xml` under `spark-3.1.3-bin-hadoop3.2/conf`
> Run Spark application:
> {code:java}
> ./spark-3.1.3-bin-hadoop3.2/bin/spark-submit --master local --class org.apache.spark.examples.sql.hive.SparkHiveExample --conf spark.sql.extensions=spark.sql.extensions org.apache.kyuubi.plugin.spark.authz.ranger.RangerSparkExtension examples/jars/spark-examples_2.12-3.1.3.jar{code}
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)