You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@kyuubi.apache.org by "mahalingamagesthian (via GitHub)" <gi...@apache.org> on 2025/03/24 14:17:14 UTC
[D] Feedback on KYUUBI #6212 & #6233 - We faced a similar issue, Fix is found to be not working [kyuubi]
GitHub user mahalingamagesthian created a discussion: Feedback on KYUUBI #6212 & #6233 - We faced a similar issue, Fix is found to be not working
Hi @amanraj2520 , @pan3793
I wish that you are doing really well. We have a scenario to run spark-submit with kyuubi-spark-authz-shaded jar. We have figured out that the Sparkdriver kept running after the completion of the spark programme execution. We are seeing nondaemon threads similar to #6212 & #6233 .
Inspired by this, we are very quick to download kyuubi-spark-authz-shaded-2.12.1.10.1.jar. We don't find it fixing the issue. The sparkdriver logs doesn't have a mention of the Logs in line
https://github.com/amanraj2520/kyuubi/blob/auditShutdown/extensions/spark/kyuubi-spark-authz/src/main/scala/org/apache/kyuubi/plugin/spark/authz/ranger/SparkRangerAdminPlugin.scala#L79
I doubt if the fix is working or Am I missing anything ? I am happy to provide additional information from the test scenario.
Spark-submit is a batch on K8s. Spark submit connects to Apache Ranger download the policy evaluate the policy. kyuubi-spark-authz-shaded.jar is a key module to perform this operation.
JSTACK thread:
`"zkConnectionManagerCallback-5-thread-1" #113 prio=5 os_prio=0 cpu=0.90ms elapsed=26.68s tid=0x00007f1de0004000 nid=0xbd waiting on condition [0x00007f1dfa1d6000]
java.lang.Thread.State: WAITING (parking)
at jdk.internal.misc.Unsafe.park(java.base@11.0.26/Native Method)
- parking to wait for <0x00000007111bdff8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(java.base@11.0.26/LockSupport.java:194)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@11.0.26/AbstractQueuedSynchronizer.java:2081)
at java.util.concurrent.LinkedBlockingQueue.take(java.base@11.0.26/LinkedBlockingQueue.java:433)
at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@11.0.26/ThreadPoolExecutor.java:1054)
at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.26/ThreadPoolExecutor.java:1114)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.26/ThreadPoolExecutor.java:628)
at java.lang.Thread.run(java.base@11.0.26/Thread.java:829)`
GitHub link: https://github.com/apache/kyuubi/discussions/6996
----
This is an automatically sent email for notifications@kyuubi.apache.org.
To unsubscribe, please send an email to: notifications-unsubscribe@kyuubi.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [D] Feedback on KYUUBI #6212 & #6233 - We faced a similar issue, Fix is found to be not working [kyuubi]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
GitHub user pan3793 added a comment to the discussion: Feedback on KYUUBI #6212 & #6233 - We faced a similar issue, Fix is found to be not working
It should fix the issue. (I don't use Solr, but I get at least 3 positive reports for this case from Kyuubi users)
GitHub link: https://github.com/apache/kyuubi/discussions/6996#discussioncomment-12609632
----
This is an automatically sent email for notifications@kyuubi.apache.org.
To unsubscribe, please send an email to: notifications-unsubscribe@kyuubi.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [D] Feedback on KYUUBI #6212 & #6233 - We faced a similar issue, Fix is found to be not working [kyuubi]
Posted by "mahalingamagesthian (via GitHub)" <gi...@apache.org>.
GitHub user mahalingamagesthian added a comment to the discussion: Feedback on KYUUBI #6212 & #6233 - We faced a similar issue, Fix is found to be not working
Thanks @pan3793 .
It is true that if we disable ranger solr audit. Spark driver pod is completing. This issue is not happening.
Does #6233 fixes this issue ? without making the solr threads as daemon threads ?
GitHub link: https://github.com/apache/kyuubi/discussions/6996#discussioncomment-12609529
----
This is an automatically sent email for notifications@kyuubi.apache.org.
To unsubscribe, please send an email to: notifications-unsubscribe@kyuubi.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [D] Feedback on KYUUBI #6212 & #6233 - We faced a similar issue, Fix is found to be not working [kyuubi]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
GitHub user pan3793 added a comment to the discussion: Feedback on KYUUBI #6212 & #6233 - We faced a similar issue, Fix is found to be not working
I think this is exactly what is explained in https://github.com/apache/kyuubi/issues/6212#issuecomment-2022040614, to quickly verify it, try disabling ranger solr audit plugin first.
GitHub link: https://github.com/apache/kyuubi/discussions/6996#discussioncomment-12609339
----
This is an automatically sent email for notifications@kyuubi.apache.org.
To unsubscribe, please send an email to: notifications-unsubscribe@kyuubi.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [D] Feedback on KYUUBI #6212 & #6233 - We faced a similar issue, Fix is found to be not working [kyuubi]
Posted by "mahalingamagesthian (via GitHub)" <gi...@apache.org>.
GitHub user mahalingamagesthian edited a discussion: Feedback on KYUUBI #6212 & #6233 - We faced a similar issue, Fix is found to be not working
Hi @amanraj2520 , @pan3793
I wish that you are doing really well. We have a scenario to run spark-submit with kyuubi-spark-authz-shaded jar. We have figured out that the Sparkdriver kept running after the completion of the spark programme execution. We are seeing nondaemon threads similar to #6212 & #6233 .
Inspired by this, we are very quick to download kyuubi-spark-authz-shaded-2.12.1.10.1.jar. We don't find it fixing the issue. The sparkdriver logs doesn't have a mention of the Logs in line
https://github.com/amanraj2520/kyuubi/blob/auditShutdown/extensions/spark/kyuubi-spark-authz/src/main/scala/org/apache/kyuubi/plugin/spark/authz/ranger/SparkRangerAdminPlugin.scala#L79
I doubt if the fix is working or Am I missing anything ? I am happy to provide additional information from the test scenario.
Spark-submit is a batch on K8s. Spark submit connects to Apache Ranger download the policy evaluate the policy. kyuubi-spark-authz-shaded.jar is a key module to perform this operation.
JSTACK thread:
```
"zkConnectionManagerCallback-5-thread-1" #113 prio=5 os_prio=0 cpu=0.90ms elapsed=26.68s tid=0x00007f1de0004000 nid=0xbd waiting on condition [0x00007f1dfa1d6000]
java.lang.Thread.State: WAITING (parking)
at jdk.internal.misc.Unsafe.park(java.base@11.0.26/Native Method)
- parking to wait for <0x00000007111bdff8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(java.base@11.0.26/LockSupport.java:194)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@11.0.26/AbstractQueuedSynchronizer.java:2081)
at java.util.concurrent.LinkedBlockingQueue.take(java.base@11.0.26/LinkedBlockingQueue.java:433)
at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@11.0.26/ThreadPoolExecutor.java:1054)
at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.26/ThreadPoolExecutor.java:1114)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.26/ThreadPoolExecutor.java:628)
at java.lang.Thread.run(java.base@11.0.26/Thread.java:829)
```
GitHub link: https://github.com/apache/kyuubi/discussions/6996
----
This is an automatically sent email for notifications@kyuubi.apache.org.
To unsubscribe, please send an email to: notifications-unsubscribe@kyuubi.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org