You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@gluten.apache.org by "beliefer (via GitHub)" <gi...@apache.org> on 2025/09/15 12:36:54 UTC
[PR] [GLUTEN-10707] Improve SparkDirectoryUtil with AtomicReference [incubator-gluten]
beliefer opened a new pull request, #10708:
URL: https://github.com/apache/incubator-gluten/pull/10708
## What changes are proposed in this pull request?
This PR proposes to improve `SparkDirectoryUtil` with `AtomicReference`
Fixes #10707
## How was this patch tested?
GA tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-10707] Improve SparkDirectoryUtil by reduce synchronized [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #10708:
URL: https://github.com/apache/incubator-gluten/pull/10708#issuecomment-3392800530
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-10707] Improve SparkDirectoryUtil by reduce synchronized [incubator-gluten]
Posted by "zhztheplayer (via GitHub)" <gi...@apache.org>.
zhztheplayer commented on code in PR #10708:
URL: https://github.com/apache/incubator-gluten/pull/10708#discussion_r2422098071
##########
gluten-core/src/main/scala/org/apache/spark/util/SparkDirectoryUtil.scala:
##########
@@ -67,30 +67,31 @@ class SparkDirectoryUtil private (val roots: Array[String]) extends Logging {
}
object SparkDirectoryUtil extends Logging {
- private var INSTANCE: SparkDirectoryUtil = _
+ @volatile private var targetRoots: Array[String] = _
+ private lazy val INSTANCE: SparkDirectoryUtil = {
+ if (targetRoots == null) {
+ throw new IllegalStateException("SparkDirectoryUtil not initialized")
+ }
+ new SparkDirectoryUtil(targetRoots)
+ }
- def init(conf: SparkConf): Unit = synchronized {
+ def init(conf: SparkConf): Unit = {
val roots = Utils.getConfiguredLocalDirs(conf)
init(roots)
}
private def init(roots: Array[String]): Unit = synchronized {
- if (INSTANCE == null) {
- INSTANCE = new SparkDirectoryUtil(roots)
- return
- }
- if (INSTANCE.roots.toSet != roots.toSet) {
+ if (targetRoots == null) {
+ targetRoots = roots
Review Comment:
Perhaps just use the same name and
```scala
this.roots = roots
```
. Just a nit. Both are fine.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-10707] Improve SparkDirectoryUtil with AtomicReference [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #10708:
URL: https://github.com/apache/incubator-gluten/pull/10708#issuecomment-3291973108
https://github.com/apache/incubator-gluten/issues/10707
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-10707] Improve SparkDirectoryUtil with AtomicBoolean [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #10708:
URL: https://github.com/apache/incubator-gluten/pull/10708#issuecomment-3297272876
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-10707] Improve SparkDirectoryUtil with AtomicBoolean [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #10708:
URL: https://github.com/apache/incubator-gluten/pull/10708#issuecomment-3301014703
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-10707] Improve SparkDirectoryUtil with AtomicBoolean [incubator-gluten]
Posted by "zhztheplayer (via GitHub)" <gi...@apache.org>.
zhztheplayer commented on code in PR #10708:
URL: https://github.com/apache/incubator-gluten/pull/10708#discussion_r2355594919
##########
gluten-core/src/main/scala/org/apache/spark/util/SparkDirectoryUtil.scala:
##########
@@ -67,28 +68,24 @@ class SparkDirectoryUtil private (val roots: Array[String]) extends Logging {
}
object SparkDirectoryUtil extends Logging {
+ private val INSTANCE_INITIALIZED = new AtomicBoolean(false)
private var INSTANCE: SparkDirectoryUtil = _
- def init(conf: SparkConf): Unit = synchronized {
+ def init(conf: SparkConf): Unit = {
val roots = Utils.getConfiguredLocalDirs(conf)
init(roots)
}
- private def init(roots: Array[String]): Unit = synchronized {
- if (INSTANCE == null) {
+ private def init(roots: Array[String]): Unit = {
+ if (INSTANCE_INITIALIZED.compareAndSet(false, true)) {
INSTANCE = new SparkDirectoryUtil(roots)
- return
- }
- if (INSTANCE.roots.toSet != roots.toSet) {
- throw new IllegalArgumentException(
- s"Reinitialize SparkDirectoryUtil with different root dirs: old: ${INSTANCE.ROOTS
- .mkString("Array(", ", ", ")")}, new: ${roots.mkString("Array(", ", ", ")")}"
- )
+ } else {
+ logWarning("SparkDirectoryUtil is already initialized")
}
}
- def get(): SparkDirectoryUtil = synchronized {
- assert(INSTANCE != null, "Default instance of SparkDirectoryUtil was not set yet")
+ def get(): SparkDirectoryUtil = {
+ assert(INSTANCE_INITIALIZED.get(), "Default instance of SparkDirectoryUtil was not set yet")
INSTANCE
}
Review Comment:
Hi @beliefer, thank you for keeping iterating the code, but the it is still problematic :(
When thread 1 reaches line 81 but hasn't yet set `INSTANCE`, thread 2 can pass line 88 and access `INSTANCE` which may give an unexpected result to caller.
I know it's a corner case, but we should make sure the new code covers what is covered by the old code completely.
##########
gluten-core/src/main/scala/org/apache/spark/util/SparkDirectoryUtil.scala:
##########
@@ -67,28 +68,24 @@ class SparkDirectoryUtil private (val roots: Array[String]) extends Logging {
}
object SparkDirectoryUtil extends Logging {
+ private val INSTANCE_INITIALIZED = new AtomicBoolean(false)
private var INSTANCE: SparkDirectoryUtil = _
- def init(conf: SparkConf): Unit = synchronized {
+ def init(conf: SparkConf): Unit = {
val roots = Utils.getConfiguredLocalDirs(conf)
init(roots)
}
- private def init(roots: Array[String]): Unit = synchronized {
- if (INSTANCE == null) {
+ private def init(roots: Array[String]): Unit = {
+ if (INSTANCE_INITIALIZED.compareAndSet(false, true)) {
INSTANCE = new SparkDirectoryUtil(roots)
- return
- }
- if (INSTANCE.roots.toSet != roots.toSet) {
- throw new IllegalArgumentException(
- s"Reinitialize SparkDirectoryUtil with different root dirs: old: ${INSTANCE.ROOTS
- .mkString("Array(", ", ", ")")}, new: ${roots.mkString("Array(", ", ", ")")}"
- )
+ } else {
+ logWarning("SparkDirectoryUtil is already initialized")
}
}
- def get(): SparkDirectoryUtil = synchronized {
- assert(INSTANCE != null, "Default instance of SparkDirectoryUtil was not set yet")
+ def get(): SparkDirectoryUtil = {
+ assert(INSTANCE_INITIALIZED.get(), "Default instance of SparkDirectoryUtil was not set yet")
INSTANCE
}
Review Comment:
Hi @beliefer, thank you for keeping iterating the code, but it is still problematic :(
When thread 1 reaches line 81 but hasn't yet set `INSTANCE`, thread 2 can pass line 88 and access `INSTANCE` which may give an unexpected result to caller.
I know it's a corner case, but we should make sure the new code covers what is covered by the old code completely.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-10707] Improve SparkDirectoryUtil with AtomicBoolean [incubator-gluten]
Posted by "beliefer (via GitHub)" <gi...@apache.org>.
beliefer commented on code in PR #10708:
URL: https://github.com/apache/incubator-gluten/pull/10708#discussion_r2354112067
##########
gluten-core/src/main/scala/org/apache/spark/util/SparkDirectoryUtil.scala:
##########
@@ -67,16 +68,19 @@ class SparkDirectoryUtil private (val roots: Array[String]) extends Logging {
}
object SparkDirectoryUtil extends Logging {
+ private val state = new AtomicBoolean(false)
private var INSTANCE: SparkDirectoryUtil = _
- def init(conf: SparkConf): Unit = synchronized {
+ def init(conf: SparkConf): Unit = {
val roots = Utils.getConfiguredLocalDirs(conf)
init(roots)
}
- private def init(roots: Array[String]): Unit = synchronized {
+ private def init(roots: Array[String]): Unit = {
if (INSTANCE == null) {
- INSTANCE = new SparkDirectoryUtil(roots)
+ if (state.compareAndSet(false, true)) {
+ INSTANCE = new SparkDirectoryUtil(roots)
+ }
return
Review Comment:
Because remove `if (INSTANCE == null)` will cause NPE. So I removed `if (INSTANCE.roots.toSet != roots.toSet)`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-10707] Improve SparkDirectoryUtil with AtomicBoolean [incubator-gluten]
Posted by "beliefer (via GitHub)" <gi...@apache.org>.
beliefer commented on code in PR #10708:
URL: https://github.com/apache/incubator-gluten/pull/10708#discussion_r2357370238
##########
gluten-core/src/main/scala/org/apache/spark/util/SparkDirectoryUtil.scala:
##########
@@ -67,28 +68,24 @@ class SparkDirectoryUtil private (val roots: Array[String]) extends Logging {
}
object SparkDirectoryUtil extends Logging {
+ private val INSTANCE_INITIALIZED = new AtomicBoolean(false)
private var INSTANCE: SparkDirectoryUtil = _
- def init(conf: SparkConf): Unit = synchronized {
+ def init(conf: SparkConf): Unit = {
val roots = Utils.getConfiguredLocalDirs(conf)
init(roots)
}
- private def init(roots: Array[String]): Unit = synchronized {
- if (INSTANCE == null) {
+ private def init(roots: Array[String]): Unit = {
+ if (INSTANCE_INITIALIZED.compareAndSet(false, true)) {
INSTANCE = new SparkDirectoryUtil(roots)
- return
- }
- if (INSTANCE.roots.toSet != roots.toSet) {
- throw new IllegalArgumentException(
- s"Reinitialize SparkDirectoryUtil with different root dirs: old: ${INSTANCE.ROOTS
- .mkString("Array(", ", ", ")")}, new: ${roots.mkString("Array(", ", ", ")")}"
- )
+ } else {
+ logWarning("SparkDirectoryUtil is already initialized")
}
}
- def get(): SparkDirectoryUtil = synchronized {
- assert(INSTANCE != null, "Default instance of SparkDirectoryUtil was not set yet")
+ def get(): SparkDirectoryUtil = {
+ assert(INSTANCE_INITIALIZED.get(), "Default instance of SparkDirectoryUtil was not set yet")
INSTANCE
}
Review Comment:
Thank you for the explanation. I will try other way.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-10707] Improve SparkDirectoryUtil by reduce synchronized [incubator-gluten]
Posted by "beliefer (via GitHub)" <gi...@apache.org>.
beliefer commented on PR #10708:
URL: https://github.com/apache/incubator-gluten/pull/10708#issuecomment-3396294737
@zhztheplayer Thank you!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-10707] Improve SparkDirectoryUtil by reduce synchronized [incubator-gluten]
Posted by "zhztheplayer (via GitHub)" <gi...@apache.org>.
zhztheplayer merged PR #10708:
URL: https://github.com/apache/incubator-gluten/pull/10708
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-10707] Improve SparkDirectoryUtil with AtomicBoolean [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #10708:
URL: https://github.com/apache/incubator-gluten/pull/10708#issuecomment-3300998716
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-10707] Improve SparkDirectoryUtil with AtomicReference [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #10708:
URL: https://github.com/apache/incubator-gluten/pull/10708#issuecomment-3296867964
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-10707] Improve SparkDirectoryUtil by reduce synchronized [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #10708:
URL: https://github.com/apache/incubator-gluten/pull/10708#issuecomment-3305217233
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-10707] Improve SparkDirectoryUtil with AtomicBoolean [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #10708:
URL: https://github.com/apache/incubator-gluten/pull/10708#issuecomment-3300995808
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-10707] Improve SparkDirectoryUtil with AtomicBoolean [incubator-gluten]
Posted by "beliefer (via GitHub)" <gi...@apache.org>.
beliefer commented on code in PR #10708:
URL: https://github.com/apache/incubator-gluten/pull/10708#discussion_r2354114204
##########
gluten-core/src/main/scala/org/apache/spark/util/SparkDirectoryUtil.scala:
##########
@@ -87,7 +91,7 @@ object SparkDirectoryUtil extends Logging {
}
}
- def get(): SparkDirectoryUtil = synchronized {
+ def get(): SparkDirectoryUtil = {
assert(INSTANCE != null, "Default instance of SparkDirectoryUtil was not set yet")
INSTANCE
}
Review Comment:
Now, it is guarded with `INSTANCE_INITIALIZED`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-10707] Improve SparkDirectoryUtil with AtomicReference [incubator-gluten]
Posted by "beliefer (via GitHub)" <gi...@apache.org>.
beliefer commented on code in PR #10708:
URL: https://github.com/apache/incubator-gluten/pull/10708#discussion_r2350501074
##########
gluten-core/src/main/scala/org/apache/spark/util/SparkDirectoryUtil.scala:
##########
@@ -67,29 +68,31 @@ class SparkDirectoryUtil private (val roots: Array[String]) extends Logging {
}
object SparkDirectoryUtil extends Logging {
- private var INSTANCE: SparkDirectoryUtil = _
+ private val INSTANCE = new AtomicReference[SparkDirectoryUtil]()
- def init(conf: SparkConf): Unit = synchronized {
+ def init(conf: SparkConf): Unit = {
val roots = Utils.getConfiguredLocalDirs(conf)
init(roots)
}
- private def init(roots: Array[String]): Unit = synchronized {
- if (INSTANCE == null) {
- INSTANCE = new SparkDirectoryUtil(roots)
+ private def init(roots: Array[String]): Unit = {
+ if (INSTANCE.get() == null) {
+ INSTANCE.compareAndSet(null, new SparkDirectoryUtil(roots))
Review Comment:
I'm sorry. I didn't understand what you said. Could you tell me more ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-10707] Improve SparkDirectoryUtil with AtomicReference [incubator-gluten]
Posted by "beliefer (via GitHub)" <gi...@apache.org>.
beliefer commented on PR #10708:
URL: https://github.com/apache/incubator-gluten/pull/10708#issuecomment-3294614589
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-10707] Improve SparkDirectoryUtil with AtomicBoolean [incubator-gluten]
Posted by "zhztheplayer (via GitHub)" <gi...@apache.org>.
zhztheplayer commented on code in PR #10708:
URL: https://github.com/apache/incubator-gluten/pull/10708#discussion_r2352760750
##########
gluten-core/src/main/scala/org/apache/spark/util/SparkDirectoryUtil.scala:
##########
@@ -67,16 +68,19 @@ class SparkDirectoryUtil private (val roots: Array[String]) extends Logging {
}
object SparkDirectoryUtil extends Logging {
+ private val state = new AtomicBoolean(false)
Review Comment:
Let's rename it. E.g., `INSTANCE_INITIALIZED`
##########
gluten-core/src/main/scala/org/apache/spark/util/SparkDirectoryUtil.scala:
##########
@@ -67,16 +68,19 @@ class SparkDirectoryUtil private (val roots: Array[String]) extends Logging {
}
object SparkDirectoryUtil extends Logging {
+ private val state = new AtomicBoolean(false)
private var INSTANCE: SparkDirectoryUtil = _
- def init(conf: SparkConf): Unit = synchronized {
+ def init(conf: SparkConf): Unit = {
val roots = Utils.getConfiguredLocalDirs(conf)
init(roots)
}
- private def init(roots: Array[String]): Unit = synchronized {
+ private def init(roots: Array[String]): Unit = {
if (INSTANCE == null) {
- INSTANCE = new SparkDirectoryUtil(roots)
+ if (state.compareAndSet(false, true)) {
+ INSTANCE = new SparkDirectoryUtil(roots)
+ }
return
Review Comment:
It's not the hot code path, let's remove the `if (INSTANCE == null)` condition.
Just directly call `compareAndSet`.
##########
gluten-core/src/main/scala/org/apache/spark/util/SparkDirectoryUtil.scala:
##########
@@ -87,7 +91,7 @@ object SparkDirectoryUtil extends Logging {
}
}
- def get(): SparkDirectoryUtil = synchronized {
+ def get(): SparkDirectoryUtil = {
assert(INSTANCE != null, "Default instance of SparkDirectoryUtil was not set yet")
INSTANCE
}
Review Comment:
The get operation is not synchronized.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-10707] Improve SparkDirectoryUtil with AtomicBoolean [incubator-gluten]
Posted by "beliefer (via GitHub)" <gi...@apache.org>.
beliefer commented on PR #10708:
URL: https://github.com/apache/incubator-gluten/pull/10708#issuecomment-3301022528
> The get operation is still not synchronized... I am inclined to keep the current main code. Because this is not the hot code path and is not complicated. Do you agree?
Now, the issue has been fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-10707] Improve SparkDirectoryUtil by reduce synchronized [incubator-gluten]
Posted by "beliefer (via GitHub)" <gi...@apache.org>.
beliefer commented on PR #10708:
URL: https://github.com/apache/incubator-gluten/pull/10708#issuecomment-3389082032
@zhztheplayer Could you help me review this again ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-10707] Improve SparkDirectoryUtil with AtomicReference [incubator-gluten]
Posted by "beliefer (via GitHub)" <gi...@apache.org>.
beliefer commented on code in PR #10708:
URL: https://github.com/apache/incubator-gluten/pull/10708#discussion_r2351572271
##########
gluten-core/src/main/scala/org/apache/spark/util/SparkDirectoryUtil.scala:
##########
@@ -67,29 +68,31 @@ class SparkDirectoryUtil private (val roots: Array[String]) extends Logging {
}
object SparkDirectoryUtil extends Logging {
- private var INSTANCE: SparkDirectoryUtil = _
+ private val INSTANCE = new AtomicReference[SparkDirectoryUtil]()
- def init(conf: SparkConf): Unit = synchronized {
+ def init(conf: SparkConf): Unit = {
val roots = Utils.getConfiguredLocalDirs(conf)
init(roots)
}
- private def init(roots: Array[String]): Unit = synchronized {
- if (INSTANCE == null) {
- INSTANCE = new SparkDirectoryUtil(roots)
+ private def init(roots: Array[String]): Unit = {
+ if (INSTANCE.get() == null) {
+ INSTANCE.compareAndSet(null, new SparkDirectoryUtil(roots))
Review Comment:
Got it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-10707] Improve SparkDirectoryUtil with AtomicBoolean [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #10708:
URL: https://github.com/apache/incubator-gluten/pull/10708#issuecomment-3297099627
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-10707] Improve SparkDirectoryUtil with AtomicReference [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #10708:
URL: https://github.com/apache/incubator-gluten/pull/10708#issuecomment-3291973105
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-10707] Improve SparkDirectoryUtil with AtomicReference [incubator-gluten]
Posted by "zhztheplayer (via GitHub)" <gi...@apache.org>.
zhztheplayer commented on code in PR #10708:
URL: https://github.com/apache/incubator-gluten/pull/10708#discussion_r2349607173
##########
gluten-core/src/main/scala/org/apache/spark/util/SparkDirectoryUtil.scala:
##########
@@ -67,29 +68,31 @@ class SparkDirectoryUtil private (val roots: Array[String]) extends Logging {
}
object SparkDirectoryUtil extends Logging {
- private var INSTANCE: SparkDirectoryUtil = _
+ private val INSTANCE = new AtomicReference[SparkDirectoryUtil]()
- def init(conf: SparkConf): Unit = synchronized {
+ def init(conf: SparkConf): Unit = {
val roots = Utils.getConfiguredLocalDirs(conf)
init(roots)
}
- private def init(roots: Array[String]): Unit = synchronized {
- if (INSTANCE == null) {
- INSTANCE = new SparkDirectoryUtil(roots)
+ private def init(roots: Array[String]): Unit = {
+ if (INSTANCE.get() == null) {
+ INSTANCE.compareAndSet(null, new SparkDirectoryUtil(roots))
Review Comment:
The code is workable but the constructor of `SparkDirectoryUtil` may be unnecessarily called in concurrent invocations.
We can use `AtomicBoolean` instead if unable to avoid this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-10707] Improve SparkDirectoryUtil by reduce synchronized [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #10708:
URL: https://github.com/apache/incubator-gluten/pull/10708#issuecomment-3305227073
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-10707] Improve SparkDirectoryUtil with AtomicBoolean [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #10708:
URL: https://github.com/apache/incubator-gluten/pull/10708#issuecomment-3305209603
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-10707] Improve SparkDirectoryUtil with AtomicBoolean [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #10708:
URL: https://github.com/apache/incubator-gluten/pull/10708#issuecomment-3301012255
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org