You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@gluten.apache.org by "zhztheplayer (via GitHub)" <gi...@apache.org> on 2024/11/28 09:23:44 UTC
[PR] [GLUTEN-8060][CORE] GlutenShuffleManager as a registry of shuffle managers [incubator-gluten]
zhztheplayer opened a new pull request, #8084:
URL: https://github.com/apache/incubator-gluten/pull/8084
TBD
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-8060][CORE] GlutenShuffleManager as a registry of shuffle managers [incubator-gluten]
Posted by "zhztheplayer (via GitHub)" <gi...@apache.org>.
zhztheplayer commented on code in PR #8084:
URL: https://github.com/apache/incubator-gluten/pull/8084#discussion_r1862971075
##########
gluten-core/src/main/scala/org/apache/spark/shuffle/ShuffleManagerRegistry.scala:
##########
@@ -0,0 +1,94 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.shuffle
+
+import org.apache.spark.SparkConf
+import org.apache.spark.util.Utils
+
+import scala.collection.mutable
+
+class ShuffleManagerRegistry private[ShuffleManagerRegistry] {
+ import ShuffleManagerRegistry._
+ private val all: mutable.Buffer[(LookupKey, String)] = mutable.Buffer()
+ private val routerBuilders: mutable.Buffer[RouterBuilder] = mutable.Buffer()
Review Comment:
The reason we don't simply adopt a lazy initialization here is because the instantiation of `RouterBuilder` is parameterized: It takes `SparkConf` and `isDriver` in ctor. So we'd guard our code against the twice invocation with different parameters, no matter it's practically possible or not.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-8060][CORE] GlutenShuffleManager as a registry of shuffle managers [incubator-gluten]
Posted by "zhztheplayer (via GitHub)" <gi...@apache.org>.
zhztheplayer commented on code in PR #8084:
URL: https://github.com/apache/incubator-gluten/pull/8084#discussion_r1862886805
##########
gluten-core/src/main/scala/org/apache/spark/shuffle/ShuffleManagerRouter.scala:
##########
@@ -0,0 +1,137 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.shuffle
+import org.apache.spark.{ShuffleDependency, TaskContext}
+import org.apache.spark.network.buffer.ManagedBuffer
+import org.apache.spark.network.shuffle.MergedBlockMeta
+import org.apache.spark.storage.{BlockId, ShuffleBlockBatchId, ShuffleBlockId, ShuffleMergedBlockId}
+
+/** The internal shuffle manager instance used by GlutenShuffleManager. */
+private class ShuffleManagerRouter(lookup: ShuffleManagerLookup) extends ShuffleManager {
+ import ShuffleManagerRouter._
+ private val cache = new Cache()
+ private val resolver = new BlockResolver(cache)
+
+ override def registerShuffle[K, V, C](
+ shuffleId: Int,
+ dependency: ShuffleDependency[K, V, C]): ShuffleHandle = {
+ val manager = lookup.findShuffleManager(dependency)
+ cache.store(shuffleId, manager).registerShuffle(shuffleId, dependency)
+ }
+
+ override def getWriter[K, V](
+ handle: ShuffleHandle,
+ mapId: Long,
+ context: TaskContext,
+ metrics: ShuffleWriteMetricsReporter): ShuffleWriter[K, V] = {
+ cache.get(handle.shuffleId).getWriter(handle, mapId, context, metrics)
Review Comment:
`ShuffleHandle` is not passing to `ShuffleBlockResolver` so it could be tricky if using it as key. Would you expect any potential issues if we use shuffle ID as the key? Maybe we need another solution to avoid them.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-8060][CORE] GlutenShuffleManager as a registry of shuffle managers [incubator-gluten]
Posted by "wForget (via GitHub)" <gi...@apache.org>.
wForget commented on code in PR #8084:
URL: https://github.com/apache/incubator-gluten/pull/8084#discussion_r1862972141
##########
gluten-core/src/main/scala/org/apache/spark/shuffle/ShuffleManagerRouter.scala:
##########
@@ -0,0 +1,137 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.shuffle
+import org.apache.spark.{ShuffleDependency, TaskContext}
+import org.apache.spark.network.buffer.ManagedBuffer
+import org.apache.spark.network.shuffle.MergedBlockMeta
+import org.apache.spark.storage.{BlockId, ShuffleBlockBatchId, ShuffleBlockId, ShuffleMergedBlockId}
+
+/** The internal shuffle manager instance used by GlutenShuffleManager. */
+private class ShuffleManagerRouter(lookup: ShuffleManagerLookup) extends ShuffleManager {
+ import ShuffleManagerRouter._
+ private val cache = new Cache()
Review Comment:
> Perhaps a simple solution could be registering `ShuffleHandle -> ShuffleManager` mappings as well which can be used for lookup on executor side. Would you think that is feasible?
Sounds feasible
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-8060][CORE] GlutenShuffleManager as a registry of shuffle managers [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #8084:
URL: https://github.com/apache/incubator-gluten/pull/8084#issuecomment-2506996023
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-8060][CORE] GlutenShuffleManager as a registry of shuffle managers [incubator-gluten]
Posted by "wForget (via GitHub)" <gi...@apache.org>.
wForget commented on code in PR #8084:
URL: https://github.com/apache/incubator-gluten/pull/8084#discussion_r1862891057
##########
gluten-core/src/main/scala/org/apache/spark/shuffle/ShuffleManagerRegistry.scala:
##########
@@ -0,0 +1,94 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.shuffle
+
+import org.apache.spark.SparkConf
+import org.apache.spark.util.Utils
+
+import scala.collection.mutable
+
+class ShuffleManagerRegistry private[ShuffleManagerRegistry] {
+ import ShuffleManagerRegistry._
+ private val all: mutable.Buffer[(LookupKey, String)] = mutable.Buffer()
+ private val routerBuilders: mutable.Buffer[RouterBuilder] = mutable.Buffer()
+ private val classDeDup: mutable.Set[String] = mutable.Set()
+
+ def register(lookupKey: LookupKey, shuffleManagerClass: String): Unit = {
+ val clazz = Utils.classForName(shuffleManagerClass)
+ require(
+ !clazz.isAssignableFrom(classOf[GlutenShuffleManager]),
+ "It's not allowed to register GlutenShuffleManager recursively")
+ require(
+ classOf[ShuffleManager].isAssignableFrom(clazz),
+ s"Shuffle manager class to register is not an implementation of Spark ShuffleManager: " +
+ s"$shuffleManagerClass"
+ )
+ require(
+ !classDeDup.contains(shuffleManagerClass),
+ s"Shuffle manager class already registered: $shuffleManagerClass")
+ this.synchronized {
+ classDeDup += shuffleManagerClass
+ all += lookupKey -> shuffleManagerClass
+ // Invalidate all shuffle managers cached in each alive router builder instances.
+ // Then, once the router builder is accessed, a new router will be forced to create.
+ routerBuilders.foreach(_.invalidateCache())
+ }
+ }
+
+ // Visible for testing
+ private[shuffle] def clear(): Unit = {
+ this.synchronized {
+ classDeDup.clear()
+ all.clear()
+ routerBuilders.foreach(_.invalidateCache())
+ }
+ }
+
+ private[shuffle] def newRouterBuilder(conf: SparkConf, isDriver: Boolean): RouterBuilder =
+ this.synchronized {
+ val out = new RouterBuilder(this, conf, isDriver)
+ routerBuilders += out
+ out
+ }
+}
+
+object ShuffleManagerRegistry {
+ private val instance = new ShuffleManagerRegistry()
+
+ def get(): ShuffleManagerRegistry = instance
+
+ class RouterBuilder(registry: ShuffleManagerRegistry, conf: SparkConf, isDriver: Boolean) {
+ private var router: Option[ShuffleManagerRouter] = None
+
+ private[ShuffleManagerRegistry] def invalidateCache(): Unit = synchronized {
+ router = None
+ }
+
+ private[shuffle] def getOrBuild(): ShuffleManagerRouter = synchronized {
Review Comment:
Makes sense
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-8060][CORE] GlutenShuffleManager as a registry of shuffle managers [incubator-gluten]
Posted by "kerwin-zk (via GitHub)" <gi...@apache.org>.
kerwin-zk commented on PR #8084:
URL: https://github.com/apache/incubator-gluten/pull/8084#issuecomment-2506980605
@zhztheplayer Although spark.shuffle.manager can be uniformly configured as GlutenShuffleManager, additional configurations are still required to choose between ColumnarShuffleManager, CelebornShuffleManager, or UniffleShuffleManager. Essentially, this is no different from the current setup where spark.shuffle.manager is configured to different managers. Moreover, from a certain perspective, the introduction of GlutenShuffleManager couples the various ShuffleManagers together, whereas the current fully decoupled setup seems clearer.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-8060][CORE] GlutenShuffleManager as a registry of shuffle managers [incubator-gluten]
Posted by "zhztheplayer (via GitHub)" <gi...@apache.org>.
zhztheplayer commented on code in PR #8084:
URL: https://github.com/apache/incubator-gluten/pull/8084#discussion_r1862889208
##########
gluten-core/src/main/scala/org/apache/spark/shuffle/ShuffleManagerRegistry.scala:
##########
@@ -0,0 +1,94 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.shuffle
+
+import org.apache.spark.SparkConf
+import org.apache.spark.util.Utils
+
+import scala.collection.mutable
+
+class ShuffleManagerRegistry private[ShuffleManagerRegistry] {
+ import ShuffleManagerRegistry._
+ private val all: mutable.Buffer[(LookupKey, String)] = mutable.Buffer()
+ private val routerBuilders: mutable.Buffer[RouterBuilder] = mutable.Buffer()
+ private val classDeDup: mutable.Set[String] = mutable.Set()
+
+ def register(lookupKey: LookupKey, shuffleManagerClass: String): Unit = {
+ val clazz = Utils.classForName(shuffleManagerClass)
+ require(
+ !clazz.isAssignableFrom(classOf[GlutenShuffleManager]),
+ "It's not allowed to register GlutenShuffleManager recursively")
+ require(
+ classOf[ShuffleManager].isAssignableFrom(clazz),
+ s"Shuffle manager class to register is not an implementation of Spark ShuffleManager: " +
+ s"$shuffleManagerClass"
+ )
+ require(
+ !classDeDup.contains(shuffleManagerClass),
+ s"Shuffle manager class already registered: $shuffleManagerClass")
+ this.synchronized {
+ classDeDup += shuffleManagerClass
+ all += lookupKey -> shuffleManagerClass
+ // Invalidate all shuffle managers cached in each alive router builder instances.
+ // Then, once the router builder is accessed, a new router will be forced to create.
+ routerBuilders.foreach(_.invalidateCache())
+ }
+ }
+
+ // Visible for testing
+ private[shuffle] def clear(): Unit = {
+ this.synchronized {
+ classDeDup.clear()
+ all.clear()
+ routerBuilders.foreach(_.invalidateCache())
+ }
+ }
+
+ private[shuffle] def newRouterBuilder(conf: SparkConf, isDriver: Boolean): RouterBuilder =
+ this.synchronized {
+ val out = new RouterBuilder(this, conf, isDriver)
+ routerBuilders += out
+ out
+ }
+}
+
+object ShuffleManagerRegistry {
+ private val instance = new ShuffleManagerRegistry()
+
+ def get(): ShuffleManagerRegistry = instance
+
+ class RouterBuilder(registry: ShuffleManagerRegistry, conf: SparkConf, isDriver: Boolean) {
+ private var router: Option[ShuffleManagerRouter] = None
+
+ private[ShuffleManagerRegistry] def invalidateCache(): Unit = synchronized {
+ router = None
+ }
+
+ private[shuffle] def getOrBuild(): ShuffleManagerRouter = synchronized {
Review Comment:
I realized it's not the case for double-checked locking... `router` will be set to `None` again when new shuffle manager is registered.
Perhaps we can simplify the logic in future by enclosing the registry once all the backends' initializations are done, I can take care of this part.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-8060][CORE] GlutenShuffleManager as a registry of shuffle managers [incubator-gluten]
Posted by "zhztheplayer (via GitHub)" <gi...@apache.org>.
zhztheplayer commented on PR #8084:
URL: https://github.com/apache/incubator-gluten/pull/8084#issuecomment-2507019901
> Do you originally want to implement a mixture of ess and rss like https://github.com/apache/incubator-uniffle/blob/master/client-spark/spark3/src/main/java/org/apache/spark/shuffle/DelegationRssShuffleManager.java and https://github.com/apache/celeborn/blob/3bf91929b6bd02974b5d15b4d6804c9b2b01cfc0/client-spark/spark-3/src/main/java/org/apache/spark/shuffle/celeborn/SparkShuffleManager.java#L165?
No, it's not intentionally a refactor against current shuffles. It's for preparing the base support of mixed backend(s): #6920 .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-8060][CORE] GlutenShuffleManager as a registry of shuffle managers [incubator-gluten]
Posted by "zhztheplayer (via GitHub)" <gi...@apache.org>.
zhztheplayer commented on code in PR #8084:
URL: https://github.com/apache/incubator-gluten/pull/8084#discussion_r1862879947
##########
gluten-core/src/main/scala/org/apache/spark/shuffle/ShuffleManagerRouter.scala:
##########
@@ -0,0 +1,137 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.shuffle
+import org.apache.spark.{ShuffleDependency, TaskContext}
+import org.apache.spark.network.buffer.ManagedBuffer
+import org.apache.spark.network.shuffle.MergedBlockMeta
+import org.apache.spark.storage.{BlockId, ShuffleBlockBatchId, ShuffleBlockId, ShuffleMergedBlockId}
+
+/** The internal shuffle manager instance used by GlutenShuffleManager. */
+private class ShuffleManagerRouter(lookup: ShuffleManagerLookup) extends ShuffleManager {
+ import ShuffleManagerRouter._
+ private val cache = new Cache()
Review Comment:
I think so. The cache key is shuffle ID which will be shared among driver and executors.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-8060][CORE] GlutenShuffleManager as a registry of shuffle managers [incubator-gluten]
Posted by "zhztheplayer (via GitHub)" <gi...@apache.org>.
zhztheplayer commented on code in PR #8084:
URL: https://github.com/apache/incubator-gluten/pull/8084#discussion_r1862893980
##########
gluten-core/src/main/scala/org/apache/spark/shuffle/ShuffleManagerRouter.scala:
##########
@@ -0,0 +1,137 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.shuffle
+import org.apache.spark.{ShuffleDependency, TaskContext}
+import org.apache.spark.network.buffer.ManagedBuffer
+import org.apache.spark.network.shuffle.MergedBlockMeta
+import org.apache.spark.storage.{BlockId, ShuffleBlockBatchId, ShuffleBlockId, ShuffleMergedBlockId}
+
+/** The internal shuffle manager instance used by GlutenShuffleManager. */
+private class ShuffleManagerRouter(lookup: ShuffleManagerLookup) extends ShuffleManager {
+ import ShuffleManagerRouter._
+ private val cache = new Cache()
Review Comment:
Perhaps a simple solution could be register `ShuffleHandle -> ShuffleManager` mappings as well which can be used for lookup on executor side.. Would you think that is feasible?
##########
gluten-core/src/main/scala/org/apache/spark/shuffle/ShuffleManagerRouter.scala:
##########
@@ -0,0 +1,137 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.shuffle
+import org.apache.spark.{ShuffleDependency, TaskContext}
+import org.apache.spark.network.buffer.ManagedBuffer
+import org.apache.spark.network.shuffle.MergedBlockMeta
+import org.apache.spark.storage.{BlockId, ShuffleBlockBatchId, ShuffleBlockId, ShuffleMergedBlockId}
+
+/** The internal shuffle manager instance used by GlutenShuffleManager. */
+private class ShuffleManagerRouter(lookup: ShuffleManagerLookup) extends ShuffleManager {
+ import ShuffleManagerRouter._
+ private val cache = new Cache()
Review Comment:
Perhaps a simple solution could be register `ShuffleHandle -> ShuffleManager` mappings as well which can be used for lookup on executor side. Would you think that is feasible?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-8060][CORE] GlutenShuffleManager as a registry of shuffle managers [incubator-gluten]
Posted by "zhztheplayer (via GitHub)" <gi...@apache.org>.
zhztheplayer commented on code in PR #8084:
URL: https://github.com/apache/incubator-gluten/pull/8084#discussion_r1862971075
##########
gluten-core/src/main/scala/org/apache/spark/shuffle/ShuffleManagerRegistry.scala:
##########
@@ -0,0 +1,94 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.shuffle
+
+import org.apache.spark.SparkConf
+import org.apache.spark.util.Utils
+
+import scala.collection.mutable
+
+class ShuffleManagerRegistry private[ShuffleManagerRegistry] {
+ import ShuffleManagerRegistry._
+ private val all: mutable.Buffer[(LookupKey, String)] = mutable.Buffer()
+ private val routerBuilders: mutable.Buffer[RouterBuilder] = mutable.Buffer()
Review Comment:
The reason we don't simply adopt a lazy initialization here is because the instantiation of `RouterBuilder` is parameterized: It takes `SparkConf` and `isDriver` in ctor. So we'd guard our code from the twice invocation with different parameters, no matter it's practically possible or not.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-8060][CORE] GlutenShuffleManager as a registry of shuffle managers [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #8084:
URL: https://github.com/apache/incubator-gluten/pull/8084#issuecomment-2505636860
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-8060][CORE] GlutenShuffleManager as a registry of shuffle managers [incubator-gluten]
Posted by "wForget (via GitHub)" <gi...@apache.org>.
wForget commented on code in PR #8084:
URL: https://github.com/apache/incubator-gluten/pull/8084#discussion_r1862836756
##########
gluten-core/src/main/scala/org/apache/spark/shuffle/ShuffleManagerRegistry.scala:
##########
@@ -0,0 +1,94 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.shuffle
+
+import org.apache.spark.SparkConf
+import org.apache.spark.util.Utils
+
+import scala.collection.mutable
+
+class ShuffleManagerRegistry private[ShuffleManagerRegistry] {
+ import ShuffleManagerRegistry._
+ private val all: mutable.Buffer[(LookupKey, String)] = mutable.Buffer()
+ private val routerBuilders: mutable.Buffer[RouterBuilder] = mutable.Buffer()
+ private val classDeDup: mutable.Set[String] = mutable.Set()
+
+ def register(lookupKey: LookupKey, shuffleManagerClass: String): Unit = {
+ val clazz = Utils.classForName(shuffleManagerClass)
+ require(
+ !clazz.isAssignableFrom(classOf[GlutenShuffleManager]),
+ "It's not allowed to register GlutenShuffleManager recursively")
+ require(
+ classOf[ShuffleManager].isAssignableFrom(clazz),
+ s"Shuffle manager class to register is not an implementation of Spark ShuffleManager: " +
+ s"$shuffleManagerClass"
+ )
+ require(
+ !classDeDup.contains(shuffleManagerClass),
+ s"Shuffle manager class already registered: $shuffleManagerClass")
+ this.synchronized {
+ classDeDup += shuffleManagerClass
+ all += lookupKey -> shuffleManagerClass
+ // Invalidate all shuffle managers cached in each alive router builder instances.
+ // Then, once the router builder is accessed, a new router will be forced to create.
+ routerBuilders.foreach(_.invalidateCache())
+ }
+ }
+
+ // Visible for testing
+ private[shuffle] def clear(): Unit = {
+ this.synchronized {
+ classDeDup.clear()
+ all.clear()
+ routerBuilders.foreach(_.invalidateCache())
+ }
+ }
+
+ private[shuffle] def newRouterBuilder(conf: SparkConf, isDriver: Boolean): RouterBuilder =
+ this.synchronized {
+ val out = new RouterBuilder(this, conf, isDriver)
+ routerBuilders += out
+ out
+ }
+}
+
+object ShuffleManagerRegistry {
+ private val instance = new ShuffleManagerRegistry()
+
+ def get(): ShuffleManagerRegistry = instance
+
+ class RouterBuilder(registry: ShuffleManagerRegistry, conf: SparkConf, isDriver: Boolean) {
+ private var router: Option[ShuffleManagerRouter] = None
+
+ private[ShuffleManagerRegistry] def invalidateCache(): Unit = synchronized {
+ router = None
+ }
+
+ private[shuffle] def getOrBuild(): ShuffleManagerRouter = synchronized {
Review Comment:
```
if (router.isEmpty) {
synchronized {
if (router.isEmpty) {
....
}
}
}
router.get
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-8060][CORE] GlutenShuffleManager as a registry of shuffle managers [incubator-gluten]
Posted by "marin-ma (via GitHub)" <gi...@apache.org>.
marin-ma commented on code in PR #8084:
URL: https://github.com/apache/incubator-gluten/pull/8084#discussion_r1862920109
##########
gluten-core/src/main/scala/org/apache/spark/shuffle/ShuffleManagerRegistry.scala:
##########
@@ -0,0 +1,94 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.shuffle
+
+import org.apache.spark.SparkConf
+import org.apache.spark.util.Utils
+
+import scala.collection.mutable
+
+class ShuffleManagerRegistry private[ShuffleManagerRegistry] {
+ import ShuffleManagerRegistry._
+ private val all: mutable.Buffer[(LookupKey, String)] = mutable.Buffer()
+ private val routerBuilders: mutable.Buffer[RouterBuilder] = mutable.Buffer()
Review Comment:
Could you explain in what cases there would be multiple RouterBuilders? Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-8060][CORE] GlutenShuffleManager as a registry of shuffle managers [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #8084:
URL: https://github.com/apache/incubator-gluten/pull/8084#issuecomment-2505942317
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-8060][CORE] GlutenShuffleManager as a registry of shuffle managers [incubator-gluten]
Posted by "wForget (via GitHub)" <gi...@apache.org>.
wForget commented on PR #8084:
URL: https://github.com/apache/incubator-gluten/pull/8084#issuecomment-2506986620
Do you originally want to implement a mixture of ess and rss like https://github.com/apache/incubator-uniffle/blob/master/client-spark/spark3/src/main/java/org/apache/spark/shuffle/DelegationRssShuffleManager.java and https://github.com/apache/celeborn/blob/3bf91929b6bd02974b5d15b4d6804c9b2b01cfc0/client-spark/spark-3/src/main/java/org/apache/spark/shuffle/celeborn/SparkShuffleManager.java#L164?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-8060][CORE] GlutenShuffleManager as a registry of shuffle managers [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #8084:
URL: https://github.com/apache/incubator-gluten/pull/8084#issuecomment-2505636412
https://github.com/apache/incubator-gluten/issues/8060
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-8060][CORE] GlutenShuffleManager as a registry of shuffle managers [incubator-gluten]
Posted by "zhztheplayer (via GitHub)" <gi...@apache.org>.
zhztheplayer commented on PR #8084:
URL: https://github.com/apache/incubator-gluten/pull/8084#issuecomment-2507011897
@kerwin-zk
> Moreover, from a certain perspective, the introduction of GlutenShuffleManager couples the various ShuffleManagers together, whereas the current fully decoupled setup seems clearer.
I haven't clearly gotten what's the new code coupling come from. Say ideally we can remove the proxied calls from RSS shuffle managers to default shuffle manager because of this work (each shuffle manager can be registered for certain shuffle dependency type it can handle). So I see this a decoupling of code rather than coupling of code.
> Essentially, this is no different from the current setup where spark.shuffle.manager is configured to different managers.
But I may understand your concern totally especially this one. The reason I am proposing this is not simply because of code refactor. The solution's kind of a trade-off for letting user enable different backends at the same time. See the umbrella https://github.com/apache/incubator-gluten/issues/6920. For that purpose, we have to develop a way to make the shuffle managers be registered together so Gluten could choose the right one to use. For example, if we introduce GPU backend, we should give back the flexibility to developers when they decide not to let GPU backend rely on Velox backend, but they may still want query planner use Velox shuffle when the query plan is not offloaded to GPU.
May be we can put off deprecating the current use of `CelebornShuffleManager` / `UniffleShuffleManager`? We can keep the support for the legacy configurations during a considerable long period. I agree this is very normal that a user want's to specify a shuffle manager via Spark config explicitly. Does that work for you?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-8060][CORE] GlutenShuffleManager as a registry of shuffle managers [incubator-gluten]
Posted by "wForget (via GitHub)" <gi...@apache.org>.
wForget commented on code in PR #8084:
URL: https://github.com/apache/incubator-gluten/pull/8084#discussion_r1862855126
##########
gluten-core/src/main/scala/org/apache/spark/shuffle/ShuffleManagerRouter.scala:
##########
@@ -0,0 +1,137 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.shuffle
+import org.apache.spark.{ShuffleDependency, TaskContext}
+import org.apache.spark.network.buffer.ManagedBuffer
+import org.apache.spark.network.shuffle.MergedBlockMeta
+import org.apache.spark.storage.{BlockId, ShuffleBlockBatchId, ShuffleBlockId, ShuffleMergedBlockId}
+
+/** The internal shuffle manager instance used by GlutenShuffleManager. */
+private class ShuffleManagerRouter(lookup: ShuffleManagerLookup) extends ShuffleManager {
+ import ShuffleManagerRouter._
+ private val cache = new Cache()
+ private val resolver = new BlockResolver(cache)
+
+ override def registerShuffle[K, V, C](
+ shuffleId: Int,
+ dependency: ShuffleDependency[K, V, C]): ShuffleHandle = {
+ val manager = lookup.findShuffleManager(dependency)
+ cache.store(shuffleId, manager).registerShuffle(shuffleId, dependency)
+ }
+
+ override def getWriter[K, V](
+ handle: ShuffleHandle,
+ mapId: Long,
+ context: TaskContext,
+ metrics: ShuffleWriteMetricsReporter): ShuffleWriter[K, V] = {
+ cache.get(handle.shuffleId).getWriter(handle, mapId, context, metrics)
Review Comment:
It may be better to determine the appropriate ShuffleManager based on ShuffleHandle
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-8060][CORE] GlutenShuffleManager as a registry of shuffle managers [incubator-gluten]
Posted by "zhztheplayer (via GitHub)" <gi...@apache.org>.
zhztheplayer commented on code in PR #8084:
URL: https://github.com/apache/incubator-gluten/pull/8084#discussion_r1862890322
##########
gluten-core/src/main/scala/org/apache/spark/shuffle/ShuffleManagerRouter.scala:
##########
@@ -0,0 +1,137 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.shuffle
+import org.apache.spark.{ShuffleDependency, TaskContext}
+import org.apache.spark.network.buffer.ManagedBuffer
+import org.apache.spark.network.shuffle.MergedBlockMeta
+import org.apache.spark.storage.{BlockId, ShuffleBlockBatchId, ShuffleBlockId, ShuffleMergedBlockId}
+
+/** The internal shuffle manager instance used by GlutenShuffleManager. */
+private class ShuffleManagerRouter(lookup: ShuffleManagerLookup) extends ShuffleManager {
+ import ShuffleManagerRouter._
+ private val cache = new Cache()
Review Comment:
I see. It's likely a problem. Let me test, thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-8060][CORE] GlutenShuffleManager as a registry of shuffle managers [incubator-gluten]
Posted by "zhztheplayer (via GitHub)" <gi...@apache.org>.
zhztheplayer merged PR #8084:
URL: https://github.com/apache/incubator-gluten/pull/8084
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-8060][CORE] GlutenShuffleManager as a registry of shuffle managers [incubator-gluten]
Posted by "wForget (via GitHub)" <gi...@apache.org>.
wForget commented on code in PR #8084:
URL: https://github.com/apache/incubator-gluten/pull/8084#discussion_r1862844848
##########
gluten-core/src/main/scala/org/apache/spark/shuffle/ShuffleManagerRouter.scala:
##########
@@ -0,0 +1,137 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.shuffle
+import org.apache.spark.{ShuffleDependency, TaskContext}
+import org.apache.spark.network.buffer.ManagedBuffer
+import org.apache.spark.network.shuffle.MergedBlockMeta
+import org.apache.spark.storage.{BlockId, ShuffleBlockBatchId, ShuffleBlockId, ShuffleMergedBlockId}
+
+/** The internal shuffle manager instance used by GlutenShuffleManager. */
+private class ShuffleManagerRouter(lookup: ShuffleManagerLookup) extends ShuffleManager {
+ import ShuffleManagerRouter._
+ private val cache = new Cache()
Review Comment:
Will this status be sensed by spark executor?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-8060][CORE] GlutenShuffleManager as a registry of shuffle managers [incubator-gluten]
Posted by "wForget (via GitHub)" <gi...@apache.org>.
wForget commented on code in PR #8084:
URL: https://github.com/apache/incubator-gluten/pull/8084#discussion_r1862889329
##########
gluten-core/src/main/scala/org/apache/spark/shuffle/ShuffleManagerRouter.scala:
##########
@@ -0,0 +1,137 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.shuffle
+import org.apache.spark.{ShuffleDependency, TaskContext}
+import org.apache.spark.network.buffer.ManagedBuffer
+import org.apache.spark.network.shuffle.MergedBlockMeta
+import org.apache.spark.storage.{BlockId, ShuffleBlockBatchId, ShuffleBlockId, ShuffleMergedBlockId}
+
+/** The internal shuffle manager instance used by GlutenShuffleManager. */
+private class ShuffleManagerRouter(lookup: ShuffleManagerLookup) extends ShuffleManager {
+ import ShuffleManagerRouter._
+ private val cache = new Cache()
Review Comment:
Will the mapping saved in `cache.store` also be synchronized to the executor? I am worried that calling `cache.get(shuffleId)` on the executor will not be able to find the ShuffleManager
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-8060][CORE] GlutenShuffleManager as a registry of shuffle managers [incubator-gluten]
Posted by "kerwin-zk (via GitHub)" <gi...@apache.org>.
kerwin-zk commented on PR #8084:
URL: https://github.com/apache/incubator-gluten/pull/8084#issuecomment-2507033809
> @kerwin-zk
>
> > Moreover, from a certain perspective, the introduction of GlutenShuffleManager couples the various ShuffleManagers together, whereas the current fully decoupled setup seems clearer.
>
> I haven't clearly gotten what's the new code coupling come from. Say ideally we can remove the proxied calls from RSS shuffle managers to default shuffle manager because of this work (each shuffle manager can be registered for certain shuffle dependency type it can handle). So I see this a decoupling of code rather than coupling of code.
>
> > Essentially, this is no different from the current setup where spark.shuffle.manager is configured to different managers.
>
> But I may understand your concern totally especially this one. The reason I am proposing this is not simply because of code refactor. The solution's kind of a trade-off for letting user enable different backends at the same time. See the umbrella #6920. For that purpose, we have to develop a way to make the shuffle managers be registered together so Gluten could choose the right one to use. For example, if we introduce GPU backend, we should give back the flexibility to developers when they decide not to let GPU backend rely on Velox backend, but they may still want query planner use Velox shuffle when the query plan is not offloaded to GPU.
>
> May be we can put off deprecating the current use of `CelebornShuffleManager` / `UniffleShuffleManager`? We can keep the support for the legacy configurations during a considerable long period. I agree this is very normal that a user wants to specify a shuffle manager via Spark config explicitly. Does that work for you?
@zhztheplayer Thank you very much for your detailed explanation. I believe that #6920 indeed requires this feature. At the same time, retaining the current option that allows users to specify the ShuffleManager through configuration is also necessary. I think this is acceptable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-8060][CORE] GlutenShuffleManager as a registry of shuffle managers [incubator-gluten]
Posted by "zhztheplayer (via GitHub)" <gi...@apache.org>.
zhztheplayer commented on code in PR #8084:
URL: https://github.com/apache/incubator-gluten/pull/8084#discussion_r1862893980
##########
gluten-core/src/main/scala/org/apache/spark/shuffle/ShuffleManagerRouter.scala:
##########
@@ -0,0 +1,137 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.shuffle
+import org.apache.spark.{ShuffleDependency, TaskContext}
+import org.apache.spark.network.buffer.ManagedBuffer
+import org.apache.spark.network.shuffle.MergedBlockMeta
+import org.apache.spark.storage.{BlockId, ShuffleBlockBatchId, ShuffleBlockId, ShuffleMergedBlockId}
+
+/** The internal shuffle manager instance used by GlutenShuffleManager. */
+private class ShuffleManagerRouter(lookup: ShuffleManagerLookup) extends ShuffleManager {
+ import ShuffleManagerRouter._
+ private val cache = new Cache()
Review Comment:
Perhaps a simple solution could be registering `ShuffleHandle -> ShuffleManager` mappings as well which can be used for lookup on executor side. Would you think that is feasible?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-8060][CORE] GlutenShuffleManager as a registry of shuffle managers [incubator-gluten]
Posted by "zhztheplayer (via GitHub)" <gi...@apache.org>.
zhztheplayer commented on code in PR #8084:
URL: https://github.com/apache/incubator-gluten/pull/8084#discussion_r1862964057
##########
gluten-core/src/main/scala/org/apache/spark/shuffle/ShuffleManagerRouter.scala:
##########
@@ -0,0 +1,137 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.shuffle
+import org.apache.spark.{ShuffleDependency, TaskContext}
+import org.apache.spark.network.buffer.ManagedBuffer
+import org.apache.spark.network.shuffle.MergedBlockMeta
+import org.apache.spark.storage.{BlockId, ShuffleBlockBatchId, ShuffleBlockId, ShuffleMergedBlockId}
+
+/** The internal shuffle manager instance used by GlutenShuffleManager. */
+private class ShuffleManagerRouter(lookup: ShuffleManagerLookup) extends ShuffleManager {
+ import ShuffleManagerRouter._
+ private val cache = new Cache()
Review Comment:
@wForget I assume the current code is fine, given that usually shuffle dependency builder (namely #getDependencies) is carried by shuffle RDD and called from task code. Will run some tests later to prove.
https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/rdd/ShuffledRDD.scala
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-8060][CORE] GlutenShuffleManager as a registry of shuffle managers [incubator-gluten]
Posted by "zhztheplayer (via GitHub)" <gi...@apache.org>.
zhztheplayer commented on code in PR #8084:
URL: https://github.com/apache/incubator-gluten/pull/8084#discussion_r1862964057
##########
gluten-core/src/main/scala/org/apache/spark/shuffle/ShuffleManagerRouter.scala:
##########
@@ -0,0 +1,137 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.shuffle
+import org.apache.spark.{ShuffleDependency, TaskContext}
+import org.apache.spark.network.buffer.ManagedBuffer
+import org.apache.spark.network.shuffle.MergedBlockMeta
+import org.apache.spark.storage.{BlockId, ShuffleBlockBatchId, ShuffleBlockId, ShuffleMergedBlockId}
+
+/** The internal shuffle manager instance used by GlutenShuffleManager. */
+private class ShuffleManagerRouter(lookup: ShuffleManagerLookup) extends ShuffleManager {
+ import ShuffleManagerRouter._
+ private val cache = new Cache()
Review Comment:
@wForget I assume the current code is fine, given that usually shuffle dependency is carried by shuffle RDD and called from task code. Will run some tests later to prove.
https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/rdd/ShuffledRDD.scala
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-8060][CORE] GlutenShuffleManager as a registry of shuffle managers [incubator-gluten]
Posted by "zhztheplayer (via GitHub)" <gi...@apache.org>.
zhztheplayer commented on code in PR #8084:
URL: https://github.com/apache/incubator-gluten/pull/8084#discussion_r1862966292
##########
gluten-core/src/main/scala/org/apache/spark/shuffle/ShuffleManagerRegistry.scala:
##########
@@ -0,0 +1,94 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.shuffle
+
+import org.apache.spark.SparkConf
+import org.apache.spark.util.Utils
+
+import scala.collection.mutable
+
+class ShuffleManagerRegistry private[ShuffleManagerRegistry] {
+ import ShuffleManagerRegistry._
+ private val all: mutable.Buffer[(LookupKey, String)] = mutable.Buffer()
+ private val routerBuilders: mutable.Buffer[RouterBuilder] = mutable.Buffer()
Review Comment:
I think we usually have one single router builder at the same time. So this practice can be simplified in future with some kind of assertions (something like `if (router created) throw error`). Though it's benign at the moment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-8060][CORE] GlutenShuffleManager as a registry of shuffle managers [incubator-gluten]
Posted by "zhztheplayer (via GitHub)" <gi...@apache.org>.
zhztheplayer commented on PR #8084:
URL: https://github.com/apache/incubator-gluten/pull/8084#issuecomment-2507121998
cc @zzcclp @baibaichen
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-8060][CORE] GlutenShuffleManager as a registry of shuffle managers [incubator-gluten]
Posted by "zhztheplayer (via GitHub)" <gi...@apache.org>.
zhztheplayer commented on PR #8084:
URL: https://github.com/apache/incubator-gluten/pull/8084#issuecomment-2506905422
cc @kerwin-zk @summaryzb @marin-ma @wForget
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org