You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@uniffle.apache.org by "zhengchenyu (via GitHub)" <gi...@apache.org> on 2025/03/14 04:37:29 UTC
[PR] [#1750] feat(remote merge): Support Spark. [uniffle]
zhengchenyu opened a new pull request, #2405:
URL: https://github.com/apache/uniffle/pull/2405
### What changes were proposed in this pull request?
Support spark Framework
### Why are the changes needed?
#1750
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
unit test, integration test, real job in cluster.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [PR] [#1750] feat(remote merge): Support Spark. [uniffle]
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on PR #2405:
URL: https://github.com/apache/uniffle/pull/2405#issuecomment-2732176234
also cc @summaryzb
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [PR] [#1750] feat(remote merge): Support Spark. [uniffle]
Posted by "jerqi (via GitHub)" <gi...@apache.org>.
jerqi commented on PR #2405:
URL: https://github.com/apache/uniffle/pull/2405#issuecomment-2732008392
> > From my sight, this feature now can't be used in Spark SQL. Maybe RDD could use this.
>
> This test is based on draft pr [apache/spark#50248](https://github.com/apache/spark/pull/50248).
cc @LuciferYang
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [PR] [#1750] feat(remote merge): Support Spark. [uniffle]
Posted by "xianjingfeng (via GitHub)" <gi...@apache.org>.
xianjingfeng commented on code in PR #2405:
URL: https://github.com/apache/uniffle/pull/2405#discussion_r2077054594
##########
client-spark/common/src/main/java/org/apache/uniffle/shuffle/manager/RssShuffleManagerBase.java:
##########
@@ -1538,6 +1546,48 @@ public boolean isValidTask(String taskId) {
return !failedTaskIds.contains(taskId);
}
+ protected <K> String encode(Ordering<K> obj) {
+ try {
+ ByteArrayOutputStream baos = new ByteArrayOutputStream();
+ ObjectOutputStream oos = new ObjectOutputStream(baos);
+ oos.writeObject(obj);
+ oos.close();
+ return "#" + Base64.getEncoder().encodeToString(baos.toByteArray());
+ } catch (Exception e) {
+ throw new RssException(e);
+ }
+ }
+
+ protected <K, V, C> MergeContext buildMergeContext(ShuffleDependency<K, V, C> dependency) {
+ if (remoteMergeEnable) {
+ MergeContext mergeContext =
+ MergeContext.newBuilder()
+ .setKeyClass(dependency.keyClassName())
+ .setValueClass(
+ dependency.mapSideCombine()
+ ? dependency.combinerClassName().get()
+ : dependency.valueClassName())
+ .setComparatorClass(
+ dependency
+ .keyOrdering()
+ .map(
+ new AbstractFunction1<Ordering<K>, String>() {
+ @Override
+ public String apply(Ordering<K> o) {
+ return encode(o);
+ }
+ })
+ .get())
Review Comment:
if dependency.keyOrdering().isDefined() == false, an exception will be thrown here
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [PR] [#1750] feat(remote merge): Support Spark. [uniffle]
Posted by "zhengchenyu (via GitHub)" <gi...@apache.org>.
zhengchenyu commented on PR #2405:
URL: https://github.com/apache/uniffle/pull/2405#issuecomment-2732093224
> > > From my sight, this feature now can't be used in Spark SQL. Maybe RDD could use this.
> >
> >
> > This test is based on draft pr [apache/spark#50248](https://github.com/apache/spark/pull/50248).
>
> This will break the code implement of Spark. You would better to insert a new logic plan represents the distribution and partitioning after shuffling. You only need to implement some optimization rules.
Are you talking about changes to Spark? My initial idea was also to see if I could add a new rule. Maybe for map side, I could add new rules. But for reduce, adding a new SortExec is determined by determining whether distribution and partitioning match, which is not easy to do by adding a new Rule.
For the draft pr about changes to spark. It is only a draft to verify the feasibility of this proposal. There are still some code architectures that need to be refactored. For example, some partial aggregation in memory logic, add some logic to the rule.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [PR] [#1750] feat(remote merge): Support Spark. [uniffle]
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on PR #2405:
URL: https://github.com/apache/uniffle/pull/2405#issuecomment-2732244939
> > From my sight, this feature now can't be used in Spark SQL. Maybe RDD could use this.
>
> This test is based on draft pr [apache/spark#50248](https://github.com/apache/spark/pull/50248).
Does this one depend on SPARK-51398 being merged first?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [PR] [#1750] feat(remote merge): Support Spark. [uniffle]
Posted by "zhengchenyu (via GitHub)" <gi...@apache.org>.
zhengchenyu commented on code in PR #2405:
URL: https://github.com/apache/uniffle/pull/2405#discussion_r2000442729
##########
integration-test/spark3/src/test/scala/org/apache/uniffle/test/RMSparkSQLTest.scala:
##########
@@ -0,0 +1,152 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.uniffle.test
+
+import com.google.common.collect.Maps
+import org.apache.commons.lang3.StringUtils
+import org.apache.spark.SparkConf
+import org.apache.spark.shuffle.RssSparkConfig
+import org.apache.spark.sql.{Dataset, Row, SparkSession}
+//import org.apache.uniffle.common.config.RssBaseConf.RSS_KRYO_REGISTRATION_CLASSES;
+import org.apache.uniffle.common.rpc.ServerType
+import org.apache.uniffle.coordinator.CoordinatorConf
+import org.apache.uniffle.server.ShuffleServerConf
+import org.apache.uniffle.server.buffer.ShuffleBufferType
+import org.apache.uniffle.storage.util.StorageType
+import org.apache.uniffle.test.IntegrationTestBase._
+import org.junit.jupiter.api.Assertions.assertEquals
+import org.junit.jupiter.api.{BeforeAll, Test}
+
+import java.io.{File, FileWriter, PrintWriter}
+import java.util
+import java.util.Map
+
+object RMSparkSQLTest {
Review Comment:
Or I will rename RM to RMS or RS. In fact, merge sort is implemented on the server side, but there is no combine, and remote sort is fine.
If so, the previous code and documentation need to be modified. Maybe a new PR is needed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [PR] [#1750] feat(remote merge): Support Spark. [uniffle]
Posted by "zhengchenyu (via GitHub)" <gi...@apache.org>.
zhengchenyu commented on code in PR #2405:
URL: https://github.com/apache/uniffle/pull/2405#discussion_r2000407108
##########
integration-test/spark3/src/test/scala/org/apache/uniffle/test/RMSparkSQLTest.scala:
##########
@@ -0,0 +1,152 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.uniffle.test
+
+import com.google.common.collect.Maps
+import org.apache.commons.lang3.StringUtils
+import org.apache.spark.SparkConf
+import org.apache.spark.shuffle.RssSparkConfig
+import org.apache.spark.sql.{Dataset, Row, SparkSession}
+//import org.apache.uniffle.common.config.RssBaseConf.RSS_KRYO_REGISTRATION_CLASSES;
+import org.apache.uniffle.common.rpc.ServerType
+import org.apache.uniffle.coordinator.CoordinatorConf
+import org.apache.uniffle.server.ShuffleServerConf
+import org.apache.uniffle.server.buffer.ShuffleBufferType
+import org.apache.uniffle.storage.util.StorageType
+import org.apache.uniffle.test.IntegrationTestBase._
+import org.junit.jupiter.api.Assertions.assertEquals
+import org.junit.jupiter.api.{BeforeAll, Test}
+
+import java.io.{File, FileWriter, PrintWriter}
+import java.util
+import java.util.Map
+
+object RMSparkSQLTest {
Review Comment:
For me, when I mention RM, the first thing that comes to my mind is Yarn ResourceManager. But I don't have any other good names.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [PR] [#1750] feat(remote merge): Support Spark. [uniffle]
Posted by "jerqi (via GitHub)" <gi...@apache.org>.
jerqi commented on PR #2405:
URL: https://github.com/apache/uniffle/pull/2405#issuecomment-2732041749
> > From my sight, this feature now can't be used in Spark SQL. Maybe RDD could use this.
>
> This test is based on draft pr [apache/spark#50248](https://github.com/apache/spark/pull/50248).
This will break the code implement of Spark. You would better to insert a new logic plan represents the distribution and partitioning after shuffling. You only need to implement some optimization rules.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [PR] [#1750] feat(remote merge): Support Spark. [uniffle]
Posted by "zuston (via GitHub)" <gi...@apache.org>.
zuston commented on code in PR #2405:
URL: https://github.com/apache/uniffle/pull/2405#discussion_r2024158373
##########
client-spark/common/src/main/java/org/apache/uniffle/shuffle/manager/RssShuffleManagerBase.java:
##########
@@ -1538,6 +1546,48 @@ public boolean isValidTask(String taskId) {
return !failedTaskIds.contains(taskId);
}
+ protected <K> String encode(Ordering<K> obj) {
+ try {
+ ByteArrayOutputStream baos = new ByteArrayOutputStream();
+ ObjectOutputStream oos = new ObjectOutputStream(baos);
+ oos.writeObject(obj);
+ oos.close();
+ return "#" + Base64.getEncoder().encodeToString(baos.toByteArray());
+ } catch (Exception e) {
+ throw new RssException(e);
+ }
+ }
+
+ protected <K, V, C> MergeContext buildMergeContext(ShuffleDependency<K, V, C> dependency) {
+ if (remoteMergeEnable) {
+ MergeContext mergeContext =
+ MergeContext.newBuilder()
+ .setKeyClass(dependency.keyClassName())
+ .setValueClass(
+ dependency.mapSideCombine()
+ ? dependency.combinerClassName().get()
+ : dependency.valueClassName())
+ .setComparatorClass(
+ dependency
+ .keyOrdering()
Review Comment:
if the keyOrdering is empty, the remote merge could be disabled? @zhengchenyu
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [PR] [#1750] feat(remote merge): Support Spark. [uniffle]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #2405:
URL: https://github.com/apache/uniffle/pull/2405#issuecomment-2723513124
## Test Results
2 895 files - 116 2 895 suites - 116 5h 29m 42s ⏱️ - 1h 10m 10s
1 186 tests + 11 1 168 ✅ - 5 2 💤 ±0 0 ❌ ±0 16 🔥 +16
14 440 runs - 459 14 382 ✅ - 487 28 💤 - 2 0 ❌ ±0 30 🔥 +30
For more details on these errors, see [this check](https://github.com/apache/uniffle/runs/38756183880).
Results for commit 5711d2a8. ± Comparison against base commit 4300f93f.
<details>
<summary>This pull request <b>removes</b> 2 and <b>adds</b> 13 tests. <i>Note that renamed tests count towards both.</i></summary>
```
org.apache.spark.shuffle.RssSpark2ShuffleUtilsTest ‑ testCreateFetchFailedException
org.apache.spark.shuffle.RssSpark2ShuffleUtilsTest ‑ testIsStageResubmitSupported
```
```
org.apache.spark.shuffle.SparkCombinerTest ‑ testSparkCombiner
org.apache.spark.shuffle.reader.RMRssShuffleReaderTest ‑ testReadMulitPartitionShuffleWithCombine
org.apache.spark.shuffle.reader.RMRssShuffleReaderTest ‑ testReadMulitPartitionShuffleWithoutCombine
org.apache.spark.shuffle.reader.RMRssShuffleReaderTest ‑ testReadShuffleWithCombine
org.apache.spark.shuffle.reader.RMRssShuffleReaderTest ‑ testReadShuffleWithoutCombine
org.apache.spark.shuffle.writer.WriteBufferManagerTest ‑ testWriteRemoteMerge
org.apache.spark.shuffle.writer.WriteBufferTest ‑ testSortRecords
org.apache.uniffle.test.AQERepartitionTest ‑ org.apache.uniffle.test.AQERepartitionTest
org.apache.uniffle.test.ContinuousSelectPartitionStrategyTest ‑ org.apache.uniffle.test.ContinuousSelectPartitionStrategyTest
org.apache.uniffle.test.GetShuffleReportForMultiPartTest ‑ org.apache.uniffle.test.GetShuffleReportForMultiPartTest
…
```
</details>
[test-results]:data:application/gzip;base64,H4sIAG6402cC/03NTQ6DIBCG4asY1l0IDgi9TDMympD60yCsmt69QJW6/B7g5c0mN487uzdCG3lr2B5duG6KHoPb1iTc9FokSuch3+Bcq3M+9mhtMaX/9nSvXKowoZsTtBVG7zefn+WQj2vJAkB7zJqFrnz9syOrT7hkyz6rXRa7LYsLaTHZc04CtTRoAAdB3MIA7aRoJOpQSE6owBr2+QKrfTfcFgEAAA==
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [PR] [#1750] feat(remote merge): Support Spark. [uniffle]
Posted by "zhengchenyu (via GitHub)" <gi...@apache.org>.
zhengchenyu commented on PR #2405:
URL: https://github.com/apache/uniffle/pull/2405#issuecomment-2732006405
> From my sight, this feature now can't be used in Spark SQL. Maybe RDD could use this.
This test is based on draft pr https://github.com/apache/spark/pull/50248.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [PR] [#1750] feat(remote merge): Support Spark. [uniffle]
Posted by "summaryzb (via GitHub)" <gi...@apache.org>.
summaryzb commented on PR #2405:
URL: https://github.com/apache/uniffle/pull/2405#issuecomment-2739436830
> Are you talking about changes to Spark? My initial idea was also to see if I could add a new rule. Maybe for map side, I could add new rules. But for reduce, adding a new SortExec is determined by determining whether distribution and partitioning match, which is not easy to do by adding a new Rule.
Both for v1 and v2 datasource api of spark
```
plan match {
case PhysicalOperation(_, _, _: DataSourceV2ScanRelation) =>
new DataSourceV2Strategy(sparkSession).apply(plan).headOption match {
case Some(head) => tryOptimize(head) :: Nil
case _ => Nil
}
case PhysicalOperation(_, _, LogicalRelation(_: HadoopFsRelation, _, _, _)) =>
FileSourceStrategy(plan).headOption match {
case Some(head) => tryOptimize(head) :: Nil
case _ => Nil
}
case _ => Nil
```
for v2 datasource, tryOptimize follow below 4 scenario to extract the scan
1. [[ProjectExec]] -> [[FilterExec]] -> [[BatchScanExec]]
2. [[ProjectExec]] -> [[BatchScanExec]]
3. [[FilterExec]] -> [[BatchScanExec]]
4. [[BatchScanExec]]
override `protected def partitions` of FileScan to implement partitioning logic
for v1 datasource, tryOptimize follow below 4 scenario to extract the scan
1. [[ProjectExec]] -> [[FilterExec]] -> [[FileSourceScanExec]]
2. [[ProjectExec]] -> [[FileSourceScanExec]]
3. [[FilterExec]] -> [[FileSourceScanExec]]
4. [[FileSourceScanExec]]
replace `private def createReadRDD` for inputRdd of FileSourceScanExec to implement partitioning logic
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [PR] [#1750] feat(remote merge): Support Spark. [uniffle]
Posted by "zuston (via GitHub)" <gi...@apache.org>.
zuston commented on code in PR #2405:
URL: https://github.com/apache/uniffle/pull/2405#discussion_r2000393536
##########
integration-test/spark3/src/test/scala/org/apache/uniffle/test/RMSparkSQLTest.scala:
##########
@@ -0,0 +1,152 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.uniffle.test
+
+import com.google.common.collect.Maps
+import org.apache.commons.lang3.StringUtils
+import org.apache.spark.SparkConf
+import org.apache.spark.shuffle.RssSparkConfig
+import org.apache.spark.sql.{Dataset, Row, SparkSession}
+//import org.apache.uniffle.common.config.RssBaseConf.RSS_KRYO_REGISTRATION_CLASSES;
+import org.apache.uniffle.common.rpc.ServerType
+import org.apache.uniffle.coordinator.CoordinatorConf
+import org.apache.uniffle.server.ShuffleServerConf
+import org.apache.uniffle.server.buffer.ShuffleBufferType
+import org.apache.uniffle.storage.util.StorageType
+import org.apache.uniffle.test.IntegrationTestBase._
+import org.junit.jupiter.api.Assertions.assertEquals
+import org.junit.jupiter.api.{BeforeAll, Test}
+
+import java.io.{File, FileWriter, PrintWriter}
+import java.util
+import java.util.Map
+
+object RMSparkSQLTest {
Review Comment:
Aha! . How about renaming the `RemoteMergeSparkSQLTest` . RM looks Yarn ResourceManager
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [PR] [#1750] feat(remote merge): Support Spark. [uniffle]
Posted by "jerqi (via GitHub)" <gi...@apache.org>.
jerqi commented on PR #2405:
URL: https://github.com/apache/uniffle/pull/2405#issuecomment-2732269885
> > > > From my sight, this feature now can't be used in Spark SQL. Maybe RDD could use this.
> > >
> > >
> > > This test is based on draft pr [apache/spark#50248](https://github.com/apache/spark/pull/50248).
> >
> >
> > This will break the code implement of Spark. You would better to insert a new logic plan represents the distribution and partitioning after shuffling. You only need to implement some optimization rules.
>
> Are you talking about changes to Spark? My initial idea was also to see if I could add a new rule. Maybe for map side, I could add new rules. But for reduce, adding a new SortExec is determined by determining whether distribution and partitioning match, which is not easy to do by adding a new Rule. For the draft pr about changes to spark. It is only a draft to verify the feasibility of this proposal. There are still some code architectures that need to be refactored. For example, some partial aggregation in memory logic, add some logic to the rule.
Yes, Meta Cosco ever did similar things. You can see
https://github.com/apache/spark/pull/32944
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [PR] [#1750] feat(remote merge): Support Spark. [uniffle]
Posted by "zhengchenyu (via GitHub)" <gi...@apache.org>.
zhengchenyu commented on code in PR #2405:
URL: https://github.com/apache/uniffle/pull/2405#discussion_r2000407108
##########
integration-test/spark3/src/test/scala/org/apache/uniffle/test/RMSparkSQLTest.scala:
##########
@@ -0,0 +1,152 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.uniffle.test
+
+import com.google.common.collect.Maps
+import org.apache.commons.lang3.StringUtils
+import org.apache.spark.SparkConf
+import org.apache.spark.shuffle.RssSparkConfig
+import org.apache.spark.sql.{Dataset, Row, SparkSession}
+//import org.apache.uniffle.common.config.RssBaseConf.RSS_KRYO_REGISTRATION_CLASSES;
+import org.apache.uniffle.common.rpc.ServerType
+import org.apache.uniffle.coordinator.CoordinatorConf
+import org.apache.uniffle.server.ShuffleServerConf
+import org.apache.uniffle.server.buffer.ShuffleBufferType
+import org.apache.uniffle.storage.util.StorageType
+import org.apache.uniffle.test.IntegrationTestBase._
+import org.junit.jupiter.api.Assertions.assertEquals
+import org.junit.jupiter.api.{BeforeAll, Test}
+
+import java.io.{File, FileWriter, PrintWriter}
+import java.util
+import java.util.Map
+
+object RMSparkSQLTest {
Review Comment:
This test is based on draft pr https://github.com/apache/spark/pull/50248.
For me, when I mention RM, the first thing that comes to my mind is Yarn ResourceManager. But I don't have any other good names.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [PR] [#1750] feat(remote merge): Support Spark. [uniffle]
Posted by "zhengchenyu (via GitHub)" <gi...@apache.org>.
zhengchenyu commented on code in PR #2405:
URL: https://github.com/apache/uniffle/pull/2405#discussion_r2026417638
##########
client-spark/common/src/main/java/org/apache/uniffle/shuffle/manager/RssShuffleManagerBase.java:
##########
@@ -1538,6 +1546,48 @@ public boolean isValidTask(String taskId) {
return !failedTaskIds.contains(taskId);
}
+ protected <K> String encode(Ordering<K> obj) {
+ try {
+ ByteArrayOutputStream baos = new ByteArrayOutputStream();
+ ObjectOutputStream oos = new ObjectOutputStream(baos);
+ oos.writeObject(obj);
+ oos.close();
+ return "#" + Base64.getEncoder().encodeToString(baos.toByteArray());
+ } catch (Exception e) {
+ throw new RssException(e);
+ }
+ }
+
+ protected <K, V, C> MergeContext buildMergeContext(ShuffleDependency<K, V, C> dependency) {
+ if (remoteMergeEnable) {
+ MergeContext mergeContext =
+ MergeContext.newBuilder()
+ .setKeyClass(dependency.keyClassName())
+ .setValueClass(
+ dependency.mapSideCombine()
+ ? dependency.combinerClassName().get()
+ : dependency.valueClassName())
+ .setComparatorClass(
+ dependency
+ .keyOrdering()
Review Comment:
The current implementation cannot enable remote merge based on whether keyOrdering is empty, and can only be done through configuration.
For spark rdd, there may be such a situation (not for spark sql), keyOrdering is empty, but combineclass is not empty, it will be sorted by hash and then combined in memory.
This can be further improved, especially for spark sql.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org