You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@gluten.apache.org by "dpengpeng (via GitHub)" <gi...@apache.org> on 2025/06/03 12:50:47 UTC
[I] Questions about the usage of Celeborn and concerns regarding the test performance results [incubator-gluten]
dpengpeng opened a new issue, #9853:
URL: https://github.com/apache/incubator-gluten/issues/9853
### Description
I have the following two questions:
Question 1:
On the official website, I saw that the usage configuration for Celeborn is as follows:
> Celeborn: celeborn-client-spark-3-shaded_2.12-[celebornVersion].jar
> Gluten: gluten-velox-bundle-spark3.x_2.12-xx_xx_xx-SNAPSHOT.jar, gluten-celeborn-package-xx-SNAPSHOT.jar
I compiled the Gluten 1.3.0 Release source code and obtained the gluten-velox-bundle-spark3.4_2.12-xx_xx_xx.jar file, along with the gluten-celeborn-1.3.0.jar. However, I only used the gluten-velox-bundle-spark3.4_2.12-xx_xx_xx.jar and did not place the gluten-celeborn-1.3.0.jar in the $SPARK_HOME/jars directory. Despite this, I was able to access the Celeborn service without any issues.
Upon checking the contents of the gluten-velox-bundle-spark3.4_2.12-xx_xx_xx.jar, I found that it includes Java and Scala class files from the gluten-celeborn module. Does this mean that the gluten-celeborn-1.3.0.jar is not necessary, and the official documentation on Celeborn Support usage is outdated?
Question 2:
Additionally, I tested a 1TB TPC-H dataset and observed a strange phenomenon. Although the overall performance improved when using Gluten + Rss compared to using Gluten alone, the performance of over ten out of the 22 SQL queries actually decreased. Has the official team also observed similar results?
### Gluten version
None
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL][CELEBORN] Questions about the usage of Celeborn and concerns regarding the test performance results [incubator-gluten]
Posted by "dpengpeng (via GitHub)" <gi...@apache.org>.
dpengpeng commented on issue #9853:
URL: https://github.com/apache/incubator-gluten/issues/9853#issuecomment-2948258770
> Similar as JVM spark, the usage of RSS isn't for performance but the seperation of computation and storage.
@FelixYBW
Thank you for your response. My intuition is that in K8S mode, the performance of Spark tasks using local disks for shuffle is inferior to using Celeborn. Therefore, I speculate that the performance of Gluten using Celeborn will be further improved in K8S mode.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL][CELEBORN]Questions about the usage of Celeborn and concerns regarding the test performance results [incubator-gluten]
Posted by "FelixYBW (via GitHub)" <gi...@apache.org>.
FelixYBW commented on issue #9853:
URL: https://github.com/apache/incubator-gluten/issues/9853#issuecomment-2936658200
Similar as JVM spark, the usage of RSS isn't for performance but the seperation of computation and storage.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL][CELEBORN] Questions about the usage of Celeborn and concerns regarding the test performance results [incubator-gluten]
Posted by "PHILO-HE (via GitHub)" <gi...@apache.org>.
PHILO-HE commented on issue #9853:
URL: https://github.com/apache/incubator-gluten/issues/9853#issuecomment-2948195244
> Upon checking the contents of the gluten-velox-bundle-spark3.4_2.12-xx_xx_xx.jar, I found that it includes Java and Scala class files from the gluten-celeborn module. Does this mean that the gluten-celeborn-1.3.0.jar is not necessary, and the official documentation on Celeborn Support usage is outdated?
@dpengpeng, the `gluten-velox-bundle-spark3.4_2.12-xx_xx_xx.jar` is a fat jar with all dependencies and modular classes included. So you don't need to deploy any modular jar. It may be different from the usage that integrating celeborn with vanilla Spark.
Please see the doc below. If there is any outdated content, please let us know.
https://github.com/apache/incubator-gluten/blob/branch-1.3/docs/get-started/Velox.md#celeborn-support
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL][CELEBORN] Questions about the usage of Celeborn and concerns regarding the test performance results [incubator-gluten]
Posted by "dpengpeng (via GitHub)" <gi...@apache.org>.
dpengpeng commented on issue #9853:
URL: https://github.com/apache/incubator-gluten/issues/9853#issuecomment-2948264222
> > Upon checking the contents of the gluten-velox-bundle-spark3.4_2.12-xx_xx_xx.jar, I found that it includes Java and Scala class files from the gluten-celeborn module. Does this mean that the gluten-celeborn-1.3.0.jar is not necessary, and the official documentation on Celeborn Support usage is outdated?
>
> [@dpengpeng](https://github.com/dpengpeng), the `gluten-velox-bundle-spark3.4_2.12-xx_xx_xx.jar` is a fat jar with all dependencies and modular classes included. So you don't need to deploy any modular jar. It may be different from the usage that integrating celeborn with vanilla Spark.
>
> Please see the doc below. If there is any outdated content, please let us know. https://github.com/apache/incubator-gluten/blob/branch-1.3/docs/get-started/Velox.md#celeborn-support
@PHILO-HE
Thanks, I was referring to the official documentation(https://gluten.apache.org/docs/getting-started/velox-backend/), and I understand it now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL][CELEBORN] Questions about the usage of Celeborn and concerns regarding the test performance results [incubator-gluten]
Posted by "FelixYBW (via GitHub)" <gi...@apache.org>.
FelixYBW commented on issue #9853:
URL: https://github.com/apache/incubator-gluten/issues/9853#issuecomment-2953158037
>
> [@FelixYBW](https://github.com/FelixYBW) Thank you for your response. My intuition is that in K8S mode, the performance of Spark tasks using local disks for shuffle is inferior to using Celeborn. Therefore, I speculate that the performance of Gluten using Celeborn will be further improved in K8S mode.
how much time perf boost when you switch from local disk to Celeborn using jvm spark? The features benefit jvm spark can also benefit Gluten.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL][CELEBORN] Questions about the usage of Celeborn and concerns regarding the test performance results [incubator-gluten]
Posted by "dpengpeng (via GitHub)" <gi...@apache.org>.
dpengpeng commented on issue #9853:
URL: https://github.com/apache/incubator-gluten/issues/9853#issuecomment-2954862965
> > [@FelixYBW](https://github.com/FelixYBW) Thank you for your response. My intuition is that in K8S mode, the performance of Spark tasks using local disks for shuffle is inferior to using Celeborn. Therefore, I speculate that the performance of Gluten using Celeborn will be further improved in K8S mode.
>
> how much time perf boost when you switch from local disk to Celeborn using jvm spark? The features benefit jvm spark can also benefit Gluten.
The first graph shows the performance improvement ratio of (Spark+RSS) compared to Spark. Two SQL queries show a decrease in performance.
The second graph shows the performance improvement ratio of (Gluen+RSS) compared to Gluten. Sixteen SQL queries show a decrease in performance.
From the above two graphs, it can be observed that the trend of Gluten differs from that of Spark. After using RSS, the performance of many SQL queries in Gluten has actually decreased. Could this be due to differences in the efficiency of operations on columnar shuffle data versus row-based shuffle data?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org