You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@gluten.apache.org by "wankunde (via GitHub)" <gi...@apache.org> on 2025/05/27 14:58:48 UTC
[PR] [GLUTEN-9769][CELEBORN] Correct shuffle type [incubator-gluten]
wankunde opened a new pull request, #9770:
URL: https://github.com/apache/incubator-gluten/pull/9770
## What changes were proposed in this pull request?
Bug fix for velox + celeborn + columnar shuffle.
gluten changes the shuffle writer type of celeborn in https://github.com/apache/incubator-gluten/blob/main/backends-velox/src/main/scala/org/apache/gluten/backendsapi/velox/VeloxSparkPlanExecApi.scala#L555-L557
As a result, we cannot determine whether the shuffle writer in VeloxCelebornColumnarShuffleWriter is sort or not, can only use the passed in shuffle time metric .
(Fixes: \#9769)
## How was this patch tested?
manual tests
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [WIP][GLUTEN-9769][CELEBORN] Use the correct shuffle time metrics for celeborn columnar shuffle [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #9770:
URL: https://github.com/apache/incubator-gluten/pull/9770#issuecomment-2921184692
Run Gluten ClickHouse CI on ARM
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-9769][CELEBORN] Use the correct shuffle time metrics for celeborn columnar shuffle [incubator-gluten]
Posted by "wForget (via GitHub)" <gi...@apache.org>.
wForget commented on code in PR #9770:
URL: https://github.com/apache/incubator-gluten/pull/9770#discussion_r2111697645
##########
backends-velox/src-celeborn/main/scala/org/apache/spark/shuffle/VeloxCelebornColumnarShuffleWriter.scala:
##########
@@ -98,12 +98,7 @@ class VeloxCelebornColumnarShuffleWriter[K, V](
val startTime = System.nanoTime()
splitResult = jniWrapper.stop(nativeShuffleWriter)
- dep.metrics("shuffleWallTime").add(System.nanoTime() - startTime)
- val nativeMetrics = if (isSort) {
- dep.metrics("sortTime")
- } else {
- dep.metrics("splitTime")
- }
+ dep.remot("shuffleWallTime").add(System.nanoTime() - startTime)
Review Comment:
```suggestion
dep.metrics("shuffleWallTime").add(System.nanoTime() - startTime)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-9769][CELEBORN] Use the correct shuffle time metrics for celeborn columnar shuffle [incubator-gluten]
Posted by "wankunde (via GitHub)" <gi...@apache.org>.
wankunde commented on PR #9770:
URL: https://github.com/apache/incubator-gluten/pull/9770#issuecomment-2918495260
The UT failed in Ubuntu 20.04 image:
```log
The following packages have unmet dependencies:
maven : Depends: default-jre-headless (>= 2:1.7) but it is not installable or
java7-runtime-headless
Depends: libjansi-java but it is not installable
Depends: libmaven3-core-java (= 3.6.3-1ubuntu0.1) but it is not going to be installed
Depends: libwagon-file-java but it is not installable
Depends: libwagon-http-shaded-java but it is not installable
openjdk-8-jdk : Depends: openjdk-8-jre (= 8u4[52](https://github.com/apache/incubator-gluten/actions/runs/15315394531/job/43092539753?pr=9770#step:7:53)-ga~us1-0ubuntu1~20.04) but it is not going to be installed
Depends: openjdk-8-jdk-headless (= 8u452-ga~us1-0ubuntu1~20.04) but it is not going to be installed
Depends: libx11-6 but it is not going to be installed
Recommends: libxt-dev but it is not installable
E: Unable to correct problems, you have held broken packages.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [WIP][GLUTEN-9769][CELEBORN] Use the correct shuffle time metrics for celeborn columnar shuffle [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #9770:
URL: https://github.com/apache/incubator-gluten/pull/9770#issuecomment-2921181141
Run Gluten ClickHouse CI on ARM
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-9769][CELEBORN] Use the correct shuffle time metrics for celeborn columnar shuffle [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #9770:
URL: https://github.com/apache/incubator-gluten/pull/9770#issuecomment-2919901312
Run Gluten ClickHouse CI on ARM
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-9769][CELEBORN] Use the correct shuffle time metrics for celeborn columnar shuffle [incubator-gluten]
Posted by "wankunde (via GitHub)" <gi...@apache.org>.
wankunde commented on PR #9770:
URL: https://github.com/apache/incubator-gluten/pull/9770#issuecomment-2918893549
> Thanks for the fix. Can we get the `isSort` from ShuffleDependency in `CelebornShuffleManager` and pass it to `CelebornColumnarShuffleWriter`? Like what we do for `ColumnarShuffleWriter` https://github.com/apache/incubator-gluten/blob/main/backends-velox/src/main/scala/org/apache/spark/shuffle/ColumnarShuffleWriter.scala#L42
I think **get the `isSort` from ShuffleDependency in `CelebornShuffleManager`** is a good idea, and I find ColumnarShuffleWriter can get the dependency reference by `private val dep = handle.dependency.asInstanceOf[ColumnarShuffleDependency[K, V, V]]` , could we add a local variable `isSort` in ColumnarShuffleWriter ?
```
private val dep = handle.dependency.asInstanceOf[ColumnarShuffleDependency[K, V, V]]
val isSort: Boolean = dep.isSort
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-9769][CELEBORN] Use the correct shuffle time metrics for celeborn columnar shuffle [incubator-gluten]
Posted by "wankunde (via GitHub)" <gi...@apache.org>.
wankunde commented on code in PR #9770:
URL: https://github.com/apache/incubator-gluten/pull/9770#discussion_r2112039558
##########
backends-velox/src-celeborn/main/scala/org/apache/spark/shuffle/VeloxCelebornColumnarShuffleWriter.scala:
##########
@@ -98,12 +98,7 @@ class VeloxCelebornColumnarShuffleWriter[K, V](
val startTime = System.nanoTime()
splitResult = jniWrapper.stop(nativeShuffleWriter)
- dep.metrics("shuffleWallTime").add(System.nanoTime() - startTime)
- val nativeMetrics = if (isSort) {
- dep.metrics("sortTime")
- } else {
- dep.metrics("splitTime")
- }
+ dep.remot("shuffleWallTime").add(System.nanoTime() - startTime)
Review Comment:
I'm sorry for this mistake....
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-9769][CELEBORN] Use the correct shuffle time metrics for celeborn columnar shuffle [incubator-gluten]
Posted by "marin-ma (via GitHub)" <gi...@apache.org>.
marin-ma commented on PR #9770:
URL: https://github.com/apache/incubator-gluten/pull/9770#issuecomment-2919064917
> > Thanks for the fix. Can we get the `isSort` from ShuffleDependency in `CelebornShuffleManager` and pass it to `CelebornColumnarShuffleWriter`? Like what we do for `ColumnarShuffleWriter` https://github.com/apache/incubator-gluten/blob/main/backends-velox/src/main/scala/org/apache/spark/shuffle/ColumnarShuffleWriter.scala#L42
>
> I think **get the `isSort` from ShuffleDependency in `CelebornShuffleManager`** is a good idea, and I find ColumnarShuffleWriter can get the dependency reference by `private val dep = handle.dependency.asInstanceOf[ColumnarShuffleDependency[K, V, V]]` , could we add a local variable `isSort` in ColumnarShuffleWriter ?
>
> ```
> private val dep = handle.dependency.asInstanceOf[ColumnarShuffleDependency[K, V, V]]
> val isSort: Boolean = dep.isSort
> ```
@wankunde You're right. Then we can make the change simpler.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-9769][CELEBORN] Use the correct shuffle time metrics for celeborn columnar shuffle [incubator-gluten]
Posted by "wankunde (via GitHub)" <gi...@apache.org>.
wankunde commented on PR #9770:
URL: https://github.com/apache/incubator-gluten/pull/9770#issuecomment-2918495212
The UT failed in Ubuntu 20.04 image:
```log
The following packages have unmet dependencies:
maven : Depends: default-jre-headless (>= 2:1.7) but it is not installable or
java7-runtime-headless
Depends: libjansi-java but it is not installable
Depends: libmaven3-core-java (= 3.6.3-1ubuntu0.1) but it is not going to be installed
Depends: libwagon-file-java but it is not installable
Depends: libwagon-http-shaded-java but it is not installable
openjdk-8-jdk : Depends: openjdk-8-jre (= 8u4[52](https://github.com/apache/incubator-gluten/actions/runs/15315394531/job/43092539753?pr=9770#step:7:53)-ga~us1-0ubuntu1~20.04) but it is not going to be installed
Depends: openjdk-8-jdk-headless (= 8u452-ga~us1-0ubuntu1~20.04) but it is not going to be installed
Depends: libx11-6 but it is not going to be installed
Recommends: libxt-dev but it is not installable
E: Unable to correct problems, you have held broken packages.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [WIP][GLUTEN-9769][CELEBORN] Use the correct shuffle time metrics for celeborn columnar shuffle [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #9770:
URL: https://github.com/apache/incubator-gluten/pull/9770#issuecomment-2921073139
Run Gluten ClickHouse CI on ARM
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-9769][CELEBORN] Use the correct shuffle time metrics for celeborn columnar shuffle [incubator-gluten]
Posted by "wForget (via GitHub)" <gi...@apache.org>.
wForget commented on PR #9770:
URL: https://github.com/apache/incubator-gluten/pull/9770#issuecomment-2916082651
please remove unused import:
```
Error: ^
Error: /__w/incubator-gluten/incubator-gluten/backends-velox/src-celeborn/main/scala/org/apache/spark/shuffle/VeloxCelebornColumnarShuffleWriter.scala:22: error: Unused import
Error: import org.apache.gluten.config.ReservedKeys
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-9769][CELEBORN] Use the correct shuffle time metrics for celeborn columnar shuffle [incubator-gluten]
Posted by "zhouyuan (via GitHub)" <gi...@apache.org>.
zhouyuan commented on code in PR #9770:
URL: https://github.com/apache/incubator-gluten/pull/9770#discussion_r2113428912
##########
backends-velox/src-celeborn/main/scala/org/apache/spark/shuffle/VeloxCelebornColumnarShuffleWriter.scala:
##########
@@ -51,8 +50,6 @@ class VeloxCelebornColumnarShuffleWriter[K, V](
celebornConf,
client,
writeMetrics) {
- private val isSort = !ReservedKeys.GLUTEN_HASH_SHUFFLE_WRITER.equals(shuffleWriterType)
Review Comment:
Cc: @marin-ma
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-9769][CELEBORN] Use the correct shuffle time metrics for celeborn columnar shuffle [incubator-gluten]
Posted by "marin-ma (via GitHub)" <gi...@apache.org>.
marin-ma commented on PR #9770:
URL: https://github.com/apache/incubator-gluten/pull/9770#issuecomment-2918730899
Thanks for the fix. Can we get the `isSort` from ShuffleDependency in `CelebornShuffleManager` and pass it to `CelebornColumnarShuffleWriter`? Like what we do for `ColumnarShuffleWriter` https://github.com/apache/incubator-gluten/blob/main/backends-velox/src/main/scala/org/apache/spark/shuffle/ColumnarShuffleWriter.scala#L42
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [WIP][GLUTEN-9769][CELEBORN] Use the correct shuffle time metrics for celeborn columnar shuffle [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #9770:
URL: https://github.com/apache/incubator-gluten/pull/9770#issuecomment-2921125916
Run Gluten ClickHouse CI on ARM
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-9769][CELEBORN] Correct shuffle type [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #9770:
URL: https://github.com/apache/incubator-gluten/pull/9770#issuecomment-2912849632
https://github.com/apache/incubator-gluten/issues/9769
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-9769][CELEBORN] Use the correct shuffle time metrics for celeborn columnar shuffle [incubator-gluten]
Posted by "wankunde (via GitHub)" <gi...@apache.org>.
wankunde commented on PR #9770:
URL: https://github.com/apache/incubator-gluten/pull/9770#issuecomment-2916996441
The UT failed cased by java.lang.OutOfMemoryError: Java heap space
```
at java.util.Arrays.copyOf(Arrays.java:3236)
at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:118)
at java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93)
at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:153)
at org.apache.spark.util.ByteBufferOutputStream.write(ByteBufferOutputStream.scala:41)
at java.io.ObjectOutputStream$BlockDataOutputStream.drain(ObjectOutputStream.java:1877)
at java.io.ObjectOutputStream$BlockDataOutputStream.write(ObjectOutputStream.java:1848)
at java.io.ObjectOutputStream.writeArray(ObjectOutputStream.java:1334)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1174)
at java.io.ObjectOutputStream.writeArray(ObjectOutputStream.java:1378)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1174)
at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
at java.io.ObjectOutputStream.writeArray(ObjectOutputStream.java:1378)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1174)
at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348)
at org.apache.spark.serializer.JavaSerializationStream.writeObject(JavaSerializer.scala:44)
at org.apache.spark.serializer.JavaSerializerInstance.serialize(JavaSerializer.scala:101)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:551)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-9769][CELEBORN] Use the correct shuffle time metrics for celeborn columnar shuffle [incubator-gluten]
Posted by "zhouyuan (via GitHub)" <gi...@apache.org>.
zhouyuan commented on PR #9770:
URL: https://github.com/apache/incubator-gluten/pull/9770#issuecomment-2917039731
> The UT failed cased by java.lang.OutOfMemoryError: Java heap space
>
> ```
> at java.util.Arrays.copyOf(Arrays.java:3236)
> at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:118)
> at java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93)
> at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:153)
> at org.apache.spark.util.ByteBufferOutputStream.write(ByteBufferOutputStream.scala:41)
> at java.io.ObjectOutputStream$BlockDataOutputStream.drain(ObjectOutputStream.java:1877)
> at java.io.ObjectOutputStream$BlockDataOutputStream.write(ObjectOutputStream.java:1848)
> at java.io.ObjectOutputStream.writeArray(ObjectOutputStream.java:1334)
> at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1174)
> at java.io.ObjectOutputStream.writeArray(ObjectOutputStream.java:1378)
> at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1174)
> at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
> at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
> at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
> at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
> at java.io.ObjectOutputStream.writeArray(ObjectOutputStream.java:1378)
> at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1174)
> at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348)
> at org.apache.spark.serializer.JavaSerializationStream.writeObject(JavaSerializer.scala:44)
> at org.apache.spark.serializer.JavaSerializerInstance.serialize(JavaSerializer.scala:101)
> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:551)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:750)
> ```
The celeborn unit tests is related:
https://github.com/apache/incubator-gluten/actions/runs/15302482810/job/43046734848?pr=9770
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-9769][CELEBORN] Use the correct shuffle time metrics for celeborn columnar shuffle [incubator-gluten]
Posted by "wankunde (via GitHub)" <gi...@apache.org>.
wankunde commented on PR #9770:
URL: https://github.com/apache/incubator-gluten/pull/9770#issuecomment-2921890163
@zhouyuan @wForget @marin-ma
Could help to review this PR again ? Thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-9769][CELEBORN] Use the correct shuffle time metrics for celeborn columnar shuffle [incubator-gluten]
Posted by "marin-ma (via GitHub)" <gi...@apache.org>.
marin-ma merged PR #9770:
URL: https://github.com/apache/incubator-gluten/pull/9770
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-9769][CELEBORN] Use the correct shuffle time metrics for celeborn columnar shuffle [incubator-gluten]
Posted by "zhouyuan (via GitHub)" <gi...@apache.org>.
zhouyuan commented on PR #9770:
URL: https://github.com/apache/incubator-gluten/pull/9770#issuecomment-2914609029
@wankunde there are some compile failures, would you please fix it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-9769][CELEBORN] Use the correct shuffle time metrics for celeborn columnar shuffle [incubator-gluten]
Posted by "wankunde (via GitHub)" <gi...@apache.org>.
wankunde commented on PR #9770:
URL: https://github.com/apache/incubator-gluten/pull/9770#issuecomment-2916529576
@zhouyuan @wForget Much thanks for all your help
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-9769][CELEBORN] Use the correct shuffle time metrics for celeborn columnar shuffle [incubator-gluten]
Posted by "wankunde (via GitHub)" <gi...@apache.org>.
wankunde commented on PR #9770:
URL: https://github.com/apache/incubator-gluten/pull/9770#issuecomment-2916992979
Retest this please
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org