You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@gluten.apache.org by "wankunde (via GitHub)" <gi...@apache.org> on 2025/05/27 14:58:48 UTC

[PR] [GLUTEN-9769][CELEBORN] Correct shuffle type [incubator-gluten]

wankunde opened a new pull request, #9770:
URL: https://github.com/apache/incubator-gluten/pull/9770

   ## What changes were proposed in this pull request?
   
   Bug fix for velox + celeborn + columnar shuffle.
   
   gluten changes the shuffle writer type of celeborn in https://github.com/apache/incubator-gluten/blob/main/backends-velox/src/main/scala/org/apache/gluten/backendsapi/velox/VeloxSparkPlanExecApi.scala#L555-L557
   
   As a result, we cannot determine whether the shuffle writer in VeloxCelebornColumnarShuffleWriter is sort or not,  can only  use the passed in shuffle time metric .
   
   (Fixes: \#9769)
   
   ## How was this patch tested?
   
   manual tests
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [PR] [WIP][GLUTEN-9769][CELEBORN] Use the correct shuffle time metrics for celeborn columnar shuffle [incubator-gluten]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #9770:
URL: https://github.com/apache/incubator-gluten/pull/9770#issuecomment-2921184692

   Run Gluten ClickHouse CI on ARM


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [PR] [GLUTEN-9769][CELEBORN] Use the correct shuffle time metrics for celeborn columnar shuffle [incubator-gluten]

Posted by "wForget (via GitHub)" <gi...@apache.org>.
wForget commented on code in PR #9770:
URL: https://github.com/apache/incubator-gluten/pull/9770#discussion_r2111697645


##########
backends-velox/src-celeborn/main/scala/org/apache/spark/shuffle/VeloxCelebornColumnarShuffleWriter.scala:
##########
@@ -98,12 +98,7 @@ class VeloxCelebornColumnarShuffleWriter[K, V](
     val startTime = System.nanoTime()
     splitResult = jniWrapper.stop(nativeShuffleWriter)
 
-    dep.metrics("shuffleWallTime").add(System.nanoTime() - startTime)
-    val nativeMetrics = if (isSort) {
-      dep.metrics("sortTime")
-    } else {
-      dep.metrics("splitTime")
-    }
+    dep.remot("shuffleWallTime").add(System.nanoTime() - startTime)

Review Comment:
   ```suggestion
       dep.metrics("shuffleWallTime").add(System.nanoTime() - startTime)
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [PR] [GLUTEN-9769][CELEBORN] Use the correct shuffle time metrics for celeborn columnar shuffle [incubator-gluten]

Posted by "wankunde (via GitHub)" <gi...@apache.org>.
wankunde commented on PR #9770:
URL: https://github.com/apache/incubator-gluten/pull/9770#issuecomment-2918495260

   The UT failed in Ubuntu 20.04 image:
   
   ```log
   The following packages have unmet dependencies:
    maven : Depends: default-jre-headless (>= 2:1.7) but it is not installable or
                     java7-runtime-headless
            Depends: libjansi-java but it is not installable
            Depends: libmaven3-core-java (= 3.6.3-1ubuntu0.1) but it is not going to be installed
            Depends: libwagon-file-java but it is not installable
            Depends: libwagon-http-shaded-java but it is not installable
    openjdk-8-jdk : Depends: openjdk-8-jre (= 8u4[52](https://github.com/apache/incubator-gluten/actions/runs/15315394531/job/43092539753?pr=9770#step:7:53)-ga~us1-0ubuntu1~20.04) but it is not going to be installed
                    Depends: openjdk-8-jdk-headless (= 8u452-ga~us1-0ubuntu1~20.04) but it is not going to be installed
                    Depends: libx11-6 but it is not going to be installed
                    Recommends: libxt-dev but it is not installable
   E: Unable to correct problems, you have held broken packages.
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [PR] [WIP][GLUTEN-9769][CELEBORN] Use the correct shuffle time metrics for celeborn columnar shuffle [incubator-gluten]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #9770:
URL: https://github.com/apache/incubator-gluten/pull/9770#issuecomment-2921181141

   Run Gluten ClickHouse CI on ARM


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [PR] [GLUTEN-9769][CELEBORN] Use the correct shuffle time metrics for celeborn columnar shuffle [incubator-gluten]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #9770:
URL: https://github.com/apache/incubator-gluten/pull/9770#issuecomment-2919901312

   Run Gluten ClickHouse CI on ARM


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [PR] [GLUTEN-9769][CELEBORN] Use the correct shuffle time metrics for celeborn columnar shuffle [incubator-gluten]

Posted by "wankunde (via GitHub)" <gi...@apache.org>.
wankunde commented on PR #9770:
URL: https://github.com/apache/incubator-gluten/pull/9770#issuecomment-2918893549

   > Thanks for the fix. Can we get the `isSort` from ShuffleDependency in `CelebornShuffleManager` and pass it to `CelebornColumnarShuffleWriter`? Like what we do for `ColumnarShuffleWriter` https://github.com/apache/incubator-gluten/blob/main/backends-velox/src/main/scala/org/apache/spark/shuffle/ColumnarShuffleWriter.scala#L42
   
   I think **get the `isSort` from ShuffleDependency in `CelebornShuffleManager`** is a good idea, and I find ColumnarShuffleWriter can get the dependency reference by `private val dep = handle.dependency.asInstanceOf[ColumnarShuffleDependency[K, V, V]]` ,  could we add a local variable `isSort` in ColumnarShuffleWriter ?
   ```
     private val dep = handle.dependency.asInstanceOf[ColumnarShuffleDependency[K, V, V]]
     val isSort: Boolean = dep.isSort
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [PR] [GLUTEN-9769][CELEBORN] Use the correct shuffle time metrics for celeborn columnar shuffle [incubator-gluten]

Posted by "wankunde (via GitHub)" <gi...@apache.org>.
wankunde commented on code in PR #9770:
URL: https://github.com/apache/incubator-gluten/pull/9770#discussion_r2112039558


##########
backends-velox/src-celeborn/main/scala/org/apache/spark/shuffle/VeloxCelebornColumnarShuffleWriter.scala:
##########
@@ -98,12 +98,7 @@ class VeloxCelebornColumnarShuffleWriter[K, V](
     val startTime = System.nanoTime()
     splitResult = jniWrapper.stop(nativeShuffleWriter)
 
-    dep.metrics("shuffleWallTime").add(System.nanoTime() - startTime)
-    val nativeMetrics = if (isSort) {
-      dep.metrics("sortTime")
-    } else {
-      dep.metrics("splitTime")
-    }
+    dep.remot("shuffleWallTime").add(System.nanoTime() - startTime)

Review Comment:
   I'm sorry for this mistake....



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [PR] [GLUTEN-9769][CELEBORN] Use the correct shuffle time metrics for celeborn columnar shuffle [incubator-gluten]

Posted by "marin-ma (via GitHub)" <gi...@apache.org>.
marin-ma commented on PR #9770:
URL: https://github.com/apache/incubator-gluten/pull/9770#issuecomment-2919064917

   > > Thanks for the fix. Can we get the `isSort` from ShuffleDependency in `CelebornShuffleManager` and pass it to `CelebornColumnarShuffleWriter`? Like what we do for `ColumnarShuffleWriter` https://github.com/apache/incubator-gluten/blob/main/backends-velox/src/main/scala/org/apache/spark/shuffle/ColumnarShuffleWriter.scala#L42
   > 
   > I think **get the `isSort` from ShuffleDependency in `CelebornShuffleManager`** is a good idea, and I find ColumnarShuffleWriter can get the dependency reference by `private val dep = handle.dependency.asInstanceOf[ColumnarShuffleDependency[K, V, V]]` , could we add a local variable `isSort` in ColumnarShuffleWriter ?
   > 
   > ```
   >   private val dep = handle.dependency.asInstanceOf[ColumnarShuffleDependency[K, V, V]]
   >   val isSort: Boolean = dep.isSort
   > ```
   
   @wankunde You're right. Then we can make the change simpler.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [PR] [GLUTEN-9769][CELEBORN] Use the correct shuffle time metrics for celeborn columnar shuffle [incubator-gluten]

Posted by "wankunde (via GitHub)" <gi...@apache.org>.
wankunde commented on PR #9770:
URL: https://github.com/apache/incubator-gluten/pull/9770#issuecomment-2918495212

   The UT failed in Ubuntu 20.04 image:
   
   ```log
   The following packages have unmet dependencies:
    maven : Depends: default-jre-headless (>= 2:1.7) but it is not installable or
                     java7-runtime-headless
            Depends: libjansi-java but it is not installable
            Depends: libmaven3-core-java (= 3.6.3-1ubuntu0.1) but it is not going to be installed
            Depends: libwagon-file-java but it is not installable
            Depends: libwagon-http-shaded-java but it is not installable
    openjdk-8-jdk : Depends: openjdk-8-jre (= 8u4[52](https://github.com/apache/incubator-gluten/actions/runs/15315394531/job/43092539753?pr=9770#step:7:53)-ga~us1-0ubuntu1~20.04) but it is not going to be installed
                    Depends: openjdk-8-jdk-headless (= 8u452-ga~us1-0ubuntu1~20.04) but it is not going to be installed
                    Depends: libx11-6 but it is not going to be installed
                    Recommends: libxt-dev but it is not installable
   E: Unable to correct problems, you have held broken packages.
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [PR] [WIP][GLUTEN-9769][CELEBORN] Use the correct shuffle time metrics for celeborn columnar shuffle [incubator-gluten]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #9770:
URL: https://github.com/apache/incubator-gluten/pull/9770#issuecomment-2921073139

   Run Gluten ClickHouse CI on ARM


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [PR] [GLUTEN-9769][CELEBORN] Use the correct shuffle time metrics for celeborn columnar shuffle [incubator-gluten]

Posted by "wForget (via GitHub)" <gi...@apache.org>.
wForget commented on PR #9770:
URL: https://github.com/apache/incubator-gluten/pull/9770#issuecomment-2916082651

   please remove unused import:
   
   ```
   Error:          ^
   Error:  /__w/incubator-gluten/incubator-gluten/backends-velox/src-celeborn/main/scala/org/apache/spark/shuffle/VeloxCelebornColumnarShuffleWriter.scala:22: error: Unused import
   Error:  import org.apache.gluten.config.ReservedKeys
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [PR] [GLUTEN-9769][CELEBORN] Use the correct shuffle time metrics for celeborn columnar shuffle [incubator-gluten]

Posted by "zhouyuan (via GitHub)" <gi...@apache.org>.
zhouyuan commented on code in PR #9770:
URL: https://github.com/apache/incubator-gluten/pull/9770#discussion_r2113428912


##########
backends-velox/src-celeborn/main/scala/org/apache/spark/shuffle/VeloxCelebornColumnarShuffleWriter.scala:
##########
@@ -51,8 +50,6 @@ class VeloxCelebornColumnarShuffleWriter[K, V](
     celebornConf,
     client,
     writeMetrics) {
-  private val isSort = !ReservedKeys.GLUTEN_HASH_SHUFFLE_WRITER.equals(shuffleWriterType)

Review Comment:
   Cc: @marin-ma 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [PR] [GLUTEN-9769][CELEBORN] Use the correct shuffle time metrics for celeborn columnar shuffle [incubator-gluten]

Posted by "marin-ma (via GitHub)" <gi...@apache.org>.
marin-ma commented on PR #9770:
URL: https://github.com/apache/incubator-gluten/pull/9770#issuecomment-2918730899

   Thanks for the fix. Can we get the `isSort` from ShuffleDependency in `CelebornShuffleManager` and pass it to `CelebornColumnarShuffleWriter`? Like what we do for `ColumnarShuffleWriter` https://github.com/apache/incubator-gluten/blob/main/backends-velox/src/main/scala/org/apache/spark/shuffle/ColumnarShuffleWriter.scala#L42


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [PR] [WIP][GLUTEN-9769][CELEBORN] Use the correct shuffle time metrics for celeborn columnar shuffle [incubator-gluten]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #9770:
URL: https://github.com/apache/incubator-gluten/pull/9770#issuecomment-2921125916

   Run Gluten ClickHouse CI on ARM


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [PR] [GLUTEN-9769][CELEBORN] Correct shuffle type [incubator-gluten]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #9770:
URL: https://github.com/apache/incubator-gluten/pull/9770#issuecomment-2912849632

   https://github.com/apache/incubator-gluten/issues/9769


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [PR] [GLUTEN-9769][CELEBORN] Use the correct shuffle time metrics for celeborn columnar shuffle [incubator-gluten]

Posted by "wankunde (via GitHub)" <gi...@apache.org>.
wankunde commented on PR #9770:
URL: https://github.com/apache/incubator-gluten/pull/9770#issuecomment-2916996441

   The UT failed cased by java.lang.OutOfMemoryError: Java heap space
   ```
   	at java.util.Arrays.copyOf(Arrays.java:3236)
   	at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:118)
   	at java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93)
   	at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:153)
   	at org.apache.spark.util.ByteBufferOutputStream.write(ByteBufferOutputStream.scala:41)
   	at java.io.ObjectOutputStream$BlockDataOutputStream.drain(ObjectOutputStream.java:1877)
   	at java.io.ObjectOutputStream$BlockDataOutputStream.write(ObjectOutputStream.java:1848)
   	at java.io.ObjectOutputStream.writeArray(ObjectOutputStream.java:1334)
   	at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1174)
   	at java.io.ObjectOutputStream.writeArray(ObjectOutputStream.java:1378)
   	at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1174)
   	at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
   	at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
   	at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
   	at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
   	at java.io.ObjectOutputStream.writeArray(ObjectOutputStream.java:1378)
   	at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1174)
   	at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348)
   	at org.apache.spark.serializer.JavaSerializationStream.writeObject(JavaSerializer.scala:44)
   	at org.apache.spark.serializer.JavaSerializerInstance.serialize(JavaSerializer.scala:101)
   	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:551)
   	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
   	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
   	at java.lang.Thread.run(Thread.java:750)
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [PR] [GLUTEN-9769][CELEBORN] Use the correct shuffle time metrics for celeborn columnar shuffle [incubator-gluten]

Posted by "zhouyuan (via GitHub)" <gi...@apache.org>.
zhouyuan commented on PR #9770:
URL: https://github.com/apache/incubator-gluten/pull/9770#issuecomment-2917039731

   > The UT failed cased by java.lang.OutOfMemoryError: Java heap space
   > 
   > ```
   > 	at java.util.Arrays.copyOf(Arrays.java:3236)
   > 	at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:118)
   > 	at java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93)
   > 	at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:153)
   > 	at org.apache.spark.util.ByteBufferOutputStream.write(ByteBufferOutputStream.scala:41)
   > 	at java.io.ObjectOutputStream$BlockDataOutputStream.drain(ObjectOutputStream.java:1877)
   > 	at java.io.ObjectOutputStream$BlockDataOutputStream.write(ObjectOutputStream.java:1848)
   > 	at java.io.ObjectOutputStream.writeArray(ObjectOutputStream.java:1334)
   > 	at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1174)
   > 	at java.io.ObjectOutputStream.writeArray(ObjectOutputStream.java:1378)
   > 	at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1174)
   > 	at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
   > 	at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
   > 	at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
   > 	at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
   > 	at java.io.ObjectOutputStream.writeArray(ObjectOutputStream.java:1378)
   > 	at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1174)
   > 	at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348)
   > 	at org.apache.spark.serializer.JavaSerializationStream.writeObject(JavaSerializer.scala:44)
   > 	at org.apache.spark.serializer.JavaSerializerInstance.serialize(JavaSerializer.scala:101)
   > 	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:551)
   > 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
   > 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
   > 	at java.lang.Thread.run(Thread.java:750)
   > ```
   
   The celeborn unit tests is related:
   https://github.com/apache/incubator-gluten/actions/runs/15302482810/job/43046734848?pr=9770


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [PR] [GLUTEN-9769][CELEBORN] Use the correct shuffle time metrics for celeborn columnar shuffle [incubator-gluten]

Posted by "wankunde (via GitHub)" <gi...@apache.org>.
wankunde commented on PR #9770:
URL: https://github.com/apache/incubator-gluten/pull/9770#issuecomment-2921890163

   @zhouyuan @wForget @marin-ma 
   Could help to review this PR again ? Thanks


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [PR] [GLUTEN-9769][CELEBORN] Use the correct shuffle time metrics for celeborn columnar shuffle [incubator-gluten]

Posted by "marin-ma (via GitHub)" <gi...@apache.org>.
marin-ma merged PR #9770:
URL: https://github.com/apache/incubator-gluten/pull/9770


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [PR] [GLUTEN-9769][CELEBORN] Use the correct shuffle time metrics for celeborn columnar shuffle [incubator-gluten]

Posted by "zhouyuan (via GitHub)" <gi...@apache.org>.
zhouyuan commented on PR #9770:
URL: https://github.com/apache/incubator-gluten/pull/9770#issuecomment-2914609029

   @wankunde there are some compile failures, would you please fix it? 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [PR] [GLUTEN-9769][CELEBORN] Use the correct shuffle time metrics for celeborn columnar shuffle [incubator-gluten]

Posted by "wankunde (via GitHub)" <gi...@apache.org>.
wankunde commented on PR #9770:
URL: https://github.com/apache/incubator-gluten/pull/9770#issuecomment-2916529576

   @zhouyuan @wForget Much thanks for all your help


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [PR] [GLUTEN-9769][CELEBORN] Use the correct shuffle time metrics for celeborn columnar shuffle [incubator-gluten]

Posted by "wankunde (via GitHub)" <gi...@apache.org>.
wankunde commented on PR #9770:
URL: https://github.com/apache/incubator-gluten/pull/9770#issuecomment-2916992979

   Retest this please


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org