You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@gluten.apache.org by "wForget (via GitHub)" <gi...@apache.org> on 2024/10/14 10:01:39 UTC

[PR] [GLUTEN-7524][VL][UNIFFLE] Reset rss.row.based configuration of uniffle [incubator-gluten]

wForget opened a new pull request, #7525:
URL: https://github.com/apache/incubator-gluten/pull/7525

   ## What changes were proposed in this pull request?
   
   closes #7524
   
   ## How was this patch tested?
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [PR] [GLUTEN-7524][VL][UNIFFLE] Reset rss.row.based configuration of uniffle [incubator-gluten]

Posted by "wForget (via GitHub)" <gi...@apache.org>.
wForget commented on code in PR #7525:
URL: https://github.com/apache/incubator-gluten/pull/7525#discussion_r1799415998


##########
gluten-uniffle/velox/src/main/java/org/apache/spark/shuffle/gluten/uniffle/UniffleShuffleManager.java:
##########
@@ -57,6 +57,7 @@ public <K, V> ShuffleWriter<K, V> getWriter(
       } else {
         writeMetrics = context.taskMetrics().shuffleWriteMetrics();
       }
+      sparkConf.set(RssSparkConfig.SPARK_RSS_CONFIG_PREFIX + RssSparkConfig.RSS_ROW_BASED.key(), "false");

Review Comment:
   ~~Currently it only affects shuffle writes, I have reset it when creating shuffle writer~~



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [PR] [GLUTEN-7524][VL][UNIFFLE] Reset rss.row.based configuration of uniffle [incubator-gluten]

Posted by "jackylee-ch (via GitHub)" <gi...@apache.org>.
jackylee-ch merged PR #7525:
URL: https://github.com/apache/incubator-gluten/pull/7525


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [PR] [GLUTEN-7524][VL][UNIFFLE] Reset rss.row.based configuration of uniffle [incubator-gluten]

Posted by "summaryzb (via GitHub)" <gi...@apache.org>.
summaryzb commented on code in PR #7525:
URL: https://github.com/apache/incubator-gluten/pull/7525#discussion_r1799397153


##########
gluten-uniffle/velox/src/main/java/org/apache/spark/shuffle/gluten/uniffle/UniffleShuffleManager.java:
##########
@@ -57,6 +57,8 @@ public <K, V> ShuffleWriter<K, V> getWriter(
       } else {
         writeMetrics = context.taskMetrics().shuffleWriteMetrics();
       }
+      sparkConf.set(
+          RssSparkConfig.SPARK_RSS_CONFIG_PREFIX + RssSparkConfig.RSS_ROW_BASED.key(), "false");

Review Comment:
   It's necessary to change initialization of `RssShuffleWriter` in uniffle to inform `WriteBufferManager` use row serialization rather than columnar serialization



##########
gluten-uniffle/velox/src/main/java/org/apache/spark/shuffle/gluten/uniffle/UniffleShuffleManager.java:
##########
@@ -57,6 +57,7 @@ public <K, V> ShuffleWriter<K, V> getWriter(
       } else {
         writeMetrics = context.taskMetrics().shuffleWriteMetrics();
       }
+      sparkConf.set(RssSparkConfig.SPARK_RSS_CONFIG_PREFIX + RssSparkConfig.RSS_ROW_BASED.key(), "false");

Review Comment:
   This conf change may effect other columnar write task



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [PR] [GLUTEN-7524][VL][UNIFFLE] Reset rss.row.based configuration of uniffle [incubator-gluten]

Posted by "jackylee-ch (via GitHub)" <gi...@apache.org>.
jackylee-ch commented on code in PR #7525:
URL: https://github.com/apache/incubator-gluten/pull/7525#discussion_r1799325509


##########
gluten-uniffle/velox/src/main/java/org/apache/spark/shuffle/gluten/uniffle/UniffleShuffleManager.java:
##########
@@ -57,6 +57,8 @@ public <K, V> ShuffleWriter<K, V> getWriter(
       } else {
         writeMetrics = context.taskMetrics().shuffleWriteMetrics();
       }
+      sparkConf.set(
+          RssSparkConfig.SPARK_RSS_CONFIG_PREFIX + RssSparkConfig.RSS_ROW_BASED.key(), "false");

Review Comment:
   It seems we have already set `rss.row.based` to false while initlizing UniflleShuffleManager, do we still need this line?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [PR] [GLUTEN-7524][VL][UNIFFLE] Reset rss.row.based configuration of uniffle [incubator-gluten]

Posted by "wForget (via GitHub)" <gi...@apache.org>.
wForget commented on code in PR #7525:
URL: https://github.com/apache/incubator-gluten/pull/7525#discussion_r1799415998


##########
gluten-uniffle/velox/src/main/java/org/apache/spark/shuffle/gluten/uniffle/UniffleShuffleManager.java:
##########
@@ -57,6 +57,7 @@ public <K, V> ShuffleWriter<K, V> getWriter(
       } else {
         writeMetrics = context.taskMetrics().shuffleWriteMetrics();
       }
+      sparkConf.set(RssSparkConfig.SPARK_RSS_CONFIG_PREFIX + RssSparkConfig.RSS_ROW_BASED.key(), "false");

Review Comment:
   Currently it only affects shuffle writes, I have reset it when creating shuffle writer



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [PR] [GLUTEN-7524][VL][UNIFFLE] Reset rss.row.based configuration of uniffle [incubator-gluten]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #7525:
URL: https://github.com/apache/incubator-gluten/pull/7525#issuecomment-2410669891

   https://github.com/apache/incubator-gluten/issues/7524


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [PR] [GLUTEN-7524][VL][UNIFFLE] Reset rss.row.based configuration of uniffle [incubator-gluten]

Posted by "summaryzb (via GitHub)" <gi...@apache.org>.
summaryzb commented on code in PR #7525:
URL: https://github.com/apache/incubator-gluten/pull/7525#discussion_r1799456712


##########
gluten-uniffle/velox/src/main/java/org/apache/spark/shuffle/gluten/uniffle/UniffleShuffleManager.java:
##########
@@ -57,6 +57,7 @@ public <K, V> ShuffleWriter<K, V> getWriter(
       } else {
         writeMetrics = context.taskMetrics().shuffleWriteMetrics();
       }
+      sparkConf.set(RssSparkConfig.SPARK_RSS_CONFIG_PREFIX + RssSparkConfig.RSS_ROW_BASED.key(), "false");

Review Comment:
   Sounds good



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [PR] [GLUTEN-7524][VL][UNIFFLE] Reset rss.row.based configuration of uniffle [incubator-gluten]

Posted by "wForget (via GitHub)" <gi...@apache.org>.
wForget commented on code in PR #7525:
URL: https://github.com/apache/incubator-gluten/pull/7525#discussion_r1799422529


##########
gluten-uniffle/velox/src/main/java/org/apache/spark/shuffle/gluten/uniffle/UniffleShuffleManager.java:
##########
@@ -57,6 +57,7 @@ public <K, V> ShuffleWriter<K, V> getWriter(
       } else {
         writeMetrics = context.taskMetrics().shuffleWriteMetrics();
       }
+      sparkConf.set(RssSparkConfig.SPARK_RSS_CONFIG_PREFIX + RssSparkConfig.RSS_ROW_BASED.key(), "false");

Review Comment:
   Oh, I see, how about we use a sparkconf.clone ?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [PR] [GLUTEN-7524][VL][UNIFFLE] Reset rss.row.based configuration of uniffle [incubator-gluten]

Posted by "wForget (via GitHub)" <gi...@apache.org>.
wForget commented on code in PR #7525:
URL: https://github.com/apache/incubator-gluten/pull/7525#discussion_r1799338599


##########
gluten-uniffle/velox/src/main/java/org/apache/spark/shuffle/gluten/uniffle/UniffleShuffleManager.java:
##########
@@ -57,6 +57,8 @@ public <K, V> ShuffleWriter<K, V> getWriter(
       } else {
         writeMetrics = context.taskMetrics().shuffleWriteMetrics();
       }
+      sparkConf.set(
+          RssSparkConfig.SPARK_RSS_CONFIG_PREFIX + RssSparkConfig.RSS_ROW_BASED.key(), "false");

Review Comment:
   For example, there are two stages running on the same executor, one shuffle without fallback and one with fallback



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [PR] [GLUTEN-7524][VL][UNIFFLE] Reset rss.row.based configuration of uniffle [incubator-gluten]

Posted by "wForget (via GitHub)" <gi...@apache.org>.
wForget commented on code in PR #7525:
URL: https://github.com/apache/incubator-gluten/pull/7525#discussion_r1799332713


##########
gluten-uniffle/velox/src/main/java/org/apache/spark/shuffle/gluten/uniffle/UniffleShuffleManager.java:
##########
@@ -57,6 +57,8 @@ public <K, V> ShuffleWriter<K, V> getWriter(
       } else {
         writeMetrics = context.taskMetrics().shuffleWriteMetrics();
       }
+      sparkConf.set(
+          RssSparkConfig.SPARK_RSS_CONFIG_PREFIX + RssSparkConfig.RSS_ROW_BASED.key(), "false");

Review Comment:
   Is it possible to create a fallbacked shuffle writer and then create a non-fallbacked shuffle writer?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [PR] [GLUTEN-7524][VL][UNIFFLE] Reset rss.row.based configuration of uniffle [incubator-gluten]

Posted by "jackylee-ch (via GitHub)" <gi...@apache.org>.
jackylee-ch commented on code in PR #7525:
URL: https://github.com/apache/incubator-gluten/pull/7525#discussion_r1799416117


##########
gluten-uniffle/velox/src/main/java/org/apache/spark/shuffle/gluten/uniffle/UniffleShuffleManager.java:
##########
@@ -57,6 +57,8 @@ public <K, V> ShuffleWriter<K, V> getWriter(
       } else {
         writeMetrics = context.taskMetrics().shuffleWriteMetrics();
       }
+      sparkConf.set(
+          RssSparkConfig.SPARK_RSS_CONFIG_PREFIX + RssSparkConfig.RSS_ROW_BASED.key(), "false");

Review Comment:
   Good Catch. Just as @summaryzb said, this may cause some problem when there are two stages, with different `rss.row.based`, trying to create writer at the same time.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [PR] [GLUTEN-7524][VL][UNIFFLE] Reset rss.row.based configuration of uniffle [incubator-gluten]

Posted by "jackylee-ch (via GitHub)" <gi...@apache.org>.
jackylee-ch commented on code in PR #7525:
URL: https://github.com/apache/incubator-gluten/pull/7525#discussion_r1799433195


##########
gluten-uniffle/velox/src/main/java/org/apache/spark/shuffle/gluten/uniffle/UniffleShuffleManager.java:
##########
@@ -57,6 +57,7 @@ public <K, V> ShuffleWriter<K, V> getWriter(
       } else {
         writeMetrics = context.taskMetrics().shuffleWriteMetrics();
       }
+      sparkConf.set(RssSparkConfig.SPARK_RSS_CONFIG_PREFIX + RssSparkConfig.RSS_ROW_BASED.key(), "false");

Review Comment:
   Sounds good to me. @summaryzb any better ideas?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [PR] [GLUTEN-7524][VL][UNIFFLE] Reset rss.row.based configuration of uniffle [incubator-gluten]

Posted by "summaryzb (via GitHub)" <gi...@apache.org>.
summaryzb commented on code in PR #7525:
URL: https://github.com/apache/incubator-gluten/pull/7525#discussion_r1799397153


##########
gluten-uniffle/velox/src/main/java/org/apache/spark/shuffle/gluten/uniffle/UniffleShuffleManager.java:
##########
@@ -57,6 +57,8 @@ public <K, V> ShuffleWriter<K, V> getWriter(
       } else {
         writeMetrics = context.taskMetrics().shuffleWriteMetrics();
       }
+      sparkConf.set(
+          RssSparkConfig.SPARK_RSS_CONFIG_PREFIX + RssSparkConfig.RSS_ROW_BASED.key(), "false");

Review Comment:
   It's necessary to change initialization of `RssShuffleWriter` in uniffle to inform `WriteBufferManager` use row serialization rather than columnar serialization



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org