You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2022/11/16 04:11:29 UTC

[GitHub] [spark] warrenzhu25 opened a new pull request, #38668: [SPARK-41153][CORE] Log migrated shuffle data size and migration time

warrenzhu25 opened a new pull request, #38668:
URL: https://github.com/apache/spark/pull/38668

   ### What changes were proposed in this pull request?
   Log migrated shuffle data size and migration time
   
   ### Why are the changes needed?
   Get info about migrated shuffle data size and migration time
   
   ### Does this PR introduce _any_ user-facing change?
   No
   
   ### How was this patch tested?
   Manually tested
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] dongjoon-hyun closed pull request #38668: [SPARK-41153][CORE] Log migrated shuffle data size and migration time

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun closed pull request #38668: [SPARK-41153][CORE] Log migrated shuffle data size and migration time
URL: https://github.com/apache/spark/pull/38668


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #38668: [SPARK-41153][CORE] Log migrated shuffle data size and migration time

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on PR #38668:
URL: https://github.com/apache/spark/pull/38668#issuecomment-1317642173

   Can one of the admins verify this patch?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] holdenk commented on pull request #38668: [SPARK-41153][CORE] Log migrated shuffle data size and migration time

Posted by GitBox <gi...@apache.org>.
holdenk commented on PR #38668:
URL: https://github.com/apache/spark/pull/38668#issuecomment-1329896793

   I think logging this info is a good idea +1


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] mridulm commented on a diff in pull request #38668: [SPARK-41153][CORE] Log migrated shuffle data size and migration time

Posted by GitBox <gi...@apache.org>.
mridulm commented on code in PR #38668:
URL: https://github.com/apache/spark/pull/38668#discussion_r1038739883


##########
core/src/main/scala/org/apache/spark/storage/BlockManagerDecommissioner.scala:
##########
@@ -125,7 +126,10 @@ private[storage] class BlockManagerDecommissioner(
                   logDebug(s"Migrated sub-block $blockId")
                 }
               }
-              logInfo(s"Migrated $shuffleBlockInfo to $peer")
+              lazy val endTime = System.currentTimeMillis()
+              lazy val totalBlockSize = Utils.bytesToString(blocks.map(b => b._2.size()).sum)
+              logInfo(s"Migrated $shuffleBlockInfo (" +
+                s"size: $totalBlockSize) to $peer in ${endTime - startTime} ms")

Review Comment:
   Yes, but you can avoid all the `lazy val` code.



##########
core/src/main/scala/org/apache/spark/storage/BlockManagerDecommissioner.scala:
##########
@@ -125,7 +126,10 @@ private[storage] class BlockManagerDecommissioner(
                   logDebug(s"Migrated sub-block $blockId")
                 }
               }
-              logInfo(s"Migrated $shuffleBlockInfo to $peer")
+              lazy val endTime = System.currentTimeMillis()
+              lazy val totalBlockSize = Utils.bytesToString(blocks.map(b => b._2.size()).sum)
+              logInfo(s"Migrated $shuffleBlockInfo (" +
+                s"size: $totalBlockSize) to $peer in ${endTime - startTime} ms")

Review Comment:
   Yes, but you can avoid all the `lazy val` code.
   It is not important though.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] LuciferYang commented on a diff in pull request #38668: [SPARK-41153][CORE] Log migrated shuffle data size and migration time

Posted by GitBox <gi...@apache.org>.
LuciferYang commented on code in PR #38668:
URL: https://github.com/apache/spark/pull/38668#discussion_r1023590958


##########
core/src/main/scala/org/apache/spark/storage/BlockManagerDecommissioner.scala:
##########
@@ -125,7 +127,11 @@ private[storage] class BlockManagerDecommissioner(
                   logDebug(s"Migrated sub-block $blockId")
                 }
               }
-              logInfo(s"Migrated $shuffleBlockInfo to $peer")
+              val endTime = System.nanoTime()
+              val duration = Duration(endTime - startTime, NANOSECONDS)
+              val totalBlockSize = Utils.bytesToString(blocks.map(b => b._2.size()).sum)

Review Comment:
   hmm... if `duration` and `totalBlockSize` are defined as `val`, they will be calculated even if `log.isInfoEnabled` is `false`
   
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] Ngone51 commented on a diff in pull request #38668: [SPARK-41153][CORE] Log migrated shuffle data size and migration time

Posted by GitBox <gi...@apache.org>.
Ngone51 commented on code in PR #38668:
URL: https://github.com/apache/spark/pull/38668#discussion_r1036658717


##########
core/src/main/scala/org/apache/spark/storage/BlockManagerDecommissioner.scala:
##########
@@ -125,7 +126,10 @@ private[storage] class BlockManagerDecommissioner(
                   logDebug(s"Migrated sub-block $blockId")
                 }
               }
-              logInfo(s"Migrated $shuffleBlockInfo to $peer")
+              lazy val endTime = System.currentTimeMillis()
+              lazy val totalBlockSize = Utils.bytesToString(blocks.map(b => b._2.size()).sum)
+              logInfo(s"Migrated $shuffleBlockInfo (" +
+                s"size: $totalBlockSize) to $peer in ${endTime - startTime} ms")

Review Comment:
   `logInfo` already does that inside. @mridulm 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] mridulm commented on pull request #38668: [SPARK-41153][CORE] Log migrated shuffle data size and migration time

Posted by GitBox <gi...@apache.org>.
mridulm commented on PR #38668:
URL: https://github.com/apache/spark/pull/38668#issuecomment-1318223332

   +CC @dongjoon-hyun, @holdenk 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] Ngone51 commented on a diff in pull request #38668: [SPARK-41153][CORE] Log migrated shuffle data size and migration time

Posted by GitBox <gi...@apache.org>.
Ngone51 commented on code in PR #38668:
URL: https://github.com/apache/spark/pull/38668#discussion_r1035963605


##########
core/src/main/scala/org/apache/spark/storage/BlockManagerDecommissioner.scala:
##########
@@ -125,7 +126,10 @@ private[storage] class BlockManagerDecommissioner(
                   logDebug(s"Migrated sub-block $blockId")
                 }
               }
-              logInfo(s"Migrated $shuffleBlockInfo to $peer")
+              lazy val endTime = System.currentTimeMillis()
+              lazy val totalBlockSize = Utils.bytesToString(blocks.map(b => b._2.size()).sum)
+              logInfo(s"Migrated $shuffleBlockInfo (" +
+                s"size: $totalBlockSize) to $peer in ${endTime - startTime} ms")

Review Comment:
   How about
   ```suggestion
                 logInfo(s"Migrated $shuffleBlockInfo (" +
                   s"size: ${Utils.bytesToString(blocks.map(b => b._2.size()).sum)}) to $peer in ${System.currentTimeMillis() - startTime} ms")
   ```
   to avoid those two lazy variables?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] dongjoon-hyun commented on pull request #38668: [SPARK-41153][CORE] Log migrated shuffle data size and migration time

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on PR #38668:
URL: https://github.com/apache/spark/pull/38668#issuecomment-1336099785

   Merged to master for Apache Spark 3.4.0. Thank you, @warrenzhu25 and all!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] warrenzhu25 commented on pull request #38668: [SPARK-41153][CORE] Log migrated shuffle data size and migration time

Posted by GitBox <gi...@apache.org>.
warrenzhu25 commented on PR #38668:
URL: https://github.com/apache/spark/pull/38668#issuecomment-1329449528

   @dongjoon-hyun Could you help take a look?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] LuciferYang commented on a diff in pull request #38668: [SPARK-41153][CORE] Log migrated shuffle data size and migration time

Posted by GitBox <gi...@apache.org>.
LuciferYang commented on code in PR #38668:
URL: https://github.com/apache/spark/pull/38668#discussion_r1023595740


##########
core/src/main/scala/org/apache/spark/storage/BlockManagerDecommissioner.scala:
##########
@@ -125,7 +127,11 @@ private[storage] class BlockManagerDecommissioner(
                   logDebug(s"Migrated sub-block $blockId")
                 }
               }
-              logInfo(s"Migrated $shuffleBlockInfo to $peer")
+              val endTime = System.nanoTime()

Review Comment:
   Why do we need take `nanoTime()` then convert it `toMillis`?  What is the problem of directly getting `System.currentTimeMillis()` for calculation?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] warrenzhu25 commented on pull request #38668: [SPARK-41153][CORE] Log migrated shuffle data size and migration time

Posted by GitBox <gi...@apache.org>.
warrenzhu25 commented on PR #38668:
URL: https://github.com/apache/spark/pull/38668#issuecomment-1329897249

   > Could you add the example log message into the PR description, @warrenzhu25 ?
   
   Updated.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] holdenk commented on a diff in pull request #38668: [SPARK-41153][CORE] Log migrated shuffle data size and migration time

Posted by GitBox <gi...@apache.org>.
holdenk commented on code in PR #38668:
URL: https://github.com/apache/spark/pull/38668#discussion_r1034166934


##########
core/src/main/scala/org/apache/spark/storage/BlockManagerDecommissioner.scala:
##########
@@ -125,7 +127,11 @@ private[storage] class BlockManagerDecommissioner(
                   logDebug(s"Migrated sub-block $blockId")
                 }
               }
-              logInfo(s"Migrated $shuffleBlockInfo to $peer")
+              val endTime = System.nanoTime()
+              val duration = Duration(endTime - startTime, NANOSECONDS)
+              val totalBlockSize = Utils.bytesToString(blocks.map(b => b._2.size()).sum)

Review Comment:
   True, it's probably not a lot of compute though. We could use a lazy val here though.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org