You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2022/02/21 11:41:29 UTC

[GitHub] [spark] Myasuka opened a new pull request #35600: [SPARK-38275] Include the writeBatch's memoy usage as the total memory usage of RocksDB state store

Myasuka opened a new pull request #35600:
URL: https://github.com/apache/spark/pull/35600


   
   ### What changes were proposed in this pull request?
   Include the writeBatch's memoy usage as the total memory usage of RocksDB state store.
   
   Moreover, this PR also includes a hotfix to clear write batch just after `commit`.
   
   ### Why are the changes needed?
   As the memory used by WriteBatch has no limit, the actual memory usage could be much larger than previously stats without considering the memoy used by write batch.
   
   ### Does this PR introduce _any_ user-facing change?
   NO
   
   ### How was this patch tested?
   Test via running jobs with large micro-batch.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] HeartSaVioR commented on pull request #35600: [SPARK-38275][SS] Include the writeBatch's memory usage as the total memory usage of RocksDB state store

Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on pull request #35600:
URL: https://github.com/apache/spark/pull/35600#issuecomment-1050475097


   Thanks! Merging to master.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] Myasuka commented on pull request #35600: [SPARK-38275] Include the writeBatch's memoy usage as the total memory usage of RocksDB state store

Posted by GitBox <gi...@apache.org>.
Myasuka commented on pull request #35600:
URL: https://github.com/apache/spark/pull/35600#issuecomment-1047386349


   @HeartSaVioR Thanks for the advice, I have created another ticket SPARK-38277 to focus on the write batch memory clearing. And I will change this PR to only focus on memory usage.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] HeartSaVioR commented on pull request #35600: [SPARK-38275][SS] Include the writeBatch's memory usage as the total memory usage of RocksDB state store

Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on pull request #35600:
URL: https://github.com/apache/spark/pull/35600#issuecomment-1050475884


   Thanks @Myasuka for your contribution! I merged this to master.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #35600: [SPARK-38275] Include the writeBatch's memoy usage as the total memory usage of RocksDB state store

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #35600:
URL: https://github.com/apache/spark/pull/35600#issuecomment-1047424503


   Can one of the admins verify this patch?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] HeartSaVioR commented on pull request #35600: [SPARK-38275] Include the writeBatch's memoy usage as the total memory usage of RocksDB state store

Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on pull request #35600:
URL: https://github.com/apache/spark/pull/35600#issuecomment-1047377279


   Thanks for the contribution! Looks like the test failure is related to the change. Could you please look into it? 
   Also, we could split the PR down to multiple PRs to address separate topics 1) about the information on memory usage 2) clear write batch after commit. I guess the test failure may be related to second one.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] HeartSaVioR closed pull request #35600: [SPARK-38275][SS] Include the writeBatch's memory usage as the total memory usage of RocksDB state store

Posted by GitBox <gi...@apache.org>.
HeartSaVioR closed pull request #35600:
URL: https://github.com/apache/spark/pull/35600


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org