You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2022/11/30 23:54:17 UTC
[GitHub] [spark] Kimahriman opened a new pull request, #38853: [SPARK-41339][SQL] Close and recreate RocksDB write batch instead of just clearing
Kimahriman opened a new pull request, #38853:
URL: https://github.com/apache/spark/pull/38853
<!--
Thanks for sending a pull request! Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://spark.apache.org/contributing.html
2. Ensure you have added or run the appropriate tests for your PR: https://spark.apache.org/developer-tools.html
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][SPARK-XXXX] Your PR title ...'.
4. Be sure to keep the PR description updated to reflect all changes.
5. Please write your PR title to summarize what this PR proposes.
6. If possible, provide a concise example to reproduce the issue for a faster review.
7. If you want to add a new configuration, please read the guideline first for naming configurations in
'core/src/main/scala/org/apache/spark/internal/config/ConfigEntry.scala'.
8. If you want to add or modify an error type or message, please read the guideline first in
'core/src/main/resources/error/README.md'.
-->
### What changes were proposed in this pull request?
<!--
Please clarify what changes you are proposing. The purpose of this section is to outline the changes and how this PR fixes the issue.
If possible, please consider writing useful notes for better and faster reviews in your PR. See the examples below.
1. If you refactor some codes with changing classes, showing the class hierarchy will help reviewers.
2. If you fix some SQL features, you can provide some references of other DBMSes.
3. If there is design documentation, please add the link.
4. If there is a discussion in the mailing list, please add the link.
-->
Instead of just calling `writeBatch.clear`, close the write batch and recreate it.
### Why are the changes needed?
<!--
Please clarify why the changes are needed. For instance,
1. If you propose a new API, clarify the use case for a new API.
2. If you fix a bug, you can clarify why it is a bug.
-->
A RocksDB `WriteBatch` (and by extension `WriteBatchWithIndex`) stores it's underlying data in a `std::string`. Why? I'm not sure. But after a partition is finished, `writeBatch.clear()` is called (somewhat indirectly through a call to `store.abort`), presumably clearing the data in the `WriteBatch`. This calls `std::string::clear` followed by `std::string::resize` underneath the hood. However, neither of these two things actual reclaims native memory. All the memory allocated for expanding the string when adding data to the `WriteBatch` will be there until the `std::string` is deallocated, which in this case means deleting the `WriteBatch`. This leads to native memory accumulation on an executor and it executes several partitions consecutively, which would happen when your total executor cores is less than your shuffle partitions for your stateful stream. So instead of just calling `writeBatch.clear()`, close the `WriteStream` and create a new one to free up the native memory.
### Does this PR introduce _any_ user-facing change?
<!--
Note that it means *any* user-facing change including all aspects such as the documentation fix.
If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible.
If possible, please also clarify if this is a user-facing change compared to the released Spark versions or within the unreleased branches such as master.
If no, write 'No'.
-->
Fix for excess native memory usage.
### How was this patch tested?
<!--
If tests were added, say they were added here. Please make sure to add some test cases that check the changes thoroughly including negative and positive cases if possible.
If it was tested in a way different from regular unit tests, please clarify how you tested step by step, ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future.
If tests were not added, please describe why they were not added and/or why it was difficult to add.
If benchmark tests were added, please run the benchmarks in GitHub Actions for the consistent environment, and the instructions could accord to: https://spark.apache.org/developer-tools.html#github-workflow-benchmarks.
-->
Existing UTs, not sure how to test for memory usage.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on pull request #38853: [SPARK-41339][SQL] Close and recreate RocksDB write batch instead of just clearing
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on PR #38853:
URL: https://github.com/apache/spark/pull/38853#issuecomment-1333077999
It's not intended. It looks like we missed to deal with ticket. Anyway we clear the writebatch before starting a new task for next microbatch, but I see issue when an executor handles multiple tasks for stateful operation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on pull request #38853: [SPARK-41339][SQL] Close and recreate RocksDB write batch instead of just clearing
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on PR #38853:
URL: https://github.com/apache/spark/pull/38853#issuecomment-1333053315
Nice finding! Looks like your explanation makes sense. I've quickly googled and there is `reserve` which may shrink the memory as desired but doesn't seem to be guaranteed. (And it's something we have to fix the code in RocksDB which they may have their own valid reason to do so.) So I agree this is the best approach we can do, and the fix is straightforward.
Just a couple of questions:
1) Have you run the fix with your production workload for a while and see there is no longer the same memory issue?
2) Did https://github.com/apache/spark/commit/b8b1fbc21c66348d25be3404d3f61099f2a7a9b5 help to figure out the "sort of leaking" memory, or did it report simply the resized memory size and didn't disclose the memory issue?
Otherwise the fix itself looks OK to me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Kimahriman commented on pull request #38853: [SPARK-41339][SQL] Close and recreate RocksDB write batch instead of just clearing
Posted by GitBox <gi...@apache.org>.
Kimahriman commented on PR #38853:
URL: https://github.com/apache/spark/pull/38853#issuecomment-1333068984
> 1. Have you run the fix with your production workload for a while and see there is no longer the same memory issue?
Yes I confirmed that today. I have executors with a 25 GiB heap that I was seeing get to 90+ GiB of resident memory within a single one of our batches. After trying out the fix it held steady at ~25-30 GiB resident memory.
> 2. Did [b8b1fbc](https://github.com/apache/spark/commit/b8b1fbc21c66348d25be3404d3f61099f2a7a9b5) help to figure out the "sort of leaking" memory, or did it report simply the resized memory size and didn't disclose the memory issue?
Didn't really help, because I could only really see the stats write after the commit when the write batch was supposed to be full. Wasn't really anyway to see in those stats that it wasn't being cleaned up. And yeah after clearing it just reports the resized memory not the actually used memory.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on pull request #38853: [SPARK-41339][SQL] Close and recreate RocksDB write batch instead of just clearing
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on PR #38853:
URL: https://github.com/apache/spark/pull/38853#issuecomment-1333287478
Thanks! Merging to master/3.3. ([SPARK-38277](https://issues.apache.org/jira/browse/SPARK-38277) can/should be handled separately.)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR closed pull request #38853: [SPARK-41339][SQL] Close and recreate RocksDB write batch instead of just clearing
Posted by GitBox <gi...@apache.org>.
HeartSaVioR closed pull request #38853: [SPARK-41339][SQL] Close and recreate RocksDB write batch instead of just clearing
URL: https://github.com/apache/spark/pull/38853
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Kimahriman commented on pull request #38853: [SPARK-41339][SQL] Close and recreate RocksDB write batch instead of just clearing
Posted by GitBox <gi...@apache.org>.
Kimahriman commented on PR #38853:
URL: https://github.com/apache/spark/pull/38853#issuecomment-1333073885
One question I did have, as I started digging into this I thought the problem was simply related to https://issues.apache.org/jira/browse/SPARK-38277, but as I dug into it I realized that `writeBatch.clear` is called, but by a task completion listener that calls `store.abort` [here](https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/package.scala#L101-L103). I wasn't sure if this was intentional or not, seems like a slightly awkward way to clear things after a partition completes. If it was intentional that issue can probably be closed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Kimahriman commented on pull request #38853: [SPARK-41339][SQL] Close and recreate RocksDB write batch instead of just clearing
Posted by GitBox <gi...@apache.org>.
Kimahriman commented on PR #38853:
URL: https://github.com/apache/spark/pull/38853#issuecomment-1333651886
> Just to give you more context for your previous comment ([#38853 (comment)](https://github.com/apache/spark/pull/38853#issuecomment-1333073885))...
>
> We have two different set of code path, 1) two physical nodes for one stateful operator (streaming aggregation) 2) one physical node for one stateful operator (others). For the latter, it only initializes read-write state store, and at the task completion, it only calls abort if the task failed to do commit. For the former, it initializes read only state store as well, which we call abort at the task completion to clean up the resource (NOTE: not to rollback as this is read-only. The name is unfortunately due to compatibility issue during the addition of new interface. See [21413b7](https://github.com/apache/spark/commit/21413b7dd4e19f725b21b92cddfbe73d1b381a05)).
>
> It is safe for read-only state store to call abort() even there is another read-write store referring the same, because read-write store would have completed to call commit() if the task works correctly and we expect (effectively) no-op from calling abort() after commit().
So the abort is there to basically make sure any writes to the read only state store get reverted unless a downstream read-write instance has already committed changes. I know there are a few different code paths for stateful things (sessions, flatmapgroupswithstate, etc), do we know if that abort call covers all the cases for https://issues.apache.org/jira/browse/SPARK-38277? Or do some of the other code paths not end up using that read-only path at all? Or would it be better just do explicitly clear the write batch anyway even if it is redundant with the abort call?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Kimahriman commented on pull request #38853: [SPARK-41339][SQL] Close and recreate RocksDB write batch instead of just clearing
Posted by GitBox <gi...@apache.org>.
Kimahriman commented on PR #38853:
URL: https://github.com/apache/spark/pull/38853#issuecomment-1333646333
> (@Kimahriman Looks like your git is not correctly set in your dev. I manually changed the author of commit based on your mail address in dev@ mailing list.)
Thanks, I did this on a new machine and forgot I haven't set that up yet.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on pull request #38853: [SPARK-41339][SQL] Close and recreate RocksDB write batch instead of just clearing
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on PR #38853:
URL: https://github.com/apache/spark/pull/38853#issuecomment-1333166532
Just to give you more context for your previous comment (https://github.com/apache/spark/pull/38853#issuecomment-1333073885)...
We have two different set of code path, 1) two physical nodes for one stateful operator (streaming aggregation) 2) one physical node for one stateful operator (others). For the latter, it only initializes read-write state store, and at the task completion, it only calls abort if the task failed to do commit. For the former, it initializes read only state store as well, which we call abort at the task completion to clean up the resource (NOTE: not to rollback as this is read-only. The name is unfortunately due to compatibility issue during the addition of new interface. See https://github.com/apache/spark/commit/21413b7dd4e19f725b21b92cddfbe73d1b381a05).
It is safe for read-only state store to call abort() even there is another read-write store referring the same, because read-write store would have completed to call commit() if the task works correctly and we expect (effectively) no-op from calling abort() after commit().
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on pull request #38853: [SPARK-41339][SQL] Close and recreate RocksDB write batch instead of just clearing
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on PR #38853:
URL: https://github.com/apache/spark/pull/38853#issuecomment-1333291415
(@Kimahriman Looks like your git is not correctly set in your dev. I manually changed the author of commit based on your mail address in dev@ mailing list.)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org