You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2023/01/18 10:31:16 UTC
[GitHub] [arrow] zhztheplayer opened a new pull request, #33744: GH-33743: [Java] Release outstanding buffers when BaseAllocator is being closed
zhztheplayer opened a new pull request, #33744:
URL: https://github.com/apache/arrow/pull/33744
<!--
Thanks for opening a pull request!
If this is your first pull request you can find detailed information on how
to contribute here:
* [New Contributor's Guide](https://arrow.apache.org/docs/dev/developers/guide/step_by_step/pr_lifecycle.html#reviews-and-merge-of-the-pull-request)
* [Contributing Overview](https://arrow.apache.org/docs/dev/developers/overview.html)
If this is not a [minor PR](https://github.com/apache/arrow/blob/master/CONTRIBUTING.md#Minor-Fixes). Could you open an issue for this pull request on GitHub? https://github.com/apache/arrow/issues/new/choose
Opening GitHub issues ahead of time contributes to the [Openness](http://theapacheway.com/open/#:~:text=Openness%20allows%20new%20users%20the,must%20happen%20in%20the%20open.) of the Apache Arrow project.
Then could you also rename the pull request title in the following format?
GH-${GITHUB_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}
or
MINOR: [${COMPONENT}] ${SUMMARY}
In the case of PARQUET issues on JIRA the title also supports:
PARQUET-${JIRA_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}
-->
### Rationale for this change
See https://github.com/apache/arrow/issues/33743
### What changes are included in this PR?
1. Close outstanding buffer(ledger)s during allocator-close;
2. Report warning messages into log although the outstanding buffers were closed;
3. To help implement the feature, construct a linked-list of buffer ledgers and hold a reference to tail element from buffer alloctor.
### Are these changes tested?
<!--
We typically require tests for all PRs in order to:
1. Prevent the code from being accidentally broken by subsequent changes
4. Serve as another way to document the expected behavior of the code
If tests are not included in your PR, please explain why (for example, are they covered by existing tests)?
-->
1. Current UTs and added UTs.
2. Benchmarking by AllocatorBenchmarks
### Are there any user-facing changes?
<!--
If there are user-facing changes then we may require documentation to be updated before approving the PR.
-->
<!--
If there are any breaking changes to public APIs, please add the `breaking-change` label.
-->
No
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] lidavidm commented on pull request #33744: GH-33743: [Java] Release outstanding buffers when BaseAllocator is being closed
Posted by GitBox <gi...@apache.org>.
lidavidm commented on PR #33744:
URL: https://github.com/apache/arrow/pull/33744#issuecomment-1387485633
Ah, ok. That sounds fine. I would explore the allocation listener further, though...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] zhztheplayer commented on pull request #33744: GH-33743: [Java] Release outstanding buffers when BaseAllocator is being closed
Posted by GitBox <gi...@apache.org>.
zhztheplayer commented on PR #33744:
URL: https://github.com/apache/arrow/pull/33744#issuecomment-1387323433
> What is the purpose? I don't think we want this.
Manual reference counting of Arrow Java library can become very tricky, even becomes the bottle neck of daily development's efficiency in codes of complicated Java systems that relies on Arrow. This is just one way to make Arrow just be usable somehow in that case.
Or a more complete solution I can come up with is to include:
1. A clean-up feature for allocator like this
2. A mechanism to let buffer instances be managed by GC by leveraging e.g. Java weak ref or cleaner toolkit.
I understand performance should be considered emphatically in Arrow, however it's true that we must rely on smarter
strategies of memory management, more or less, in software development including the development around Arrow. I would suggest we provide such a smarter way to Java developers to let them choose between using manual and using auto. All of the current restrictions including the strong leak-checking can be preserved when user chooses the legacy manual way.
Would you think this topic is worth discussion? I am indeed not hurry to get this patch merged until the best solution is shaped. Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #33744: GH-33743: [Java] Release outstanding buffers when BaseAllocator is being closed
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #33744:
URL: https://github.com/apache/arrow/pull/33744#issuecomment-1386829954
:warning: GitHub issue #33743 **has been automatically assigned in GitHub** to PR creator.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] zhztheplayer commented on pull request #33744: GH-33743: [Java] Release outstanding buffers when BaseAllocator is being closed
Posted by GitBox <gi...@apache.org>.
zhztheplayer commented on PR #33744:
URL: https://github.com/apache/arrow/pull/33744#issuecomment-1387359104
Thanks and would you suggest to add a new implementation of interface `BufferAllocator` (e.g. `AutoCleanBufferAllocator`)? If yes I can try start from there.
> the GC does not see into native allocations and you can run out of memory despite having 'free' memory because the GC has not collected the small Java-side objects that hold the large native buffers.
Yes I can image something like that too. What I can try is to reuse the JVM's direct memory counter which can trigger GC when reaching its limit, or rebuild a similar counter like that. Not sure if there are some better solutions but this one is probably OK since it was at least from JVM itself.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] lidavidm commented on pull request #33744: GH-33743: [Java] Release outstanding buffers when BaseAllocator is being closed
Posted by GitBox <gi...@apache.org>.
lidavidm commented on PR #33744:
URL: https://github.com/apache/arrow/pull/33744#issuecomment-1387373744
I don't think it needs a new interface.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-33743: [Java] Release outstanding buffers when BaseAllocator is being closed [arrow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #33744:
URL: https://github.com/apache/arrow/pull/33744#issuecomment-2501749784
:warning: GitHub issue #33743 **has no components**, please add labels for components.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #33744: GH-33743: [Java] Release outstanding buffers when BaseAllocator is being closed
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #33744:
URL: https://github.com/apache/arrow/pull/33744#issuecomment-1386829877
* Closes: #33743
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] zhztheplayer commented on pull request #33744: GH-33743: [Java] Release outstanding buffers when BaseAllocator is being closed
Posted by GitBox <gi...@apache.org>.
zhztheplayer commented on PR #33744:
URL: https://github.com/apache/arrow/pull/33744#issuecomment-1387480855
> I don't think it needs a new interface.
I meant a new implementation of the interface, e.g.
`class AutoCleanBufferAllocator implements BufferAllocator`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] amol- commented on pull request #33744: GH-33743: [Java] Release outstanding buffers when BaseAllocator is being closed
Posted by "amol- (via GitHub)" <gi...@apache.org>.
amol- commented on PR #33744:
URL: https://github.com/apache/arrow/pull/33744#issuecomment-1490643308
Closing because it has been untouched for a while, in case it's still relevant feel free to reopen and move it forward 👍
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] amol- closed pull request #33744: GH-33743: [Java] Release outstanding buffers when BaseAllocator is being closed
Posted by "amol- (via GitHub)" <gi...@apache.org>.
amol- closed pull request #33744: GH-33743: [Java] Release outstanding buffers when BaseAllocator is being closed
URL: https://github.com/apache/arrow/pull/33744
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] lidavidm commented on pull request #33744: GH-33743: [Java] Release outstanding buffers when BaseAllocator is being closed
Posted by GitBox <gi...@apache.org>.
lidavidm commented on PR #33744:
URL: https://github.com/apache/arrow/pull/33744#issuecomment-1387331312
This patch forces everyone to take the opposite tradeoff, however.
Some sort of nursery that auto-closes buffers may be useful, but I think it should be separate from the base allocator. (The allocator also has a 'listener' to be notified of allocation/deallocation events; possibly you could build this mechanism on top of that.) And nondeterministic collection of buffers is liable to bite you; the GC does not see into native allocations and you can run out of memory despite having 'free' memory because the GC has not collected the small Java-side objects that hold the large native buffers.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] zhztheplayer commented on pull request #33744: GH-33743: [Java] Release outstanding buffers when BaseAllocator is being closed
Posted by GitBox <gi...@apache.org>.
zhztheplayer commented on PR #33744:
URL: https://github.com/apache/arrow/pull/33744#issuecomment-1387500741
I see. I'll find out to what extent can allocation listener to be leveraged in this feature. Thanks for the suggestion.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org