You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "dafeiw (via GitHub)" <gi...@apache.org> on 2023/06/06 07:20:39 UTC
[GitHub] [arrow] dafeiw opened a new issue, #35928: [Java] RefCnt has gone negative error
dafeiw opened a new issue, #35928:
URL: https://github.com/apache/arrow/issues/35928
### Describe the usage question you have. Please include as many useful details as possible.
Hello!
I have one test suite sending many SQL queries to a local flight server via flight jdbc driver to verify the backend flight producer implementation. The SQL queries are passing if I only run this individual test suite. However, if I run this suite with other various automation tests, the same test suite will fail with ` RefCnt has gone negative` in `getStreamPreparedStatement` method.
Could you please help me understand what scenario this `RefCnt has gone negative` error can happen? It's weird that I don't see this error if I only run the test suite. Thanks in advance.
### Component(s)
Java
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] dafeiw commented on issue #35928: [Java] RefCnt has gone negative error
Posted by "dafeiw (via GitHub)" <gi...@apache.org>.
dafeiw commented on issue #35928:
URL: https://github.com/apache/arrow/issues/35928#issuecomment-1579432390
I got the stacktrace:
`rootAllocator.newChildAllocator(name, Long.MAX_VALUE, Long.MAX_VALUE)`
=>
`Accountant#parent.allocateBytes(Long.MAX_VALUE)`
=>
`locallyHeldMemory.addAndGet(Long.MAX_VALUE)`
After this point, if we allocate an additional buffer, let's say 64.
`Long.MAX_VALUE + 64` will overflow becoming a negative number.
Is it a bug? it's not supposed to allocate more bytes if Long.MAX_VALUE is already allocated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] dafeiw commented on issue #35928: [Java] RefCnt has gone negative error
Posted by "dafeiw (via GitHub)" <gi...@apache.org>.
dafeiw commented on issue #35928:
URL: https://github.com/apache/arrow/issues/35928#issuecomment-1580315689
Thanks @lidavidm , I'm gonna close this thread and raise a bug issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] dafeiw closed issue #35928: [Java] RefCnt has gone negative error
Posted by "dafeiw (via GitHub)" <gi...@apache.org>.
dafeiw closed issue #35928: [Java] RefCnt has gone negative error
URL: https://github.com/apache/arrow/issues/35928
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] lidavidm commented on issue #35928: [Java] RefCnt has gone negative error
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on issue #35928:
URL: https://github.com/apache/arrow/issues/35928#issuecomment-1579319327
Hmm, I'm surprised that worked at all before, since that basically asks the child allocator to reserve `Long.MaxValue` bytes immediately on creation, which would then make further allocations fail (since there's nothing left to be allocated), unless you only used the child allocator for allocations after that point perhaps.
Without being able to debug the program itself, though, it's hard to say in general. Were there other exceptions? It's quite possible that an allocation failing could have led to a bug where something gets freed twice during cleanup.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] dafeiw commented on issue #35928: [Java] RefCnt has gone negative error
Posted by "dafeiw (via GitHub)" <gi...@apache.org>.
dafeiw commented on issue #35928:
URL: https://github.com/apache/arrow/issues/35928#issuecomment-1579131475
Hi @lidavidm, thanks for your reply! The culprit of this issue is that we set `initReservation` to `Long.MaxValue` in one of child allocator.
```
rootAllocator.newChildAllocator("FileWriter", Long.MaxValue, Long.MaxValue)
```
If I changed the `initReservation` to 0. I don't see the `RefCnt has gone negative error` anymore.
I'm still wondering how it relates to the issue. What's the best practice to configure the initReservation parameter? Do we always set it to 0?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] lidavidm commented on issue #35928: [Java] RefCnt has gone negative error
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on issue #35928:
URL: https://github.com/apache/arrow/issues/35928#issuecomment-1578612762
It means a buffer has been released too many times.
If you pause in a debugger, you can inspect the state of the allocator/buffer to get a history of release/retain operations and try to see what happened.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] dafeiw closed issue #35928: [Java] RefCnt has gone negative error
Posted by "dafeiw (via GitHub)" <gi...@apache.org>.
dafeiw closed issue #35928: [Java] RefCnt has gone negative error
URL: https://github.com/apache/arrow/issues/35928
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] dafeiw commented on issue #35928: [Java] RefCnt has gone negative error
Posted by "dafeiw (via GitHub)" <gi...@apache.org>.
dafeiw commented on issue #35928:
URL: https://github.com/apache/arrow/issues/35928#issuecomment-1579453403
It seems child allocator will ask parent allocator do allocation.
I have two child allocators. The first one has `Long.MAX_VALUE` `initReservation`, second one has 0 as `initReservation`.
`rootAllocator.newChildAllocator("A", Long.MAX_VALUE, Long.MAX_VALUE)`
`rootAllocator.newChildAllocator("B", 0, Long.MAX_VALUE)`
```
if (beyondReservation > 0 && parent != null) {
// we need to get memory from our parent.
final long parentRequest = Math.min(beyondReservation, size);
parentOutcome = parent.allocate(parentRequest, updatePeak, forceAllocation, details);
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] lidavidm commented on issue #35928: [Java] RefCnt has gone negative error
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on issue #35928:
URL: https://github.com/apache/arrow/issues/35928#issuecomment-1579460216
Yes, the allocators are nested - this is intended.
However if it's not properly accounting for overflow that is indeed a bug.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org