You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "aqni (via GitHub)" <gi...@apache.org> on 2024/05/14 09:37:24 UTC
[I] Memory Leak about splitAndTransfer [arrow]
aqni opened a new issue, #41648:
URL: https://github.com/apache/arrow/issues/41648
### Describe the bug, including details regarding any error messages, version, and platform.
version: 16.0.0
I am currently working on a project using Apache Arrow in Java and I have encountered a potential memory leak issue.
run the showMemoryLeak method
```java
@Test
public void showMemoryLeak() {
RootAllocator allocator = new RootAllocator(Long.MAX_VALUE);
ValueVector slice;
try (BufferAllocator child = allocator.newChildAllocator("child", 0, Long.MAX_VALUE)) {
try (IntVector vector = new IntVector("vector", child)) {
vector.setSafe(0, 1);
vector.setValueCount(1);
TransferPair transferPair = vector.getTransferPair(allocator);
transferPair.splitAndTransfer(0, 1);
slice = transferPair.getTo();
}
}
slice.close();
}
```
then a exception is thrown
```
java.lang.IllegalStateException: Attempting operation on allocator when allocator is closed.
Allocator(child) 0/0/16384/9223372036854775807 (res/actual/peak/limit)
at org.apache.arrow.memory.BaseAllocator.assertOpen(BaseAllocator.java:172)
at org.apache.arrow.memory.BufferLedger.decrement(BufferLedger.java:149)
at org.apache.arrow.memory.BufferLedger.release(BufferLedger.java:125)
at org.apache.arrow.memory.BufferLedger.release(BufferLedger.java:105)
at org.apache.arrow.vector.BaseValueVector.releaseBuffer(BaseValueVector.java:114)
at org.apache.arrow.vector.BaseFixedWidthVector.clear(BaseFixedWidthVector.java:247)
at org.apache.arrow.vector.BaseFixedWidthVector.close(BaseFixedWidthVector.java:238)
at cn.edu.tsinghua.iginx.parquet.util.arrow.ArrowVectorsTest.showMemoryLeak(ArrowVectorsTest.java:30)
...
```
If I continue to execute transfer() explicitly, there will be no memory leak, as shown below
```java
@Test
public void showMemoryLeak() {
RootAllocator allocator = new RootAllocator(Long.MAX_VALUE);
ValueVector slice;
try (BufferAllocator child = allocator.newChildAllocator("child", 0, Long.MAX_VALUE)) {
try (IntVector vector = new IntVector("vector", child)) {
vector.setSafe(0, 1);
vector.setValueCount(1);
TransferPair transferPair = vector.getTransferPair(allocator);
transferPair.splitAndTransfer(0, 1);
try(ValueVector to = transferPair.getTo()) {
TransferPair transferPair2 = to.getTransferPair(allocator);
transferPair2.transfer();
slice = transferPair2.getTo();
}
}
}
slice.close();
}
```
This problem is caused by the fact that when startIndex == 0, splitAndTransfer() does not transfer the validity buffer.
### Component(s)
Java
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Java] Memory Leak about splitAndTransfer [arrow]
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on issue #41648:
URL: https://github.com/apache/arrow/issues/41648#issuecomment-2144952617
Issue resolved by pull request 41898
https://github.com/apache/arrow/pull/41898
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Java] Memory Leak about splitAndTransfer [arrow]
Posted by "vibhatha (via GitHub)" <gi...@apache.org>.
vibhatha commented on issue #41648:
URL: https://github.com/apache/arrow/issues/41648#issuecomment-2141038682
@aqni could you please try the following
```java
@Test
public void testSplitAndTransferFixedWithVector2() {
RootAllocator allocator = new RootAllocator(Long.MAX_VALUE);
try (BufferAllocator child = allocator.newChildAllocator("child", 0, Long.MAX_VALUE)) {
try (IntVector vector = new IntVector("vector", child)) {
vector.setSafe(0, 1);
vector.setValueCount(1);
TransferPair transferPair = vector.getTransferPair(allocator);
transferPair.splitAndTransfer(0, 1);
try (ValueVector slice = transferPair.getTo()) {
// no-op try-with-resource
}
}
}
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Memory Leak about splitAndTransfer [arrow]
Posted by "aqni (via GitHub)" <gi...@apache.org>.
aqni commented on issue #41648:
URL: https://github.com/apache/arrow/issues/41648#issuecomment-2138595992
hello
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Java] Memory Leak about splitAndTransfer [arrow]
Posted by "vibhatha (via GitHub)" <gi...@apache.org>.
vibhatha commented on issue #41648:
URL: https://github.com/apache/arrow/issues/41648#issuecomment-2141272194
I think rather than temporarily fixing it here. Adding that util and adding test cases would be sustainable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Java] Memory Leak about splitAndTransfer [arrow]
Posted by "vibhatha (via GitHub)" <gi...@apache.org>.
vibhatha commented on issue #41648:
URL: https://github.com/apache/arrow/issues/41648#issuecomment-2141271100
https://github.com/apache/arrow/issues/41897
In here we will add a single util.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Java] Memory Leak about splitAndTransfer [arrow]
Posted by "aqni (via GitHub)" <gi...@apache.org>.
aqni commented on issue #41648:
URL: https://github.com/apache/arrow/issues/41648#issuecomment-2141118416
Thank you for replying.
Just a reminder，the problem seems to also appear in other classes, such as
1. `BaseLargeVariableWidthVector`
3. `LargeListVector`
4. `ListVector`
5. `MapVector`
6. `StructVector.java`
7. ……
It seems that these were not considered in your PR
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Java] Memory Leak about splitAndTransfer [arrow]
Posted by "aqni (via GitHub)" <gi...@apache.org>.
aqni commented on issue #41648:
URL: https://github.com/apache/arrow/issues/41648#issuecomment-2141130703
> @aqni could you please try the following
>
> ```java
> @Test
> public void testSplitAndTransferFixedWithVector2() {
> RootAllocator allocator = new RootAllocator(Long.MAX_VALUE);
> try (BufferAllocator child = allocator.newChildAllocator("child", 0, Long.MAX_VALUE)) {
> try (IntVector vector = new IntVector("vector", child)) {
> vector.setSafe(0, 1);
> vector.setValueCount(1);
> TransferPair transferPair = vector.getTransferPair(allocator);
> transferPair.splitAndTransfer(0, 1);
> try (ValueVector slice = transferPair.getTo()) {
> // no-op try-with-resource
> }
> }
> }
> }
> ```
This example will not throw an exception. But I need to transfer ownership between different allocators, and this example does not reflect the problem very well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Memory Leak about splitAndTransfer [arrow]
Posted by "vibhatha (via GitHub)" <gi...@apache.org>.
vibhatha commented on issue #41648:
URL: https://github.com/apache/arrow/issues/41648#issuecomment-2139612385
@aqni I can reproduce this. I am working on a fix. I will keep you posted.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Memory Leak about splitAndTransfer [arrow]
Posted by "vibhatha (via GitHub)" <gi...@apache.org>.
vibhatha commented on issue #41648:
URL: https://github.com/apache/arrow/issues/41648#issuecomment-2138647185
@aqni I will take a look later today.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Java] Memory Leak about splitAndTransfer [arrow]
Posted by "vibhatha (via GitHub)" <gi...@apache.org>.
vibhatha commented on issue #41648:
URL: https://github.com/apache/arrow/issues/41648#issuecomment-2141269185
I already created a PR for this issue. I will make sure to address this in other places. We are planning on using a single utility across these classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Java] Memory Leak about splitAndTransfer [arrow]
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm closed issue #41648: [Java] Memory Leak about splitAndTransfer
URL: https://github.com/apache/arrow/issues/41648
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org