You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@arrow.apache.org by Daniel Hsu <da...@bytedance.com> on 2021/09/13 07:17:32 UTC
[Java] How to detect if there are dangling vectors that we need call close() for?
Hi All,
Is there a way to figure out which vectors I haven't called close() on yet? My concern is that if I have a bug in my code where I accidentally forget to call close() for many vectors, the space occupied by these vectors could grow forever and crash my system.
How do I detect these vector "memory leaks"?
Best,
Daniel
Re: [Java] How to detect if there are dangling vectors that we need
call close() for?
Posted by Benjamin Wilhelm <be...@knime.com>.
Hi Daniel,
When the JVM terminates, all allocated memory is given back to the OS. The
vectors and buffers lie somewhere in the off-heap portion of the allocated
memory of the JVM (search for off-heap memory in Java).
Therefore, you do not need to close anything after a Java process dies.
Best,
Benjamin
On Fri, Sep 17, 2021 at 6:42 AM Daniel Hsu <da...@bytedance.com> wrote:
> Hi Benjamin,
>
> Thanks for the pointers. Is there a recommended way to clean up servers
> that were running code that didn't not properly close vectors?
>
> For example if I ran a Java program that did not properly close vectors,
> could I terminate the Java program and somehow find the un-closed vectors
> and manually close them from the command line?
>
> Essentially, how would I clean up vectors on a server after the Java
> program is terminated and I no longer have access to the `BufferAllocator`
> I used to create the vectors?
>
> Best,
> Daniel
>
> On 2021/09/13 12:27:41, Benjamin Wilhelm <be...@knime.com>
> wrote:
> > Hi Daniel,
> >
> > For testing you can close the `BufferAllocator` used. If assertions are
> > enabled, `close` will throw an exception if buffers are outstanding. For
> > production you should test your code well enough to be sure that all
> > vectors are closed.
> >
> > Other things that might help to detect memory leaks (in production if you
> > wish so):
> > * Child allocators (`BufferAllocator#newChildAllocator`). You can close
> an
> > child allocator to release all the memory allocated with it
> > * `BufferAllocator#getAllocatedMemory`: If this is !=0 this means that
> you
> > have still something allocated with this allocator
> >
> > I hope that helps
> > Benjamin
> >
> >
> > On Mon, Sep 13, 2021 at 9:17 AM Daniel Hsu <da...@bytedance.com>
> wrote:
> >
> > > Hi All,
> > >
> > > Is there a way to figure out which vectors I haven't called close() on
> > > yet? My concern is that if I have a bug in my code where I accidentally
> > > forget to call close() for many vectors, the space occupied by these
> > > vectors could grow forever and crash my system.
> > >
> > > How do I detect these vector "memory leaks"?
> > >
> > > Best,
> > > Daniel
> > >
> >
>
Re: [Java] How to detect if there are dangling vectors that we need call close() for?
Posted by Daniel Hsu <da...@bytedance.com>.
Hi Benjamin,
Thanks for the pointers. Is there a recommended way to clean up servers that were running code that didn't not properly close vectors?
For example if I ran a Java program that did not properly close vectors, could I terminate the Java program and somehow find the un-closed vectors and manually close them from the command line?
Essentially, how would I clean up vectors on a server after the Java program is terminated and I no longer have access to the `BufferAllocator` I used to create the vectors?
Best,
Daniel
On 2021/09/13 12:27:41, Benjamin Wilhelm <be...@knime.com> wrote:
> Hi Daniel,
>
> For testing you can close the `BufferAllocator` used. If assertions are
> enabled, `close` will throw an exception if buffers are outstanding. For
> production you should test your code well enough to be sure that all
> vectors are closed.
>
> Other things that might help to detect memory leaks (in production if you
> wish so):
> * Child allocators (`BufferAllocator#newChildAllocator`). You can close an
> child allocator to release all the memory allocated with it
> * `BufferAllocator#getAllocatedMemory`: If this is !=0 this means that you
> have still something allocated with this allocator
>
> I hope that helps
> Benjamin
>
>
> On Mon, Sep 13, 2021 at 9:17 AM Daniel Hsu <da...@bytedance.com> wrote:
>
> > Hi All,
> >
> > Is there a way to figure out which vectors I haven't called close() on
> > yet? My concern is that if I have a bug in my code where I accidentally
> > forget to call close() for many vectors, the space occupied by these
> > vectors could grow forever and crash my system.
> >
> > How do I detect these vector "memory leaks"?
> >
> > Best,
> > Daniel
> >
>
Re: [Java] How to detect if there are dangling vectors that we need
call close() for?
Posted by Benjamin Wilhelm <be...@knime.com>.
Hi Daniel,
For testing you can close the `BufferAllocator` used. If assertions are
enabled, `close` will throw an exception if buffers are outstanding. For
production you should test your code well enough to be sure that all
vectors are closed.
Other things that might help to detect memory leaks (in production if you
wish so):
* Child allocators (`BufferAllocator#newChildAllocator`). You can close an
child allocator to release all the memory allocated with it
* `BufferAllocator#getAllocatedMemory`: If this is !=0 this means that you
have still something allocated with this allocator
I hope that helps
Benjamin
On Mon, Sep 13, 2021 at 9:17 AM Daniel Hsu <da...@bytedance.com> wrote:
> Hi All,
>
> Is there a way to figure out which vectors I haven't called close() on
> yet? My concern is that if I have a bug in my code where I accidentally
> forget to call close() for many vectors, the space occupied by these
> vectors could grow forever and crash my system.
>
> How do I detect these vector "memory leaks"?
>
> Best,
> Daniel
>