deleteOnExit when JVM shutsdown non gracefully

[Asaf Mesika <as...@gmail.com>]

Hi,

In the CoprocessorHost.java file, there's the following code section used
to load a coprocessor jar:

      fs.copyToLocalFile(path, dst);

      File tmpLocal = new File(dst.toString());

      tmpLocal.deleteOnExit();

There's an assumption here that the JVM will gracefully shutdown (as
opposed to machine went dead, kill -9, OutOfMemory which cause kill -9 in
plain vanilla HBase).

If this happens, the temp jar and CRC files will remain there.

We saw in a production environment a case where there were 300k files there
(only jar and crc files releated to our coprocessor jar filename).

It seems right to me, to add code section which deletes the stale files
upon startup of the region server (just deletes all *.jar before a certain
timestamp). Or better - place them in /tmp/hbase, and delete the contents
of this folder upon startup.

What do you think?

Re: deleteOnExit when JVM shutsdown non gracefully

[Ted Yu <yu...@gmail.com>]

Interesting. 
File a JIRA ?

Thanks

On Apr 10, 2013, at 2:30 AM, Asaf Mesika <as...@gmail.com> wrote:

> Hi,
> 
> In the CoprocessorHost.java file, there's the following code section used
> to load a coprocessor jar:
> 
>      fs.copyToLocalFile(path, dst);
> 
>      File tmpLocal = new File(dst.toString());
> 
>      tmpLocal.deleteOnExit();
> 
> There's an assumption here that the JVM will gracefully shutdown (as
> opposed to machine went dead, kill -9, OutOfMemory which cause kill -9 in
> plain vanilla HBase).
> 
> If this happens, the temp jar and CRC files will remain there.
> 
> We saw in a production environment a case where there were 300k files there
> (only jar and crc files releated to our coprocessor jar filename).
> 
> It seems right to me, to add code section which deletes the stale files
> upon startup of the region server (just deletes all *.jar before a certain
> timestamp). Or better - place them in /tmp/hbase, and delete the contents
> of this folder upon startup.
> 
> What do you think?

Re: deleteOnExit when JVM shutsdown non gracefully

[Andrew Purtell <ap...@apache.org>]

Perhaps we can simply unlink the file after load. On *nix the OS would GC
the file data after the JVM process terminates and the filehandles are
closed. Of course this won't work on Windows. (But I don't care about that.)

We added a change such that now all coprocessor jars are brought locally to
a specific directory that can be specified in hbase-site.xml. Placing them
in hbase.tmp.dir inflexibly would be a step backwards, although I believe
that is a default. -1 on the suggestion to automatically clear
hbase.tmp.dir, people actually run with that when first evaluating hbase
sometimes, it would be an unwelcome surprise to nuke their test data. All
of the scenarios you mention are unclean/exceptional shutdowns, perhaps it
makes sense to add a note in the book so ops folks can be aware that dead
coprocessor jars might collect in those cases.


On Wed, Apr 10, 2013 at 2:30 AM, Asaf Mesika <as...@gmail.com> wrote:

> Hi,
>
> In the CoprocessorHost.java file, there's the following code section used
> to load a coprocessor jar:
>
>       fs.copyToLocalFile(path, dst);
>
>       File tmpLocal = new File(dst.toString());
>
>       tmpLocal.deleteOnExit();
>
> There's an assumption here that the JVM will gracefully shutdown (as
> opposed to machine went dead, kill -9, OutOfMemory which cause kill -9 in
> plain vanilla HBase).
>
> If this happens, the temp jar and CRC files will remain there.
>
> We saw in a production environment a case where there were 300k files there
> (only jar and crc files releated to our coprocessor jar filename).
>
> It seems right to me, to add code section which deletes the stale files
> upon startup of the region server (just deletes all *.jar before a certain
> timestamp). Or better - place them in /tmp/hbase, and delete the contents
> of this folder upon startup.
>
> What do you think?
>



-- 
Best regards,

   - Andy

Problems worthy of attack prove their worth by hitting back. - Piet Hein
(via Tom White)