You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Michael Lee <ma...@gmail.com> on 2010/01/26 12:26:35 UTC
too many opened file during snapshot
During snapshoting, FileUtils.createHardLink create too many opened pipe, which only can be freed at GC, if one node has a lots of SSTable files.
If so, the snapshot will failed, left a ‘half snapshot’.
[root@xxxxx.com fd]# pwd
/proc/3992/fd
[root@db-spi-newcir83.db01.baidu.com fd]# ll | grep pipe | wc -l
12207
Here is my work out:
public static void createHardLink(File sourceFile, File destinationFile) throws IOException
{
String osname = System.getProperty("os.name");
ProcessBuilder pb;
if (osname.startsWith("Windows"))
{
float osversion = Float.parseFloat(System.getProperty("os.version"));
if (osversion >= 6.0f)
{
pb = new ProcessBuilder("cmd", "/c", "mklink", "/H", destinationFile.getAbsolutePath(), sourceFile.getAbsolutePath());
}
else
{
pb = new ProcessBuilder("fsutil", "hardlink", "create", destinationFile.getAbsolutePath(), sourceFile.getAbsolutePath());
}
}
else
{
pb = new ProcessBuilder("ln", sourceFile.getAbsolutePath(), destinationFile.getAbsolutePath());
pb.redirectErrorStream(true);
}
Process p = pb.start();
try
{
p.waitFor();
+ p.getErrorStream().close();
+ p.getInputStream().close();
+ p.getOutputStream().close();
}
catch (InterruptedException e)
{
throw new RuntimeException(e);
}
}
Any advice??
---------END----------
Re: too many opened file during snapshot
Posted by Jonathan Ellis <jb...@gmail.com>.
does adding p.destroy() after p.waitfor fix it?
On Tue, Jan 26, 2010 at 5:26 AM, Michael Lee
<ma...@gmail.com> wrote:
> During snapshoting, FileUtils.createHardLink create too many opened pipe,
> which only can be freed at GC, if one node has a lots of SSTable files.
>
> If so, the snapshot will failed, left a ‘half snapshot’.
>
>
>
> [root@xxxxx.com fd]# pwd
>
> /proc/3992/fd
>
> [root@db-spi-newcir83.db01.baidu.com fd]# ll | grep pipe | wc -l
>
> 12207
>
>
>
> Here is my work out:
>
>
>
> public static void createHardLink(File sourceFile, File destinationFile)
> throws IOException
>
> {
>
> String osname = System.getProperty("os.name");
>
> ProcessBuilder pb;
>
> if (osname.startsWith("Windows"))
>
> {
>
> float osversion =
> Float.parseFloat(System.getProperty("os.version"));
>
> if (osversion >= 6.0f)
>
> {
>
> pb = new ProcessBuilder("cmd", "/c", "mklink", "/H",
> destinationFile.getAbsolutePath(), sourceFile.getAbsolutePath());
>
> }
>
> else
>
> {
>
> pb = new ProcessBuilder("fsutil", "hardlink", "create",
> destinationFile.getAbsolutePath(), sourceFile.getAbsolutePath());
>
> }
>
> }
>
> else
>
> {
>
> pb = new ProcessBuilder("ln", sourceFile.getAbsolutePath(),
> destinationFile.getAbsolutePath());
>
> pb.redirectErrorStream(true);
>
> }
>
> Process p = pb.start();
>
> try
>
> {
>
> p.waitFor();
>
> + p.getErrorStream().close();
>
> + p.getInputStream().close();
>
> + p.getOutputStream().close();
>
>
>
> }
>
> catch (InterruptedException e)
>
> {
>
> throw new RuntimeException(e);
>
> }
>
> }
>
>
>
> Any advice??
>
>
>
> ---------END----------
>
>