You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by "Dr. Andrew Perella" <aj...@eutechnyx.com> on 2010/12/15 12:20:47 UTC
Unstable cassandra - , Cannot allocate memory
This is a crash with Cassandra 0.7.0-beta3 but I am pretty sure I also have got this on a rc1 server too ( We are dropping and recreating keyspaces when this happens)
I do have the JNA enabled but it looks like it is still calling an external program - presumably this would cause it to ask for the same memory as the parent process and hence die?
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.io.IOException: Cannot run program "ln": java.io.IOException: error=12, Cannot allocate memory
at java.lang.ProcessBuilder.start(ProcessBuilder.java:460)
at org.apache.cassandra.io.util.FileUtils.createHardLinkWithExec(FileUtils.java:263)
at org.apache.cassandra.io.util.FileUtils.createHardLink(FileUtils.java:229)
at org.apache.cassandra.db.ColumnFamilyStore.snapshot(ColumnFamilyStore.java:1539)
... 9 more
Caused by: java.io.IOException: java.io.IOException: error=12, Cannot allocate memory
at java.lang.UNIXProcess.<init>(UNIXProcess.java:148)
at java.lang.ProcessImpl.start(ProcessImpl.java:65)
at java.lang.ProcessBuilder.start(ProcessBuilder.java:453)
... 12 more
ERROR 12:11:15,673 Fatal exception in thread Thread[MigrationStage:1,5,main]
java.io.IOError: java.io.IOException: Cannot run program "ln": java.io.IOException: error=12, Cannot allocate memory
at org.apache.cassandra.db.ColumnFamilyStore.snapshot(ColumnFamilyStore.java:1547)
at org.apache.cassandra.db.Table.snapshot(Table.java:177)
at org.apache.cassandra.db.migration.DropKeyspace.beforeApplyModels(DropKeyspace.java:61)
at org.apache.cassandra.db.migration.Migration.apply(Migration.java:109)
at org.apache.cassandra.thrift.CassandraServer$2.call(CassandraServer.java:672)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.io.IOException: Cannot run program "ln": java.io.IOException: error=12, Cannot allocate memory
at java.lang.ProcessBuilder.start(ProcessBuilder.java:460)
at org.apache.cassandra.io.util.FileUtils.createHardLinkWithExec(FileUtils.java:263)
at org.apache.cassandra.io.util.FileUtils.createHardLink(FileUtils.java:229)
at org.apache.cassandra.db.ColumnFamilyStore.snapshot(ColumnFamilyStore.java:1539)
... 9 more
Caused by: java.io.IOException: java.io.IOException: error=12, Cannot allocate memory
at java.lang.UNIXProcess.<init>(UNIXProcess.java:148)
at java.lang.ProcessImpl.start(ProcessImpl.java:65)
at java.lang.ProcessBuilder.start(ProcessBuilder.java:453)
... 12 more
Dr. Andrew Perella
CTO / Chief Software Architect
Eutechnyx Limited. Metro Centre East Business Park, Waterside Drive, Gateshead, Tyne & Wear NE11 9HU UK Co.Reg.No. 2172322
T +44 (0) 191 460 6060 F +44 (0) 191 460 2266 E ajp@eutechnyx.com<ma...@eutechnyx.com> W www.eutechnyx.com<http://www.eutechnyx.com/>
This e-mail is confidential and may be privileged. It may be read, copied and used only by the intended recipient. No communication sent by e-mail to or from Eutechnyx is intended to give rise to contractual or other legal liability, apart from liability which cannot be excluded under English law.
This e-mail is confidential and may be privileged. It may be read, copied and used only by the intended recipient. No communication sent by e-mail to or from Eutechnyx is intended to give rise to contractual or other legal liability, apart from liability which cannot be excluded under English law.
This email has been scanned for all known viruses by the Email Protection Agency. http://www.epagency.net
www.eutechnyx.com Eutechnyx Limited. Registered in England No: 2172322
Re: Unstable cassandra - , Cannot allocate memory
Posted by Jonathan Ellis <jb...@gmail.com>.
Those would both work; the "right" answer is "get JNA working." :)
rc2 has improved JNA error reporting, upgrade to that and watch the messages
on startup, and then when try the snapshot.
On Wed, Dec 15, 2010 at 6:37 AM, Peter Schuller <peter.schuller@infidyne.com
> wrote:
> > This is a crash with Cassandra 0.7.0-beta3 but I am pretty sure I also
> have got this on a rc1 server too ( We are dropping and recreating keyspaces
> when this happens)
> >
> >
> >
> > I do have the JNA enabled but it looks like it is still calling an
> external program – presumably this would cause it to ask for the same memory
> as the parent process and hence die?
>
> Yes. Either enable overcommit in the Linux kernel, or use a
> sufficiently new cassandra with working JNA to avoid use of fork+exec
> for running ln.
>
> (I forget when the change wen in so not sure whether beta3/rc1 is
> recent enough for the ln fix.)
>
> --
> / Peter Schuller
>
--
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of Riptano, the source for professional Cassandra support
http://riptano.com
Re: Unstable cassandra - , Cannot allocate memory
Posted by Peter Schuller <pe...@infidyne.com>.
> This is a crash with Cassandra 0.7.0-beta3 but I am pretty sure I also have got this on a rc1 server too ( We are dropping and recreating keyspaces when this happens)
>
>
>
> I do have the JNA enabled but it looks like it is still calling an external program – presumably this would cause it to ask for the same memory as the parent process and hence die?
Yes. Either enable overcommit in the Linux kernel, or use a
sufficiently new cassandra with working JNA to avoid use of fork+exec
for running ln.
(I forget when the change wen in so not sure whether beta3/rc1 is
recent enough for the ln fix.)
--
/ Peter Schuller