You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Soichi Hayashi <ha...@indiana.edu> on 2010/04/18 23:30:39 UTC
Can't start cassandra
Hi.
I just downloaded Cassandra 0.6.0 and followed "Getting Started" tutorial.
When I try to start up Cassandra, I get following error message.
2010-04-18 20:53:39 UTC [hayashis@data-itb:~/app/apache-cassandra-0.6.0]$
./bin/cassandra -f
INFO 21:27:29,127 Auto DiskAccessMode determined to be mmap
INFO 21:27:29,482 Replaying
/home/hayashis/app/apache-cassandra-0.6.0/log/log.txt
java.lang.OutOfMemoryError: Java heap space
Dumping heap to java_pid19185.hprof ...
Heap dump file created [3630366 bytes in 0.065 secs]
ERROR 21:27:29,720 Exception encountered during startup.
java.lang.OutOfMemoryError: Java heap space
at
org.apache.cassandra.db.commitlog.CommitLogHeader.readCommitLogHeader(CommitLogHeader.java:146)
at
org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:181)
at
org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:167)
at
org.apache.cassandra.thrift.CassandraDaemon.setup(CassandraDaemon.java:95)
at
org.apache.cassandra.thrift.CassandraDaemon.main(CassandraDaemon.java:177)
Exception encountered during startup.
java.lang.OutOfMemoryError: Java heap space
at
org.apache.cassandra.db.commitlog.CommitLogHeader.readCommitLogHeader(CommitLogHeader.java:146)
at
org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:181)
at
org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:167)
at
org.apache.cassandra.thrift.CassandraDaemon.setup(CassandraDaemon.java:95)
at
org.apache.cassandra.thrift.CassandraDaemon.main(CassandraDaemon.java:177)
I've modified bin/cassandra.in.sh so that it will have
# Arguments to pass to the JVM
JVM_OPTS=" \
-ea \
-Xms1G\
-Xmx1G \
.. but I got an identical error message.
I tried to find some clues on the mailing list archive, or the Jira but so
far I didn't find any.
I tried installing cassandra on various other machines but ended up having
the same issue.. I am just trying to learn about Cassandra and I appreciate
any help you could give me.
Thanks,
Soichi
Re: Can't start cassandra
Posted by Soichi Hayashi <so...@gmail.com>.
> commitlog must be pointed to a directory that _only_ has commitlog data in
it
This was exactly it.. I was outputting log4j.properties log on /log
directory which I have created to store commitlog - also. I apologize if
this is documented in the tutorial somewhere!
On Sun, Apr 18, 2010 at 9:07 PM, Jonathan Ellis <jb...@gmail.com> wrote:
> Ha! Nice catch.
>
> Yes, commitlog must be pointed to a directory that _only_ has
> commitlog data in it.
>
> On Sun, Apr 18, 2010 at 7:27 PM, Brandon Williams <dr...@gmail.com>
> wrote:
> > On Sun, Apr 18, 2010 at 4:30 PM, Soichi Hayashi <ha...@indiana.edu>
> > wrote:
> >>
> >> INFO 21:27:29,482 Replaying
> >> /home/hayashis/app/apache-cassandra-0.6.0/log/log.txt
> >> java.lang.OutOfMemoryError: Java heap space
> >
> > I'm not sure what log.txt might be, but based on the filename it is not a
> > Cassandra commitlog. You can probably move it out of the way and startup
> > will succeed.
> > -Brandon
>
Re: Can't start cassandra
Posted by Jonathan Ellis <jb...@gmail.com>.
Ha! Nice catch.
Yes, commitlog must be pointed to a directory that _only_ has
commitlog data in it.
On Sun, Apr 18, 2010 at 7:27 PM, Brandon Williams <dr...@gmail.com> wrote:
> On Sun, Apr 18, 2010 at 4:30 PM, Soichi Hayashi <ha...@indiana.edu>
> wrote:
>>
>> INFO 21:27:29,482 Replaying
>> /home/hayashis/app/apache-cassandra-0.6.0/log/log.txt
>> java.lang.OutOfMemoryError: Java heap space
>
> I'm not sure what log.txt might be, but based on the filename it is not a
> Cassandra commitlog. You can probably move it out of the way and startup
> will succeed.
> -Brandon
Re: Can't start cassandra
Posted by Brandon Williams <dr...@gmail.com>.
On Sun, Apr 18, 2010 at 4:30 PM, Soichi Hayashi <ha...@indiana.edu>wrote:
> INFO 21:27:29,482 Replaying
> /home/hayashis/app/apache-cassandra-0.6.0/log/log.txt
> java.lang.OutOfMemoryError: Java heap space
>
I'm not sure what log.txt might be, but based on the filename it is not a
Cassandra commitlog. You can probably move it out of the way and startup
will succeed.
-Brandon
Re: Can't start cassandra
Posted by Jonathan Ellis <jb...@gmail.com>.
You have a corrupt commitlogheader. The easiest fix is deleting the
commitlog segment in question.
I created https://issues.apache.org/jira/browse/CASSANDRA-999 to add
checksumming to the commitlog headers (to detect log segments that
didn't fsync cleanly), but that wouldn't help in your case because you
need to have enough ram to read a 2GB byte array (worst case) before
computing the checksum.
On Sun, Apr 18, 2010 at 7:02 PM, Soichi Hayashi <so...@gmail.com> wrote:
> Ah.. Thanks for the great tip!
>
> I tried decreasing it and tried various combination (including the one you
> have suggested) but I am still getting the same error message. Following is
> the /prop/meminfo on the machine that I am using.
>
> 2010-04-18 23:56:54 UTC [root@data-itb:/usr/local/apache-cassandra-0.6.0]#
> cat /proc/meminfo
> MemTotal: 1026880 kB
> MemFree: 518320 kB
> Buffers: 48680 kB
> Cached: 319532 kB
> SwapCached: 12 kB
> Active: 206172 kB
> Inactive: 226324 kB
> HighTotal: 0 kB
> HighFree: 0 kB
> LowTotal: 1026880 kB
> LowFree: 518320 kB
> SwapTotal: 977088 kB
> SwapFree: 977076 kB
> Dirty: 3784 kB
> Writeback: 0 kB
> AnonPages: 64284 kB
> Mapped: 23436 kB
> Slab: 54316 kB
> PageTables: 8852 kB
> NFS_Unstable: 0 kB
> Bounce: 0 kB
> CommitLimit: 1490528 kB
> Committed_AS: 284596 kB
> VmallocTotal: 34359738367 kB
> VmallocUsed: 1068 kB
> VmallocChunk: 34359737239 kB
> HugePages_Total: 0
> HugePages_Free: 0
> HugePages_Rsvd: 0
> Hugepagesize: 2048 kB
>
> I have made sure that the cassandra.in.sh that I am editing is actually
> used... I've installed a new VM with Ubuntu 9.10 and I was able to recreate
> the same issue on Ubuntu as well. The machine that I am having this issue is
> RHEL5.
>
> What else can I try to resolve this?
>
> On Sun, Apr 18, 2010 at 7:43 PM, Benjamin Black <b...@b3k.us> wrote:
>>
>> If you are trying to run on machines with less than 1GB of memory, or
>> OS resource limits that prevent allocation of 1GB of memory, that is
>> what happens. You shouldn't be increasing -Xms, you should be
>> decreasing -Xmx. Try -Xms16M -Xmx500M.
>>
>>
>> b
>>
>> On Sun, Apr 18, 2010 at 2:30 PM, Soichi Hayashi <ha...@indiana.edu>
>> wrote:
>> > Hi.
>> >
>> > I just downloaded Cassandra 0.6.0 and followed "Getting Started"
>> > tutorial.
>> >
>> > When I try to start up Cassandra, I get following error message.
>> >
>> > 2010-04-18 20:53:39 UTC
>> > [hayashis@data-itb:~/app/apache-cassandra-0.6.0]$
>> > ./bin/cassandra -f
>> > INFO 21:27:29,127 Auto DiskAccessMode determined to be mmap
>> > INFO 21:27:29,482 Replaying
>> > /home/hayashis/app/apache-cassandra-0.6.0/log/log.txt
>> > java.lang.OutOfMemoryError: Java heap space
>> > Dumping heap to java_pid19185.hprof ...
>> > Heap dump file created [3630366 bytes in 0.065 secs]
>> > ERROR 21:27:29,720 Exception encountered during startup.
>> > java.lang.OutOfMemoryError: Java heap space
>> > at
>> >
>> > org.apache.cassandra.db.commitlog.CommitLogHeader.readCommitLogHeader(CommitLogHeader.java:146)
>> > at
>> > org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:181)
>> > at
>> > org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:167)
>> > at
>> >
>> > org.apache.cassandra.thrift.CassandraDaemon.setup(CassandraDaemon.java:95)
>> > at
>> >
>> > org.apache.cassandra.thrift.CassandraDaemon.main(CassandraDaemon.java:177)
>> > Exception encountered during startup.
>> > java.lang.OutOfMemoryError: Java heap space
>> > at
>> >
>> > org.apache.cassandra.db.commitlog.CommitLogHeader.readCommitLogHeader(CommitLogHeader.java:146)
>> > at
>> > org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:181)
>> > at
>> > org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:167)
>> > at
>> >
>> > org.apache.cassandra.thrift.CassandraDaemon.setup(CassandraDaemon.java:95)
>> > at
>> >
>> > org.apache.cassandra.thrift.CassandraDaemon.main(CassandraDaemon.java:177)
>> >
>> > I've modified bin/cassandra.in.sh so that it will have
>> >
>> > # Arguments to pass to the JVM
>> > JVM_OPTS=" \
>> > -ea \
>> > -Xms1G\
>> > -Xmx1G \
>> >
>> > .. but I got an identical error message.
>> >
>> > I tried to find some clues on the mailing list archive, or the Jira but
>> > so
>> > far I didn't find any.
>> >
>> > I tried installing cassandra on various other machines but ended up
>> > having
>> > the same issue.. I am just trying to learn about Cassandra and I
>> > appreciate
>> > any help you could give me.
>> >
>> > Thanks,
>> > Soichi
>> >
>> >
>
>
Re: Can't start cassandra
Posted by Soichi Hayashi <so...@gmail.com>.
Ah.. Thanks for the great tip!
I tried decreasing it and tried various combination (including the one you
have suggested) but I am still getting the same error message. Following is
the /prop/meminfo on the machine that I am using.
2010-04-18 23:56:54 UTC [root@data-itb:/usr/local/apache-cassandra-0.6.0]#
cat /proc/meminfo
MemTotal: 1026880 kB
MemFree: 518320 kB
Buffers: 48680 kB
Cached: 319532 kB
SwapCached: 12 kB
Active: 206172 kB
Inactive: 226324 kB
HighTotal: 0 kB
HighFree: 0 kB
LowTotal: 1026880 kB
LowFree: 518320 kB
SwapTotal: 977088 kB
SwapFree: 977076 kB
Dirty: 3784 kB
Writeback: 0 kB
AnonPages: 64284 kB
Mapped: 23436 kB
Slab: 54316 kB
PageTables: 8852 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
CommitLimit: 1490528 kB
Committed_AS: 284596 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 1068 kB
VmallocChunk: 34359737239 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
Hugepagesize: 2048 kB
I have made sure that the cassandra.in.sh that I am editing is actually
used... I've installed a new VM with Ubuntu 9.10 and I was able to recreate
the same issue on Ubuntu as well. The machine that I am having this issue is
RHEL5.
What else can I try to resolve this?
On Sun, Apr 18, 2010 at 7:43 PM, Benjamin Black <b...@b3k.us> wrote:
> If you are trying to run on machines with less than 1GB of memory, or
> OS resource limits that prevent allocation of 1GB of memory, that is
> what happens. You shouldn't be increasing -Xms, you should be
> decreasing -Xmx. Try -Xms16M -Xmx500M.
>
>
> b
>
> On Sun, Apr 18, 2010 at 2:30 PM, Soichi Hayashi <ha...@indiana.edu>
> wrote:
> > Hi.
> >
> > I just downloaded Cassandra 0.6.0 and followed "Getting Started"
> tutorial.
> >
> > When I try to start up Cassandra, I get following error message.
> >
> > 2010-04-18 20:53:39 UTC [hayashis@data-itb
> :~/app/apache-cassandra-0.6.0]$
> > ./bin/cassandra -f
> > INFO 21:27:29,127 Auto DiskAccessMode determined to be mmap
> > INFO 21:27:29,482 Replaying
> > /home/hayashis/app/apache-cassandra-0.6.0/log/log.txt
> > java.lang.OutOfMemoryError: Java heap space
> > Dumping heap to java_pid19185.hprof ...
> > Heap dump file created [3630366 bytes in 0.065 secs]
> > ERROR 21:27:29,720 Exception encountered during startup.
> > java.lang.OutOfMemoryError: Java heap space
> > at
> >
> org.apache.cassandra.db.commitlog.CommitLogHeader.readCommitLogHeader(CommitLogHeader.java:146)
> > at
> > org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:181)
> > at
> > org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:167)
> > at
> >
> org.apache.cassandra.thrift.CassandraDaemon.setup(CassandraDaemon.java:95)
> > at
> >
> org.apache.cassandra.thrift.CassandraDaemon.main(CassandraDaemon.java:177)
> > Exception encountered during startup.
> > java.lang.OutOfMemoryError: Java heap space
> > at
> >
> org.apache.cassandra.db.commitlog.CommitLogHeader.readCommitLogHeader(CommitLogHeader.java:146)
> > at
> > org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:181)
> > at
> > org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:167)
> > at
> >
> org.apache.cassandra.thrift.CassandraDaemon.setup(CassandraDaemon.java:95)
> > at
> >
> org.apache.cassandra.thrift.CassandraDaemon.main(CassandraDaemon.java:177)
> >
> > I've modified bin/cassandra.in.sh so that it will have
> >
> > # Arguments to pass to the JVM
> > JVM_OPTS=" \
> > -ea \
> > -Xms1G\
> > -Xmx1G \
> >
> > .. but I got an identical error message.
> >
> > I tried to find some clues on the mailing list archive, or the Jira but
> so
> > far I didn't find any.
> >
> > I tried installing cassandra on various other machines but ended up
> having
> > the same issue.. I am just trying to learn about Cassandra and I
> appreciate
> > any help you could give me.
> >
> > Thanks,
> > Soichi
> >
> >
>
Re: Can't start cassandra
Posted by Benjamin Black <b...@b3k.us>.
If you are trying to run on machines with less than 1GB of memory, or
OS resource limits that prevent allocation of 1GB of memory, that is
what happens. You shouldn't be increasing -Xms, you should be
decreasing -Xmx. Try -Xms16M -Xmx500M.
b
On Sun, Apr 18, 2010 at 2:30 PM, Soichi Hayashi <ha...@indiana.edu> wrote:
> Hi.
>
> I just downloaded Cassandra 0.6.0 and followed "Getting Started" tutorial.
>
> When I try to start up Cassandra, I get following error message.
>
> 2010-04-18 20:53:39 UTC [hayashis@data-itb:~/app/apache-cassandra-0.6.0]$
> ./bin/cassandra -f
> INFO 21:27:29,127 Auto DiskAccessMode determined to be mmap
> INFO 21:27:29,482 Replaying
> /home/hayashis/app/apache-cassandra-0.6.0/log/log.txt
> java.lang.OutOfMemoryError: Java heap space
> Dumping heap to java_pid19185.hprof ...
> Heap dump file created [3630366 bytes in 0.065 secs]
> ERROR 21:27:29,720 Exception encountered during startup.
> java.lang.OutOfMemoryError: Java heap space
> at
> org.apache.cassandra.db.commitlog.CommitLogHeader.readCommitLogHeader(CommitLogHeader.java:146)
> at
> org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:181)
> at
> org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:167)
> at
> org.apache.cassandra.thrift.CassandraDaemon.setup(CassandraDaemon.java:95)
> at
> org.apache.cassandra.thrift.CassandraDaemon.main(CassandraDaemon.java:177)
> Exception encountered during startup.
> java.lang.OutOfMemoryError: Java heap space
> at
> org.apache.cassandra.db.commitlog.CommitLogHeader.readCommitLogHeader(CommitLogHeader.java:146)
> at
> org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:181)
> at
> org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:167)
> at
> org.apache.cassandra.thrift.CassandraDaemon.setup(CassandraDaemon.java:95)
> at
> org.apache.cassandra.thrift.CassandraDaemon.main(CassandraDaemon.java:177)
>
> I've modified bin/cassandra.in.sh so that it will have
>
> # Arguments to pass to the JVM
> JVM_OPTS=" \
> -ea \
> -Xms1G\
> -Xmx1G \
>
> .. but I got an identical error message.
>
> I tried to find some clues on the mailing list archive, or the Jira but so
> far I didn't find any.
>
> I tried installing cassandra on various other machines but ended up having
> the same issue.. I am just trying to learn about Cassandra and I appreciate
> any help you could give me.
>
> Thanks,
> Soichi
>
>