You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Ravikumar Govindarajan <ra...@gmail.com> on 2012/06/14 13:38:00 UTC
MurmurHash NPE during compaction
We received the following NPE during compaction of a large row. We are on
cassandra-1.0.7. Need some help here to find the root cause of the issue
ERROR [CompactionExecutor:595] 2012-06-13 09:44:46,718
AbstractCassandraDaemon.java (line 139) Fatal exception in thread
Thread[CompactionExecutor:595,1,main]
java.lang.NullPointerException
at org.apache.cassandra.utils.MurmurHash.hash64(MurmurHash.java:102)
at
org.apache.cassandra.utils.BloomFilter.getHashBuckets(BloomFilter.java:103)
at
org.apache.cassandra.utils.BloomFilter.getHashBuckets(BloomFilter.java:92)
at org.apache.cassandra.utils.BloomFilter.add(BloomFilter.java:114)
at
org.apache.cassandra.db.ColumnIndexer.serialize(ColumnIndexer.java:96)
at
org.apache.cassandra.db.ColumnIndexer.serialize(ColumnIndexer.java:51)
at
org.apache.cassandra.db.compaction.PrecompactedRow.write(PrecompactedRow.java:135)
at
org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:160)
at
org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:159)
at
org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:134)
at
org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:114)
at
java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)
Thanks and Regards,
Ravi
Re: Column names overhead
Posted by aaron morton <aa...@thelastpickle.com>.
> What is the penalty for using longer column names?
Each column name is stored in each -Data file were a value is stored for it. So if you have muchos overwrites the column name may be stored many places.
> Should I sacrifice longer self-explanatory names for shorter cryptic ones to save the disk space?
If you have lots of COBOL programmers around it may be OK. If you are at the extremes of capacity it may also be ok.
You may also get some value by storing the schema separately from the data.
> On one hand, I understand that Cassandra row id a key-value map, but on another hand, it probably uses compression when storing them.
Compression is (currently) off by default, see http://www.datastax.com/dev/blog/whats-new-in-cassandra-1-0-compression
Cheers
-----------------
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com
On 23/06/2012, at 5:03 AM, Leonid Ilyevsky wrote:
> What is the penalty for using longer column names?
> Should I sacrifice longer self-explanatory names for shorter cryptic ones to save the disk space?
> On one hand, I understand that Cassandra row id a key-value map, but on another hand, it probably uses compression when storing them.
>
> This email, along with any attachments, is confidential and may be legally privileged or otherwise protected from disclosure. Any unauthorized dissemination, copying or use of the contents of this email is strictly prohibited and may be in violation of law. If you are not the intended recipient, any disclosure, copying, forwarding or distribution of this email is strictly prohibited and this email and any attachments should be deleted immediately. This email and any attachments do not constitute an offer to sell or a solicitation of an offer to purchase any interest in any investment vehicle sponsored by Moon Capital Management LP (“Moon Capital”). Moon Capital does not provide legal, accounting or tax advice. Any statement regarding legal, accounting or tax matters was not intended or written to be relied upon by any person as advice. Moon Capital does not waive confidentiality or privilege as a result of this email.
Column names overhead
Posted by Leonid Ilyevsky <li...@mooncapital.com>.
What is the penalty for using longer column names?
Should I sacrifice longer self-explanatory names for shorter cryptic ones to save the disk space?
On one hand, I understand that Cassandra row id a key-value map, but on another hand, it probably uses compression when storing them.
________________________________
This email, along with any attachments, is confidential and may be legally privileged or otherwise protected from disclosure. Any unauthorized dissemination, copying or use of the contents of this email is strictly prohibited and may be in violation of law. If you are not the intended recipient, any disclosure, copying, forwarding or distribution of this email is strictly prohibited and this email and any attachments should be deleted immediately. This email and any attachments do not constitute an offer to sell or a solicitation of an offer to purchase any interest in any investment vehicle sponsored by Moon Capital Management LP ("Moon Capital"). Moon Capital does not provide legal, accounting or tax advice. Any statement regarding legal, accounting or tax matters was not intended or written to be relied upon by any person as advice. Moon Capital does not waive confidentiality or privilege as a result of this email.
Re: MurmurHash NPE during compaction
Posted by Ravikumar Govindarajan <ra...@gmail.com>.
Thanks Aaron.
Created a ticket https://issues.apache.org/jira/browse/CASSANDRA-4367
Funnny thing is, I don't see any of the SSTables that participated in the
failed compaction.
Will do an upgradesstables and find out if problem still persists
On Mon, Jun 18, 2012 at 6:43 AM, aaron morton <aa...@thelastpickle.com>wrote:
> Can you please create a ticket on
> https://issues.apache.org/jira/browse/CASSANDRA
>
> Please include:
> * CF definition including the bloom_filter_fp_chance
> * If the data was upgraded from a previous version of cassandra.
> * The names of the files that were being compacted.
>
> As a work around you can try using nodetool upgradetables to re-write the
> files - this may also fail, but its could be worth trying.
>
> The next step would be to remove determine which files were causing the
> issue (looking at the logs) and remove them from the data directory. Then
> run repair to restore consistency.
>
> Cheers
>
> -----------------
> Aaron Morton
> Freelance Developer
> @aaronmorton
> http://www.thelastpickle.com
>
> On 14/06/2012, at 11:38 PM, Ravikumar Govindarajan wrote:
>
> We received the following NPE during compaction of a large row. We are on
> cassandra-1.0.7. Need some help here to find the root cause of the issue
>
> ERROR [CompactionExecutor:595] 2012-06-13 09:44:46,718
> AbstractCassandraDaemon.java (line 139) Fatal exception in thread
> Thread[CompactionExecutor:595,1,main]
> java.lang.NullPointerException
> at
> org.apache.cassandra.utils.MurmurHash.hash64(MurmurHash.java:102)
> at
> org.apache.cassandra.utils.BloomFilter.getHashBuckets(BloomFilter.java:103)
> at
> org.apache.cassandra.utils.BloomFilter.getHashBuckets(BloomFilter.java:92)
> at org.apache.cassandra.utils.BloomFilter.add(BloomFilter.java:114)
> at
> org.apache.cassandra.db.ColumnIndexer.serialize(ColumnIndexer.java:96)
> at
> org.apache.cassandra.db.ColumnIndexer.serialize(ColumnIndexer.java:51)
> at
> org.apache.cassandra.db.compaction.PrecompactedRow.write(PrecompactedRow.java:135)
> at
> org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:160)
> at
> org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:159)
> at
> org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:134)
> at
> org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:114)
> at
> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:619)
>
> Thanks and Regards,
> Ravi
>
>
>
Re: MurmurHash NPE during compaction
Posted by aaron morton <aa...@thelastpickle.com>.
Can you please create a ticket on https://issues.apache.org/jira/browse/CASSANDRA
Please include:
* CF definition including the bloom_filter_fp_chance
* If the data was upgraded from a previous version of cassandra.
* The names of the files that were being compacted.
As a work around you can try using nodetool upgradetables to re-write the files - this may also fail, but its could be worth trying.
The next step would be to remove determine which files were causing the issue (looking at the logs) and remove them from the data directory. Then run repair to restore consistency.
Cheers
-----------------
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com
On 14/06/2012, at 11:38 PM, Ravikumar Govindarajan wrote:
> We received the following NPE during compaction of a large row. We are on cassandra-1.0.7. Need some help here to find the root cause of the issue
>
> ERROR [CompactionExecutor:595] 2012-06-13 09:44:46,718 AbstractCassandraDaemon.java (line 139) Fatal exception in thread Thread[CompactionExecutor:595,1,main]
> java.lang.NullPointerException
> at org.apache.cassandra.utils.MurmurHash.hash64(MurmurHash.java:102)
> at org.apache.cassandra.utils.BloomFilter.getHashBuckets(BloomFilter.java:103)
> at org.apache.cassandra.utils.BloomFilter.getHashBuckets(BloomFilter.java:92)
> at org.apache.cassandra.utils.BloomFilter.add(BloomFilter.java:114)
> at org.apache.cassandra.db.ColumnIndexer.serialize(ColumnIndexer.java:96)
> at org.apache.cassandra.db.ColumnIndexer.serialize(ColumnIndexer.java:51)
> at org.apache.cassandra.db.compaction.PrecompactedRow.write(PrecompactedRow.java:135)
> at org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:160)
> at org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:159)
> at org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:134)
> at org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:114)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:619)
>
> Thanks and Regards,
> Ravi