You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by 严超 <ya...@gmail.com> on 2014/12/10 06:56:35 UTC
[Cassandra][SStableLoader Out of Heap Memory]
Hi, Everyone:
I'm importing a CSV file into Cassandra using SStableLoader. And
I'm following the example here:
https://github.com/yukim/cassandra-bulkload-example/
When i try to run the sstableloader, it fails with OOM. I also
changed the sstableloader.sh script (that runs the java -cp ...BulkLoader )
to have more mem using -Xms and -Xmx args but still i keep hitting the same
issue.
Any hints/directions would be really helpful .
*Stack Trace : *
/usr/bin/sstableloader -v -d <internal-ip> /tmp/nitin_test/nitin_test_load/
Established connection to initial hosts
Opening sstables and calculating sections to stream
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
at java.util.ArrayList.<init>(ArrayList.java:144)
at org.apache.cassandra.db.RowIndexEntry$Serializer.
deserialize(RowIndexEntry.java:120)
at org.apache.cassandra.io.sstable.SSTableReader.buildSummary(SSTableReader.
java:457)
at org.apache.cassandra.io.sstable.SSTableReader.openForBatch(SSTableReader.
java:170)
at org.apache.cassandra.io.sstable.SSTableLoader$1.
accept(SSTableLoader.java:112)
at java.io.File.list(File.java:1155)
at org.apache.cassandra.io.sstable.SSTableLoader.openSSTables(SSTableLoader.
java:73)
at org.apache.cassandra.io.sstable.SSTableLoader.stream(
SSTableLoader.java:155)
at org.apache.cassandra.tools.BulkLoader.main(BulkLoader.java:66)
*Best Regards！*
*Chao Yan--------------**My twitter：Andy Yan @yanchao727
<https://twitter.com/yanchao727>*
*My Weibo：http://weibo.com/herewearenow
<http://weibo.com/herewearenow>--------------*
Re: [Cassandra][SStableLoader Out of Heap Memory]
Posted by 严超 <ya...@gmail.com>.
Thank you very much. It worked.
Actually, these three files are needed:
*CompressionInfo.db, Index.db and Data.db.*
*Best Regards！*
*Chao Yan--------------**My twitter：Andy Yan @yanchao727
<https://twitter.com/yanchao727>*
*My Weibo：http://weibo.com/herewearenow
<http://weibo.com/herewearenow>--------------*
2014-12-10 20:24 GMT+08:00 严超 <ya...@gmail.com>:
> Thank you very much.
> My version is 2.0.11. I'll have a try.
>
> *Best Regards！*
>
>
> *Chao Yan--------------**My twitter：Andy Yan @yanchao727
> <https://twitter.com/yanchao727>*
>
>
> *My Weibo：http://weibo.com/herewearenow
> <http://weibo.com/herewearenow>--------------*
>
> 2014-12-10 19:21 GMT+08:00 Rory Douglas <ro...@gmail.com>:
>
>> Hi Andy
>>
>> Not sure what Cassandra version you're using, but with 2.0 I found I had
>> to remove all the non-essential SSTAble files from the directory (e.g only
>> have .Index and .Data). Otherwise I got OOM like you, even after stung -Xmx4
>>
>> With 2.1 I was able load successfully even with all generated files (e.g.
>> .Filter etc) in the target directory.
>>
>> Rory
>>
>>
>> On Wed, Dec 10, 2014, 12:58 AM 严超 <ya...@gmail.com> wrote:
>>
>>> Hi, Everyone:
>>> I'm importing a CSV file into Cassandra using SStableLoader. And
>>> I'm following the example here:
>>> https://github.com/yukim/cassandra-bulkload-example/
>>> When i try to run the sstableloader, it fails with OOM. I also
>>> changed the sstableloader.sh script (that runs the java -cp ...BulkLoader )
>>> to have more mem using -Xms and -Xmx args but still i keep hitting the same
>>> issue.
>>> Any hints/directions would be really helpful .
>>>
>>> *Stack Trace : *
>>> /usr/bin/sstableloader -v -d <internal-ip> /tmp/nitin_test/nitin_test_
>>> load/
>>>
>>> Established connection to initial hosts
>>> Opening sstables and calculating sections to stream
>>> Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
>>> at java.util.ArrayList.<init>(ArrayList.java:144)
>>> at org.apache.cassandra.db.RowIndexEntry$Serializer.
>>> deserialize(RowIndexEntry.java:120)
>>> at org.apache.cassandra.io.sstable.SSTableReader.
>>> buildSummary(SSTableReader.java:457)
>>> at org.apache.cassandra.io.sstable.SSTableReader.
>>> openForBatch(SSTableReader.java:170)
>>> at org.apache.cassandra.io.sstable.SSTableLoader$1.
>>> accept(SSTableLoader.java:112)
>>> at java.io.File.list(File.java:1155)
>>> at org.apache.cassandra.io.sstable.SSTableLoader.
>>> openSSTables(SSTableLoader.java:73)
>>> at org.apache.cassandra.io.sstable.SSTableLoader.stream(
>>> SSTableLoader.java:155)
>>> at org.apache.cassandra.tools.BulkLoader.main(BulkLoader.java:66)
>>>
>>>
>>> *Best Regards！*
>>>
>>>
>>> *Chao Yan--------------**My twitter：Andy Yan @yanchao727
>>> <https://twitter.com/yanchao727>*
>>>
>>>
>>> *My Weibo：http://weibo.com/herewearenow
>>> <http://weibo.com/herewearenow>--------------*
>>>
>>
>
Re: [Cassandra][SStableLoader Out of Heap Memory]
Posted by 严超 <ya...@gmail.com>.
Thank you very much.
My version is 2.0.11. I'll have a try.
*Best Regards！*
*Chao Yan--------------**My twitter：Andy Yan @yanchao727
<https://twitter.com/yanchao727>*
*My Weibo：http://weibo.com/herewearenow
<http://weibo.com/herewearenow>--------------*
2014-12-10 19:21 GMT+08:00 Rory Douglas <ro...@gmail.com>:
> Hi Andy
>
> Not sure what Cassandra version you're using, but with 2.0 I found I had
> to remove all the non-essential SSTAble files from the directory (e.g only
> have .Index and .Data). Otherwise I got OOM like you, even after stung -Xmx4
>
> With 2.1 I was able load successfully even with all generated files (e.g.
> .Filter etc) in the target directory.
>
> Rory
>
>
> On Wed, Dec 10, 2014, 12:58 AM 严超 <ya...@gmail.com> wrote:
>
>> Hi, Everyone:
>> I'm importing a CSV file into Cassandra using SStableLoader. And
>> I'm following the example here:
>> https://github.com/yukim/cassandra-bulkload-example/
>> When i try to run the sstableloader, it fails with OOM. I also
>> changed the sstableloader.sh script (that runs the java -cp ...BulkLoader )
>> to have more mem using -Xms and -Xmx args but still i keep hitting the same
>> issue.
>> Any hints/directions would be really helpful .
>>
>> *Stack Trace : *
>> /usr/bin/sstableloader -v -d <internal-ip> /tmp/nitin_test/nitin_test_
>> load/
>>
>> Established connection to initial hosts
>> Opening sstables and calculating sections to stream
>> Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
>> at java.util.ArrayList.<init>(ArrayList.java:144)
>> at org.apache.cassandra.db.RowIndexEntry$Serializer.
>> deserialize(RowIndexEntry.java:120)
>> at org.apache.cassandra.io.sstable.SSTableReader.
>> buildSummary(SSTableReader.java:457)
>> at org.apache.cassandra.io.sstable.SSTableReader.
>> openForBatch(SSTableReader.java:170)
>> at org.apache.cassandra.io.sstable.SSTableLoader$1.
>> accept(SSTableLoader.java:112)
>> at java.io.File.list(File.java:1155)
>> at org.apache.cassandra.io.sstable.SSTableLoader.
>> openSSTables(SSTableLoader.java:73)
>> at org.apache.cassandra.io.sstable.SSTableLoader.stream(
>> SSTableLoader.java:155)
>> at org.apache.cassandra.tools.BulkLoader.main(BulkLoader.java:66)
>>
>>
>> *Best Regards！*
>>
>>
>> *Chao Yan--------------**My twitter：Andy Yan @yanchao727
>> <https://twitter.com/yanchao727>*
>>
>>
>> *My Weibo：http://weibo.com/herewearenow
>> <http://weibo.com/herewearenow>--------------*
>>
>
Re: [Cassandra][SStableLoader Out of Heap Memory]
Posted by Rory Douglas <ro...@gmail.com>.
Hi Andy
Not sure what Cassandra version you're using, but with 2.0 I found I had to
remove all the non-essential SSTAble files from the directory (e.g only
have .Index and .Data). Otherwise I got OOM like you, even after stung -Xmx4
With 2.1 I was able load successfully even with all generated files (e.g.
.Filter etc) in the target directory.
Rory
On Wed, Dec 10, 2014, 12:58 AM 严超 <ya...@gmail.com> wrote:
> Hi, Everyone:
> I'm importing a CSV file into Cassandra using SStableLoader. And
> I'm following the example here:
> https://github.com/yukim/cassandra-bulkload-example/
> When i try to run the sstableloader, it fails with OOM. I also
> changed the sstableloader.sh script (that runs the java -cp ...BulkLoader )
> to have more mem using -Xms and -Xmx args but still i keep hitting the same
> issue.
> Any hints/directions would be really helpful .
>
> *Stack Trace : *
> /usr/bin/sstableloader -v -d <internal-ip> /tmp/nitin_test/nitin_test_
> load/
>
> Established connection to initial hosts
> Opening sstables and calculating sections to stream
> Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
> at java.util.ArrayList.<init>(ArrayList.java:144)
> at org.apache.cassandra.db.RowIndexEntry$Serializer.
> deserialize(RowIndexEntry.java:120)
> at org.apache.cassandra.io.sstable.SSTableReader.
> buildSummary(SSTableReader.java:457)
> at org.apache.cassandra.io.sstable.SSTableReader.
> openForBatch(SSTableReader.java:170)
> at org.apache.cassandra.io.sstable.SSTableLoader$1.
> accept(SSTableLoader.java:112)
> at java.io.File.list(File.java:1155)
> at org.apache.cassandra.io.sstable.SSTableLoader.
> openSSTables(SSTableLoader.java:73)
> at org.apache.cassandra.io.sstable.SSTableLoader.stream(
> SSTableLoader.java:155)
> at org.apache.cassandra.tools.BulkLoader.main(BulkLoader.java:66)
>
>
> *Best Regards！*
>
>
> *Chao Yan--------------**My twitter：Andy Yan @yanchao727
> <https://twitter.com/yanchao727>*
>
>
> *My Weibo：http://weibo.com/herewearenow
> <http://weibo.com/herewearenow>--------------*
>