You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@cassandra.apache.org by Jonathan Ellis <jb...@gmail.com> on 2010/10/27 15:50:19 UTC
Re: 0.7.0beta2 with 128G memory = bizarre tiny files. help?
Connect with the cli and run "show keyspaces." It sounds like
Cassandra somehow picked ludicrously low memtable thresholds. There
will be a line like
Memtable thresholds: 0.2953125/63/60
This may not be in the b2 cli, in which case you will need to upgrade
to the nightly from
http://hudson.zones.apache.org/hudson/job/Cassandra/lastSuccessfulBuild/artifact/cassandra/build/
(essentially the same as rc1 being voted on).
On Wed, Oct 27, 2010 at 4:17 AM, Chip Salzenberg <ch...@pobox.com> wrote:
> I recently tested 0.7.0beta2 with good results on a reasonably powerful
> machine: 8 Xeon cores (16 if you count hyperthreads), 64G memory, and
> some nice HP RAID. So far so good. But when I took the same config and
> moved it to a basically identical box with 128G of memory, cassandra
> started responding to writes by creating a plethora of teeny tiny
> sstables -- something it had not done before. For example:
>
> INFO [FLUSH-WRITER-POOL:1] 2010-10-27 01:53:27,881 Memtable.java (line
> 150) Writing Memtable-Standard1@950886315(651 bytes, 18 operations)
> INFO [MUTATION_STAGE:8] 2010-10-27 01:53:27,882 ColumnFamilyStore.java
> (line 459) switching in a fresh Memtable for Standard1 at
> CommitLogContext(file='/var/lib/cassandra/commitlog/_chip/CommitLog-1288169534132.log',
> position=27598)
> INFO [MUTATION_STAGE:8] 2010-10-27 01:53:27,883 ColumnFamilyStore.java
> (line 771) Enqueuing flush of Memtable-Standard1@1383310803(384 bytes,
> 10 operations)
> INFO [FLUSH-WRITER-POOL:1] 2010-10-27 01:53:27,902 Memtable.java (line
> 157) Completed flushing
> /var/lib/cassandra/data/_chip/Keyspace1/Standard1-e-7-Data.db
> INFO [FLUSH-WRITER-POOL:1] 2010-10-27 01:53:27,903 Memtable.java (line
> 150) Writing Memtable-Standard1@996627145(360 bytes, 10 operations)
> INFO [MUTATION_STAGE:3] 2010-10-27 01:53:27,927 ColumnFamilyStore.java
> (line 459) switching in a fresh Memtable for Standard1 at
> CommitLogContext(file='/var/lib/cassandra/commitlog/_chip/CommitLog-1288169534132.log',
> position=31446)
> INFO [MUTATION_STAGE:3] 2010-10-27 01:53:27,927 ColumnFamilyStore.java
> (line 771) Enqueuing flush of Memtable-Standard1@1909350010(957 bytes,
> 26 operations)
> INFO [FLUSH-WRITER-POOL:1] 2010-10-27 01:53:27,932 Memtable.java (line
> 157) Completed flushing
> /var/lib/cassandra/data/_chip/Keyspace1/Standard1-e-8-Data.db
> INFO [FLUSH-WRITER-POOL:1] 2010-10-27 01:53:27,933 Memtable.java (line
> 150) Writing Memtable-Standard1@1383310803(384 bytes, 10 operations)
> INFO [CompactionExecutor:1] 2010-10-27 01:53:27,934
> CompactionManager.java (line 233) Compacting
> [org.apache.cassandra.io.sstable.SSTableReader(path='/var/lib/cassandra/data/_chip/Keyspace1/Standard1-e-5-Data.db'),org.apache.cassandra.io.sstable.SSTableReader(path='/var/lib/cassandra/data/_chip/Keyspace1/Standard1-e-6-Data.db'),org.apache.cassandra.io.sstable.SSTableReader(path='/var/lib/cassandra/data/_chip/Keyspace1/Standard1-e-7-Data.db'),org.apache.cassandra.io.sstable.SSTableReader(path='/var/lib/cassandra/data/_chip/Keyspace1/Standard1-e-8-Data.db')]
> INFO [MUTATION_STAGE:17] 2010-10-27 01:53:27,936 ColumnFamilyStore.java
> (line 459) switching in a fresh Memtable for Standard1 at
> CommitLogContext(file='/var/lib/cassandra/commitlog/_chip/CommitLog-1288169534132.log',
> position=33074)
> INFO [MUTATION_STAGE:17] 2010-10-27 01:53:27,936 ColumnFamilyStore.java
> (line 771) Enqueuing flush of Memtable-Standard1@595066677(396 bytes, 11
> operations)
> INFO [FLUSH-WRITER-POOL:1] 2010-10-27 01:53:28,037 Memtable.java (line
> 157) Completed flushing
> /var/lib/cassandra/data/_chip/Keyspace1/Standard1-e-9-Data.db
>
> I tried lowering the available memory reported by bin/cassandra:
>
> system_memory_in_mb=`free -m | awk '/Mem:/ {print $2}'`
> [ "$system_memory_in_mb" -gt 65536 ] &&
> system_memory_in_mb=65536 #<<<< new
>
> Didn't help. I also tried maually setting the max memtable sizes:
>
> binary_memtable_throughput_in_mb: 512
> memtable_throughput_in_mb: 4096
>
> Didn't help.
>
> Help?
>
--
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of Riptano, the source for professional Cassandra support
http://riptano.com
Re: 0.7.0beta2 with 128G memory = bizarre tiny files. help?
Posted by Chip Salzenberg <ch...@pobox.com>.
Since I had to upgrade, I have to fix my Perl client to match the new
thrift interface. It'll be later today. Then I can retry.
On 10/27/2010 2:09 PM, Jonathan Ellis wrote:
> Thanks for the clarification.
>
> Are you still seeing the tiny files?
>
> On Wed, Oct 27, 2010 at 4:02 PM, Chip Salzenberg <ch...@pobox.com> wrote:
>> That's what you're looking at. Sorry for the confusion. I left all the
>> files in place for the upgrade. What I meant was that back when b2 was
>> running, "import" was how I had created Standard1 in the first place.
>>
>> On 10/27/2010 1:54 PM, Jonathan Ellis wrote:
>>> Sorry, I should have been more clear: rc1 is upgrade-in-place
>>> compatible with b2, so I would have liked to see what the settings
>>> were before running it through an export/import cycle.
>>>
>>> These settings look fine and I doubt you will see bizarre tiny files.
>>>
>>> On Wed, Oct 27, 2010 at 2:52 PM, Chip Salzenberg <ch...@pobox.com> wrote:
>>>> OK, after upgrading to "apache-cassandra-2010-10-27_12-30-38", here's
>>>> what "show keyspaces" says about the relevant keyspace:
>>>>
>>>> ColumnFamily: Standard1
>>>> Columns sorted by: org.apache.cassandra.db.marshal.BytesType
>>>> Row cache size / save period: 0.0/0
>>>> Key cache size / save period: 200000.0/3600
>>>> Memtable thresholds: 9.590625/2046/60
>>>> GC grace seconds: 864000
>>>> Compaction min/max thresholds: 4/32
>>>>
>>>> I created it with "schematool import" off of the stock config in b2.
Re: 0.7.0beta2 with 128G memory = bizarre tiny files. help?
Posted by Jonathan Ellis <jb...@gmail.com>.
Thanks for the clarification.
Are you still seeing the tiny files?
On Wed, Oct 27, 2010 at 4:02 PM, Chip Salzenberg <ch...@pobox.com> wrote:
> That's what you're looking at. Sorry for the confusion. I left all the
> files in place for the upgrade. What I meant was that back when b2 was
> running, "import" was how I had created Standard1 in the first place.
>
> On 10/27/2010 1:54 PM, Jonathan Ellis wrote:
>> Sorry, I should have been more clear: rc1 is upgrade-in-place
>> compatible with b2, so I would have liked to see what the settings
>> were before running it through an export/import cycle.
>>
>> These settings look fine and I doubt you will see bizarre tiny files.
>>
>> On Wed, Oct 27, 2010 at 2:52 PM, Chip Salzenberg <ch...@pobox.com> wrote:
>>> OK, after upgrading to "apache-cassandra-2010-10-27_12-30-38", here's
>>> what "show keyspaces" says about the relevant keyspace:
>>>
>>> ColumnFamily: Standard1
>>> Columns sorted by: org.apache.cassandra.db.marshal.BytesType
>>> Row cache size / save period: 0.0/0
>>> Key cache size / save period: 200000.0/3600
>>> Memtable thresholds: 9.590625/2046/60
>>> GC grace seconds: 864000
>>> Compaction min/max thresholds: 4/32
>>>
>>> I created it with "schematool import" off of the stock config in b2.
--
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of Riptano, the source for professional Cassandra support
http://riptano.com
Re: 0.7.0beta2 with 128G memory = bizarre tiny files. help?
Posted by Chip Salzenberg <ch...@pobox.com>.
That's what you're looking at. Sorry for the confusion. I left all the
files in place for the upgrade. What I meant was that back when b2 was
running, "import" was how I had created Standard1 in the first place.
On 10/27/2010 1:54 PM, Jonathan Ellis wrote:
> Sorry, I should have been more clear: rc1 is upgrade-in-place
> compatible with b2, so I would have liked to see what the settings
> were before running it through an export/import cycle.
>
> These settings look fine and I doubt you will see bizarre tiny files.
>
> On Wed, Oct 27, 2010 at 2:52 PM, Chip Salzenberg <ch...@pobox.com> wrote:
>> OK, after upgrading to "apache-cassandra-2010-10-27_12-30-38", here's
>> what "show keyspaces" says about the relevant keyspace:
>>
>> ColumnFamily: Standard1
>> Columns sorted by: org.apache.cassandra.db.marshal.BytesType
>> Row cache size / save period: 0.0/0
>> Key cache size / save period: 200000.0/3600
>> Memtable thresholds: 9.590625/2046/60
>> GC grace seconds: 864000
>> Compaction min/max thresholds: 4/32
>>
>> I created it with "schematool import" off of the stock config in b2.
>>
>>
>> On 10/27/2010 11:36 AM, Jonathan Ellis wrote:
>>> Not in b2.
>>> On Oct 27, 2010 11:08 AM, "Chip Salzenberg" <ch...@pobox.com> wrote:
>>>> Are these values available in JMX? If so, which ones are they? I'd
>>>> prefer not to possibly destroy the evidence by upgrading.
>>>>
>>>> On 10/27/2010 6:50 AM, Jonathan Ellis wrote:
>>>>> Connect with the cli and run "show keyspaces." It sounds like
>>>>> Cassandra somehow picked ludicrously low memtable thresholds. There
>>>>> will be a line like
>>>>>
>>>>> Memtable thresholds: 0.2953125/63/60
>>>>>
>>>>> This may not be in the b2 cli, in which case you will need to upgrade
>>>>> to the nightly from
>>>>>
>>> http://hudson.zones.apache.org/hudson/job/Cassandra/lastSuccessfulBuild/artifact/cassandra/build/
>>>>> (essentially the same as rc1 being voted on).
>>>>>
>>>>> On Wed, Oct 27, 2010 at 4:17 AM, Chip Salzenberg <ch...@pobox.com> wrote:
>>>>>> I recently tested 0.7.0beta2 with good results on a reasonably powerful
>>>>>> machine: 8 Xeon cores (16 if you count hyperthreads), 64G memory, and
>>>>>> some nice HP RAID. So far so good. But when I took the same config and
>>>>>> moved it to a basically identical box with 128G of memory, cassandra
>>>>>> started responding to writes by creating a plethora of teeny tiny
>>>>>> sstables -- something it had not done before. For example:
>>>>>>
>>>>>> INFO [FLUSH-WRITER-POOL:1] 2010-10-27 01:53:27,881 Memtable.java (line
>>>>>> 150) Writing Memtable-Standard1@950886315(651 bytes, 18 operations)
>>>>>> INFO [MUTATION_STAGE:8] 2010-10-27 01:53:27,882 ColumnFamilyStore.java
>>>>>> (line 459) switching in a fresh Memtable for Standard1 at
>>>>>>
>>> CommitLogContext(file='/var/lib/cassandra/commitlog/_chip/CommitLog-1288169534132.log',
>>>>>> position=27598)
>>>>>> INFO [MUTATION_STAGE:8] 2010-10-27 01:53:27,883 ColumnFamilyStore.java
>>>>>> (line 771) Enqueuing flush of Memtable-Standard1@1383310803(384 bytes,
>>>>>> 10 operations)
>>>>>> INFO [FLUSH-WRITER-POOL:1] 2010-10-27 01:53:27,902 Memtable.java (line
>>>>>> 157) Completed flushing
>>>>>> /var/lib/cassandra/data/_chip/Keyspace1/Standard1-e-7-Data.db
>>>>>> INFO [FLUSH-WRITER-POOL:1] 2010-10-27 01:53:27,903 Memtable.java (line
>>>>>> 150) Writing Memtable-Standard1@996627145(360 bytes, 10 operations)
>>>>>> INFO [MUTATION_STAGE:3] 2010-10-27 01:53:27,927 ColumnFamilyStore.java
>>>>>> (line 459) switching in a fresh Memtable for Standard1 at
>>>>>>
>>> CommitLogContext(file='/var/lib/cassandra/commitlog/_chip/CommitLog-1288169534132.log',
>>>>>> position=31446)
>>>>>> INFO [MUTATION_STAGE:3] 2010-10-27 01:53:27,927 ColumnFamilyStore.java
>>>>>> (line 771) Enqueuing flush of Memtable-Standard1@1909350010(957 bytes,
>>>>>> 26 operations)
>>>>>> INFO [FLUSH-WRITER-POOL:1] 2010-10-27 01:53:27,932 Memtable.java (line
>>>>>> 157) Completed flushing
>>>>>> /var/lib/cassandra/data/_chip/Keyspace1/Standard1-e-8-Data.db
>>>>>> INFO [FLUSH-WRITER-POOL:1] 2010-10-27 01:53:27,933 Memtable.java (line
>>>>>> 150) Writing Memtable-Standard1@1383310803(384 bytes, 10 operations)
>>>>>> INFO [CompactionExecutor:1] 2010-10-27 01:53:27,934
>>>>>> CompactionManager.java (line 233) Compacting
>>>>>>
>>> [org.apache.cassandra.io.sstable.SSTableReader(path='/var/lib/cassandra/data/_chip/Keyspace1/Standard1-e-5-Data.db'),org.apache.cassandra.io.sstable.SSTableReader(path='/var/lib/cassandra/data/_chip/Keyspace1/Standard1-e-6-Data.db'),org.apache.cassandra.io.sstable.SSTableReader(path='/var/lib/cassandra/data/_chip/Keyspace1/Standard1-e-7-Data.db'),org.apache.cassandra.io.sstable.SSTableReader(path='/var/lib/cassandra/data/_chip/Keyspace1/Standard1-e-8-Data.db')]
>>>>>> INFO [MUTATION_STAGE:17] 2010-10-27 01:53:27,936 ColumnFamilyStore.java
>>>>>> (line 459) switching in a fresh Memtable for Standard1 at
>>>>>>
>>> CommitLogContext(file='/var/lib/cassandra/commitlog/_chip/CommitLog-1288169534132.log',
>>>>>> position=33074)
>>>>>> INFO [MUTATION_STAGE:17] 2010-10-27 01:53:27,936 ColumnFamilyStore.java
>>>>>> (line 771) Enqueuing flush of Memtable-Standard1@595066677(396 bytes, 11
>>>>>> operations)
>>>>>> INFO [FLUSH-WRITER-POOL:1] 2010-10-27 01:53:28,037 Memtable.java (line
>>>>>> 157) Completed flushing
>>>>>> /var/lib/cassandra/data/_chip/Keyspace1/Standard1-e-9-Data.db
>>>>>>
>>>>>> I tried lowering the available memory reported by bin/cassandra:
>>>>>>
>>>>>> system_memory_in_mb=`free -m | awk '/Mem:/ {print $2}'`
>>>>>> [ "$system_memory_in_mb" -gt 65536 ] &&
>>>>>> system_memory_in_mb=65536 #<<<< new
>>>>>>
>>>>>> Didn't help. I also tried maually setting the max memtable sizes:
>>>>>>
>>>>>> binary_memtable_throughput_in_mb: 512
>>>>>> memtable_throughput_in_mb: 4096
>>>>>>
>>>>>> Didn't help.
>>>>>>
>>>>>> Help?
>>>>>>
>
>
Re: 0.7.0beta2 with 128G memory = bizarre tiny files. help?
Posted by Jonathan Ellis <jb...@gmail.com>.
Sorry, I should have been more clear: rc1 is upgrade-in-place
compatible with b2, so I would have liked to see what the settings
were before running it through an export/import cycle.
These settings look fine and I doubt you will see bizarre tiny files.
On Wed, Oct 27, 2010 at 2:52 PM, Chip Salzenberg <ch...@pobox.com> wrote:
> OK, after upgrading to "apache-cassandra-2010-10-27_12-30-38", here's
> what "show keyspaces" says about the relevant keyspace:
>
> ColumnFamily: Standard1
> Columns sorted by: org.apache.cassandra.db.marshal.BytesType
> Row cache size / save period: 0.0/0
> Key cache size / save period: 200000.0/3600
> Memtable thresholds: 9.590625/2046/60
> GC grace seconds: 864000
> Compaction min/max thresholds: 4/32
>
> I created it with "schematool import" off of the stock config in b2.
>
>
> On 10/27/2010 11:36 AM, Jonathan Ellis wrote:
>> Not in b2.
>> On Oct 27, 2010 11:08 AM, "Chip Salzenberg" <ch...@pobox.com> wrote:
>>> Are these values available in JMX? If so, which ones are they? I'd
>>> prefer not to possibly destroy the evidence by upgrading.
>>>
>>> On 10/27/2010 6:50 AM, Jonathan Ellis wrote:
>>>> Connect with the cli and run "show keyspaces." It sounds like
>>>> Cassandra somehow picked ludicrously low memtable thresholds. There
>>>> will be a line like
>>>>
>>>> Memtable thresholds: 0.2953125/63/60
>>>>
>>>> This may not be in the b2 cli, in which case you will need to upgrade
>>>> to the nightly from
>>>>
>> http://hudson.zones.apache.org/hudson/job/Cassandra/lastSuccessfulBuild/artifact/cassandra/build/
>>>> (essentially the same as rc1 being voted on).
>>>>
>>>> On Wed, Oct 27, 2010 at 4:17 AM, Chip Salzenberg <ch...@pobox.com> wrote:
>>>>> I recently tested 0.7.0beta2 with good results on a reasonably powerful
>>>>> machine: 8 Xeon cores (16 if you count hyperthreads), 64G memory, and
>>>>> some nice HP RAID. So far so good. But when I took the same config and
>>>>> moved it to a basically identical box with 128G of memory, cassandra
>>>>> started responding to writes by creating a plethora of teeny tiny
>>>>> sstables -- something it had not done before. For example:
>>>>>
>>>>> INFO [FLUSH-WRITER-POOL:1] 2010-10-27 01:53:27,881 Memtable.java (line
>>>>> 150) Writing Memtable-Standard1@950886315(651 bytes, 18 operations)
>>>>> INFO [MUTATION_STAGE:8] 2010-10-27 01:53:27,882 ColumnFamilyStore.java
>>>>> (line 459) switching in a fresh Memtable for Standard1 at
>>>>>
>> CommitLogContext(file='/var/lib/cassandra/commitlog/_chip/CommitLog-1288169534132.log',
>>>>> position=27598)
>>>>> INFO [MUTATION_STAGE:8] 2010-10-27 01:53:27,883 ColumnFamilyStore.java
>>>>> (line 771) Enqueuing flush of Memtable-Standard1@1383310803(384 bytes,
>>>>> 10 operations)
>>>>> INFO [FLUSH-WRITER-POOL:1] 2010-10-27 01:53:27,902 Memtable.java (line
>>>>> 157) Completed flushing
>>>>> /var/lib/cassandra/data/_chip/Keyspace1/Standard1-e-7-Data.db
>>>>> INFO [FLUSH-WRITER-POOL:1] 2010-10-27 01:53:27,903 Memtable.java (line
>>>>> 150) Writing Memtable-Standard1@996627145(360 bytes, 10 operations)
>>>>> INFO [MUTATION_STAGE:3] 2010-10-27 01:53:27,927 ColumnFamilyStore.java
>>>>> (line 459) switching in a fresh Memtable for Standard1 at
>>>>>
>> CommitLogContext(file='/var/lib/cassandra/commitlog/_chip/CommitLog-1288169534132.log',
>>>>> position=31446)
>>>>> INFO [MUTATION_STAGE:3] 2010-10-27 01:53:27,927 ColumnFamilyStore.java
>>>>> (line 771) Enqueuing flush of Memtable-Standard1@1909350010(957 bytes,
>>>>> 26 operations)
>>>>> INFO [FLUSH-WRITER-POOL:1] 2010-10-27 01:53:27,932 Memtable.java (line
>>>>> 157) Completed flushing
>>>>> /var/lib/cassandra/data/_chip/Keyspace1/Standard1-e-8-Data.db
>>>>> INFO [FLUSH-WRITER-POOL:1] 2010-10-27 01:53:27,933 Memtable.java (line
>>>>> 150) Writing Memtable-Standard1@1383310803(384 bytes, 10 operations)
>>>>> INFO [CompactionExecutor:1] 2010-10-27 01:53:27,934
>>>>> CompactionManager.java (line 233) Compacting
>>>>>
>> [org.apache.cassandra.io.sstable.SSTableReader(path='/var/lib/cassandra/data/_chip/Keyspace1/Standard1-e-5-Data.db'),org.apache.cassandra.io.sstable.SSTableReader(path='/var/lib/cassandra/data/_chip/Keyspace1/Standard1-e-6-Data.db'),org.apache.cassandra.io.sstable.SSTableReader(path='/var/lib/cassandra/data/_chip/Keyspace1/Standard1-e-7-Data.db'),org.apache.cassandra.io.sstable.SSTableReader(path='/var/lib/cassandra/data/_chip/Keyspace1/Standard1-e-8-Data.db')]
>>>>> INFO [MUTATION_STAGE:17] 2010-10-27 01:53:27,936 ColumnFamilyStore.java
>>>>> (line 459) switching in a fresh Memtable for Standard1 at
>>>>>
>> CommitLogContext(file='/var/lib/cassandra/commitlog/_chip/CommitLog-1288169534132.log',
>>>>> position=33074)
>>>>> INFO [MUTATION_STAGE:17] 2010-10-27 01:53:27,936 ColumnFamilyStore.java
>>>>> (line 771) Enqueuing flush of Memtable-Standard1@595066677(396 bytes, 11
>>>>> operations)
>>>>> INFO [FLUSH-WRITER-POOL:1] 2010-10-27 01:53:28,037 Memtable.java (line
>>>>> 157) Completed flushing
>>>>> /var/lib/cassandra/data/_chip/Keyspace1/Standard1-e-9-Data.db
>>>>>
>>>>> I tried lowering the available memory reported by bin/cassandra:
>>>>>
>>>>> system_memory_in_mb=`free -m | awk '/Mem:/ {print $2}'`
>>>>> [ "$system_memory_in_mb" -gt 65536 ] &&
>>>>> system_memory_in_mb=65536 #<<<< new
>>>>>
>>>>> Didn't help. I also tried maually setting the max memtable sizes:
>>>>>
>>>>> binary_memtable_throughput_in_mb: 512
>>>>> memtable_throughput_in_mb: 4096
>>>>>
>>>>> Didn't help.
>>>>>
>>>>> Help?
>>>>>
>>>>
>
--
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of Riptano, the source for professional Cassandra support
http://riptano.com
Re: 0.7.0beta2 with 128G memory = bizarre tiny files. help?
Posted by Chip Salzenberg <ch...@pobox.com>.
OK, after upgrading to "apache-cassandra-2010-10-27_12-30-38", here's
what "show keyspaces" says about the relevant keyspace:
ColumnFamily: Standard1
Columns sorted by: org.apache.cassandra.db.marshal.BytesType
Row cache size / save period: 0.0/0
Key cache size / save period: 200000.0/3600
Memtable thresholds: 9.590625/2046/60
GC grace seconds: 864000
Compaction min/max thresholds: 4/32
I created it with "schematool import" off of the stock config in b2.
On 10/27/2010 11:36 AM, Jonathan Ellis wrote:
> Not in b2.
> On Oct 27, 2010 11:08 AM, "Chip Salzenberg" <ch...@pobox.com> wrote:
>> Are these values available in JMX? If so, which ones are they? I'd
>> prefer not to possibly destroy the evidence by upgrading.
>>
>> On 10/27/2010 6:50 AM, Jonathan Ellis wrote:
>>> Connect with the cli and run "show keyspaces." It sounds like
>>> Cassandra somehow picked ludicrously low memtable thresholds. There
>>> will be a line like
>>>
>>> Memtable thresholds: 0.2953125/63/60
>>>
>>> This may not be in the b2 cli, in which case you will need to upgrade
>>> to the nightly from
>>>
> http://hudson.zones.apache.org/hudson/job/Cassandra/lastSuccessfulBuild/artifact/cassandra/build/
>>> (essentially the same as rc1 being voted on).
>>>
>>> On Wed, Oct 27, 2010 at 4:17 AM, Chip Salzenberg <ch...@pobox.com> wrote:
>>>> I recently tested 0.7.0beta2 with good results on a reasonably powerful
>>>> machine: 8 Xeon cores (16 if you count hyperthreads), 64G memory, and
>>>> some nice HP RAID. So far so good. But when I took the same config and
>>>> moved it to a basically identical box with 128G of memory, cassandra
>>>> started responding to writes by creating a plethora of teeny tiny
>>>> sstables -- something it had not done before. For example:
>>>>
>>>> INFO [FLUSH-WRITER-POOL:1] 2010-10-27 01:53:27,881 Memtable.java (line
>>>> 150) Writing Memtable-Standard1@950886315(651 bytes, 18 operations)
>>>> INFO [MUTATION_STAGE:8] 2010-10-27 01:53:27,882 ColumnFamilyStore.java
>>>> (line 459) switching in a fresh Memtable for Standard1 at
>>>>
> CommitLogContext(file='/var/lib/cassandra/commitlog/_chip/CommitLog-1288169534132.log',
>>>> position=27598)
>>>> INFO [MUTATION_STAGE:8] 2010-10-27 01:53:27,883 ColumnFamilyStore.java
>>>> (line 771) Enqueuing flush of Memtable-Standard1@1383310803(384 bytes,
>>>> 10 operations)
>>>> INFO [FLUSH-WRITER-POOL:1] 2010-10-27 01:53:27,902 Memtable.java (line
>>>> 157) Completed flushing
>>>> /var/lib/cassandra/data/_chip/Keyspace1/Standard1-e-7-Data.db
>>>> INFO [FLUSH-WRITER-POOL:1] 2010-10-27 01:53:27,903 Memtable.java (line
>>>> 150) Writing Memtable-Standard1@996627145(360 bytes, 10 operations)
>>>> INFO [MUTATION_STAGE:3] 2010-10-27 01:53:27,927 ColumnFamilyStore.java
>>>> (line 459) switching in a fresh Memtable for Standard1 at
>>>>
> CommitLogContext(file='/var/lib/cassandra/commitlog/_chip/CommitLog-1288169534132.log',
>>>> position=31446)
>>>> INFO [MUTATION_STAGE:3] 2010-10-27 01:53:27,927 ColumnFamilyStore.java
>>>> (line 771) Enqueuing flush of Memtable-Standard1@1909350010(957 bytes,
>>>> 26 operations)
>>>> INFO [FLUSH-WRITER-POOL:1] 2010-10-27 01:53:27,932 Memtable.java (line
>>>> 157) Completed flushing
>>>> /var/lib/cassandra/data/_chip/Keyspace1/Standard1-e-8-Data.db
>>>> INFO [FLUSH-WRITER-POOL:1] 2010-10-27 01:53:27,933 Memtable.java (line
>>>> 150) Writing Memtable-Standard1@1383310803(384 bytes, 10 operations)
>>>> INFO [CompactionExecutor:1] 2010-10-27 01:53:27,934
>>>> CompactionManager.java (line 233) Compacting
>>>>
> [org.apache.cassandra.io.sstable.SSTableReader(path='/var/lib/cassandra/data/_chip/Keyspace1/Standard1-e-5-Data.db'),org.apache.cassandra.io.sstable.SSTableReader(path='/var/lib/cassandra/data/_chip/Keyspace1/Standard1-e-6-Data.db'),org.apache.cassandra.io.sstable.SSTableReader(path='/var/lib/cassandra/data/_chip/Keyspace1/Standard1-e-7-Data.db'),org.apache.cassandra.io.sstable.SSTableReader(path='/var/lib/cassandra/data/_chip/Keyspace1/Standard1-e-8-Data.db')]
>>>> INFO [MUTATION_STAGE:17] 2010-10-27 01:53:27,936 ColumnFamilyStore.java
>>>> (line 459) switching in a fresh Memtable for Standard1 at
>>>>
> CommitLogContext(file='/var/lib/cassandra/commitlog/_chip/CommitLog-1288169534132.log',
>>>> position=33074)
>>>> INFO [MUTATION_STAGE:17] 2010-10-27 01:53:27,936 ColumnFamilyStore.java
>>>> (line 771) Enqueuing flush of Memtable-Standard1@595066677(396 bytes, 11
>>>> operations)
>>>> INFO [FLUSH-WRITER-POOL:1] 2010-10-27 01:53:28,037 Memtable.java (line
>>>> 157) Completed flushing
>>>> /var/lib/cassandra/data/_chip/Keyspace1/Standard1-e-9-Data.db
>>>>
>>>> I tried lowering the available memory reported by bin/cassandra:
>>>>
>>>> system_memory_in_mb=`free -m | awk '/Mem:/ {print $2}'`
>>>> [ "$system_memory_in_mb" -gt 65536 ] &&
>>>> system_memory_in_mb=65536 #<<<< new
>>>>
>>>> Didn't help. I also tried maually setting the max memtable sizes:
>>>>
>>>> binary_memtable_throughput_in_mb: 512
>>>> memtable_throughput_in_mb: 4096
>>>>
>>>> Didn't help.
>>>>
>>>> Help?
>>>>
>>>
Re: 0.7.0beta2 with 128G memory = bizarre tiny files. help?
Posted by Jonathan Ellis <jb...@gmail.com>.
Not in b2.
On Oct 27, 2010 11:08 AM, "Chip Salzenberg" <ch...@pobox.com> wrote:
> Are these values available in JMX? If so, which ones are they? I'd
> prefer not to possibly destroy the evidence by upgrading.
>
> On 10/27/2010 6:50 AM, Jonathan Ellis wrote:
>> Connect with the cli and run "show keyspaces." It sounds like
>> Cassandra somehow picked ludicrously low memtable thresholds. There
>> will be a line like
>>
>> Memtable thresholds: 0.2953125/63/60
>>
>> This may not be in the b2 cli, in which case you will need to upgrade
>> to the nightly from
>>
http://hudson.zones.apache.org/hudson/job/Cassandra/lastSuccessfulBuild/artifact/cassandra/build/
>> (essentially the same as rc1 being voted on).
>>
>> On Wed, Oct 27, 2010 at 4:17 AM, Chip Salzenberg <ch...@pobox.com> wrote:
>>> I recently tested 0.7.0beta2 with good results on a reasonably powerful
>>> machine: 8 Xeon cores (16 if you count hyperthreads), 64G memory, and
>>> some nice HP RAID. So far so good. But when I took the same config and
>>> moved it to a basically identical box with 128G of memory, cassandra
>>> started responding to writes by creating a plethora of teeny tiny
>>> sstables -- something it had not done before. For example:
>>>
>>> INFO [FLUSH-WRITER-POOL:1] 2010-10-27 01:53:27,881 Memtable.java (line
>>> 150) Writing Memtable-Standard1@950886315(651 bytes, 18 operations)
>>> INFO [MUTATION_STAGE:8] 2010-10-27 01:53:27,882 ColumnFamilyStore.java
>>> (line 459) switching in a fresh Memtable for Standard1 at
>>>
CommitLogContext(file='/var/lib/cassandra/commitlog/_chip/CommitLog-1288169534132.log',
>>> position=27598)
>>> INFO [MUTATION_STAGE:8] 2010-10-27 01:53:27,883 ColumnFamilyStore.java
>>> (line 771) Enqueuing flush of Memtable-Standard1@1383310803(384 bytes,
>>> 10 operations)
>>> INFO [FLUSH-WRITER-POOL:1] 2010-10-27 01:53:27,902 Memtable.java (line
>>> 157) Completed flushing
>>> /var/lib/cassandra/data/_chip/Keyspace1/Standard1-e-7-Data.db
>>> INFO [FLUSH-WRITER-POOL:1] 2010-10-27 01:53:27,903 Memtable.java (line
>>> 150) Writing Memtable-Standard1@996627145(360 bytes, 10 operations)
>>> INFO [MUTATION_STAGE:3] 2010-10-27 01:53:27,927 ColumnFamilyStore.java
>>> (line 459) switching in a fresh Memtable for Standard1 at
>>>
CommitLogContext(file='/var/lib/cassandra/commitlog/_chip/CommitLog-1288169534132.log',
>>> position=31446)
>>> INFO [MUTATION_STAGE:3] 2010-10-27 01:53:27,927 ColumnFamilyStore.java
>>> (line 771) Enqueuing flush of Memtable-Standard1@1909350010(957 bytes,
>>> 26 operations)
>>> INFO [FLUSH-WRITER-POOL:1] 2010-10-27 01:53:27,932 Memtable.java (line
>>> 157) Completed flushing
>>> /var/lib/cassandra/data/_chip/Keyspace1/Standard1-e-8-Data.db
>>> INFO [FLUSH-WRITER-POOL:1] 2010-10-27 01:53:27,933 Memtable.java (line
>>> 150) Writing Memtable-Standard1@1383310803(384 bytes, 10 operations)
>>> INFO [CompactionExecutor:1] 2010-10-27 01:53:27,934
>>> CompactionManager.java (line 233) Compacting
>>>
[org.apache.cassandra.io.sstable.SSTableReader(path='/var/lib/cassandra/data/_chip/Keyspace1/Standard1-e-5-Data.db'),org.apache.cassandra.io.sstable.SSTableReader(path='/var/lib/cassandra/data/_chip/Keyspace1/Standard1-e-6-Data.db'),org.apache.cassandra.io.sstable.SSTableReader(path='/var/lib/cassandra/data/_chip/Keyspace1/Standard1-e-7-Data.db'),org.apache.cassandra.io.sstable.SSTableReader(path='/var/lib/cassandra/data/_chip/Keyspace1/Standard1-e-8-Data.db')]
>>> INFO [MUTATION_STAGE:17] 2010-10-27 01:53:27,936 ColumnFamilyStore.java
>>> (line 459) switching in a fresh Memtable for Standard1 at
>>>
CommitLogContext(file='/var/lib/cassandra/commitlog/_chip/CommitLog-1288169534132.log',
>>> position=33074)
>>> INFO [MUTATION_STAGE:17] 2010-10-27 01:53:27,936 ColumnFamilyStore.java
>>> (line 771) Enqueuing flush of Memtable-Standard1@595066677(396 bytes, 11
>>> operations)
>>> INFO [FLUSH-WRITER-POOL:1] 2010-10-27 01:53:28,037 Memtable.java (line
>>> 157) Completed flushing
>>> /var/lib/cassandra/data/_chip/Keyspace1/Standard1-e-9-Data.db
>>>
>>> I tried lowering the available memory reported by bin/cassandra:
>>>
>>> system_memory_in_mb=`free -m | awk '/Mem:/ {print $2}'`
>>> [ "$system_memory_in_mb" -gt 65536 ] &&
>>> system_memory_in_mb=65536 #<<<< new
>>>
>>> Didn't help. I also tried maually setting the max memtable sizes:
>>>
>>> binary_memtable_throughput_in_mb: 512
>>> memtable_throughput_in_mb: 4096
>>>
>>> Didn't help.
>>>
>>> Help?
>>>
>>
>>
Re: 0.7.0beta2 with 128G memory = bizarre tiny files. help?
Posted by Chip Salzenberg <ch...@pobox.com>.
Are these values available in JMX? If so, which ones are they? I'd
prefer not to possibly destroy the evidence by upgrading.
On 10/27/2010 6:50 AM, Jonathan Ellis wrote:
> Connect with the cli and run "show keyspaces." It sounds like
> Cassandra somehow picked ludicrously low memtable thresholds. There
> will be a line like
>
> Memtable thresholds: 0.2953125/63/60
>
> This may not be in the b2 cli, in which case you will need to upgrade
> to the nightly from
> http://hudson.zones.apache.org/hudson/job/Cassandra/lastSuccessfulBuild/artifact/cassandra/build/
> (essentially the same as rc1 being voted on).
>
> On Wed, Oct 27, 2010 at 4:17 AM, Chip Salzenberg <ch...@pobox.com> wrote:
>> I recently tested 0.7.0beta2 with good results on a reasonably powerful
>> machine: 8 Xeon cores (16 if you count hyperthreads), 64G memory, and
>> some nice HP RAID. So far so good. But when I took the same config and
>> moved it to a basically identical box with 128G of memory, cassandra
>> started responding to writes by creating a plethora of teeny tiny
>> sstables -- something it had not done before. For example:
>>
>> INFO [FLUSH-WRITER-POOL:1] 2010-10-27 01:53:27,881 Memtable.java (line
>> 150) Writing Memtable-Standard1@950886315(651 bytes, 18 operations)
>> INFO [MUTATION_STAGE:8] 2010-10-27 01:53:27,882 ColumnFamilyStore.java
>> (line 459) switching in a fresh Memtable for Standard1 at
>> CommitLogContext(file='/var/lib/cassandra/commitlog/_chip/CommitLog-1288169534132.log',
>> position=27598)
>> INFO [MUTATION_STAGE:8] 2010-10-27 01:53:27,883 ColumnFamilyStore.java
>> (line 771) Enqueuing flush of Memtable-Standard1@1383310803(384 bytes,
>> 10 operations)
>> INFO [FLUSH-WRITER-POOL:1] 2010-10-27 01:53:27,902 Memtable.java (line
>> 157) Completed flushing
>> /var/lib/cassandra/data/_chip/Keyspace1/Standard1-e-7-Data.db
>> INFO [FLUSH-WRITER-POOL:1] 2010-10-27 01:53:27,903 Memtable.java (line
>> 150) Writing Memtable-Standard1@996627145(360 bytes, 10 operations)
>> INFO [MUTATION_STAGE:3] 2010-10-27 01:53:27,927 ColumnFamilyStore.java
>> (line 459) switching in a fresh Memtable for Standard1 at
>> CommitLogContext(file='/var/lib/cassandra/commitlog/_chip/CommitLog-1288169534132.log',
>> position=31446)
>> INFO [MUTATION_STAGE:3] 2010-10-27 01:53:27,927 ColumnFamilyStore.java
>> (line 771) Enqueuing flush of Memtable-Standard1@1909350010(957 bytes,
>> 26 operations)
>> INFO [FLUSH-WRITER-POOL:1] 2010-10-27 01:53:27,932 Memtable.java (line
>> 157) Completed flushing
>> /var/lib/cassandra/data/_chip/Keyspace1/Standard1-e-8-Data.db
>> INFO [FLUSH-WRITER-POOL:1] 2010-10-27 01:53:27,933 Memtable.java (line
>> 150) Writing Memtable-Standard1@1383310803(384 bytes, 10 operations)
>> INFO [CompactionExecutor:1] 2010-10-27 01:53:27,934
>> CompactionManager.java (line 233) Compacting
>> [org.apache.cassandra.io.sstable.SSTableReader(path='/var/lib/cassandra/data/_chip/Keyspace1/Standard1-e-5-Data.db'),org.apache.cassandra.io.sstable.SSTableReader(path='/var/lib/cassandra/data/_chip/Keyspace1/Standard1-e-6-Data.db'),org.apache.cassandra.io.sstable.SSTableReader(path='/var/lib/cassandra/data/_chip/Keyspace1/Standard1-e-7-Data.db'),org.apache.cassandra.io.sstable.SSTableReader(path='/var/lib/cassandra/data/_chip/Keyspace1/Standard1-e-8-Data.db')]
>> INFO [MUTATION_STAGE:17] 2010-10-27 01:53:27,936 ColumnFamilyStore.java
>> (line 459) switching in a fresh Memtable for Standard1 at
>> CommitLogContext(file='/var/lib/cassandra/commitlog/_chip/CommitLog-1288169534132.log',
>> position=33074)
>> INFO [MUTATION_STAGE:17] 2010-10-27 01:53:27,936 ColumnFamilyStore.java
>> (line 771) Enqueuing flush of Memtable-Standard1@595066677(396 bytes, 11
>> operations)
>> INFO [FLUSH-WRITER-POOL:1] 2010-10-27 01:53:28,037 Memtable.java (line
>> 157) Completed flushing
>> /var/lib/cassandra/data/_chip/Keyspace1/Standard1-e-9-Data.db
>>
>> I tried lowering the available memory reported by bin/cassandra:
>>
>> system_memory_in_mb=`free -m | awk '/Mem:/ {print $2}'`
>> [ "$system_memory_in_mb" -gt 65536 ] &&
>> system_memory_in_mb=65536 #<<<< new
>>
>> Didn't help. I also tried maually setting the max memtable sizes:
>>
>> binary_memtable_throughput_in_mb: 512
>> memtable_throughput_in_mb: 4096
>>
>> Didn't help.
>>
>> Help?
>>
>
>