You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Eric Czech <er...@nextbigsound.com> on 2011/09/19 20:04:13 UTC
mmap segment underflow
I'm getting a lot of errors that look something like "java.io.IOError:
java.io.IOException: mmap segment underflow; remaining is 348268797
but 892417075 requested" on one node in a 10 node cluster. I'm
currently running version 0.8.4 but this is data that was carried over
from much earlier versions. Should I try to run scrub or are there
any other general guidelines for dealing with this sort of error?
Thanks everyone!
Re: mmap segment underflow
Posted by Eric Czech <er...@nextbigsound.com>.
I watched the logs pretty carefully as scrub ran and it definitely
skipped/deleted some problematic rows so perhaps that would explain the tmp
sstables but I'm not sure. Either way, they weren't still being written to
and after casting my restart spell, everything is totally fine and the tmp
tables are gone.
Thanks for the help Aaron!
On Sun, Oct 2, 2011 at 5:20 PM, aaron morton <aa...@thelastpickle.com>wrote:
> tmp files are temporary in nature, if they fail to live up to their name
> then they are against nature.
>
> If they are still been written to then something is happening, either a
> flush or a compaction. Check nodetool compactstats
>
> They may indicate errors during the scrub (or compact), check the logs for
> ERRORs. Note that scrub may log a lot non crashing errors that it could read
> a row, so read the error message carefully.
>
> When the server restarts it will clear tmp files.
>
> So if you unnerved at the prospect of unnatural temp files, exorcise them
> casting a restart spell.
>
> Hope that helps.
>
> -----------------
> Aaron Morton
> Freelance Cassandra Developer
> @aaronmorton
> http://www.thelastpickle.com
>
> On 2/10/2011, at 11:41 AM, Eric Czech wrote:
>
> Scrub seems to have worked. Thanks again!
>
> Will a major compaction delete the "tmp" sstables genereated though? Scrub
> seems to have generated a lot of them and they're taking up an unnerving
> amount of disk space.
>
> On Mon, Sep 19, 2011 at 5:34 PM, Eric Czech <er...@nextbigsound.com> wrote:
>
>> Ok then I'll shutdown the server, change the access mode, restart, and
>> run scrub (and then change the access mode back).
>>
>> Thanks for the pointers and I'll let you know how it goes one way or the
>> other.
>>
>> On Tue, Sep 20, 2011 at 12:29 AM, aaron morton <aa...@thelastpickle.com>
>> wrote:
>> > I've also found it useful to disable memmapped file access until the
>> scrub is complete by adding this to the yaml
>> >
>> > disk_access_mode: standard
>> >
>> > Cheers
>> >
>> > -----------------
>> > Aaron Morton
>> > Freelance Cassandra Developer
>> > @aaronmorton
>> > http://www.thelastpickle.com
>> >
>> > On 20/09/2011, at 6:55 AM, Jonathan Ellis wrote:
>> >
>> >> You should start with scrub.
>> >>
>> >> On Mon, Sep 19, 2011 at 1:04 PM, Eric Czech <er...@nextbigsound.com>
>> wrote:
>> >>> I'm getting a lot of errors that look something like "java.io.IOError:
>> >>> java.io.IOException: mmap segment underflow; remaining is 348268797
>> >>> but 892417075 requested" on one node in a 10 node cluster. I'm
>> >>> currently running version 0.8.4 but this is data that was carried over
>> >>> from much earlier versions. Should I try to run scrub or are there
>> >>> any other general guidelines for dealing with this sort of error?
>> >>>
>> >>> Thanks everyone!
>> >>>
>> >>
>> >>
>> >>
>> >> --
>> >> Jonathan Ellis
>> >> Project Chair, Apache Cassandra
>> >> co-founder of DataStax, the source for professional Cassandra support
>> >> http://www.datastax.com
>> >
>> >
>>
>
>
>
Re: mmap segment underflow
Posted by aaron morton <aa...@thelastpickle.com>.
tmp files are temporary in nature, if they fail to live up to their name then they are against nature.
If they are still been written to then something is happening, either a flush or a compaction. Check nodetool compactstats
They may indicate errors during the scrub (or compact), check the logs for ERRORs. Note that scrub may log a lot non crashing errors that it could read a row, so read the error message carefully.
When the server restarts it will clear tmp files.
So if you unnerved at the prospect of unnatural temp files, exorcise them casting a restart spell.
Hope that helps.
-----------------
Aaron Morton
Freelance Cassandra Developer
@aaronmorton
http://www.thelastpickle.com
On 2/10/2011, at 11:41 AM, Eric Czech wrote:
> Scrub seems to have worked. Thanks again!
>
> Will a major compaction delete the "tmp" sstables genereated though? Scrub seems to have generated a lot of them and they're taking up an unnerving amount of disk space.
>
> On Mon, Sep 19, 2011 at 5:34 PM, Eric Czech <er...@nextbigsound.com> wrote:
> Ok then I'll shutdown the server, change the access mode, restart, and
> run scrub (and then change the access mode back).
>
> Thanks for the pointers and I'll let you know how it goes one way or the other.
>
> On Tue, Sep 20, 2011 at 12:29 AM, aaron morton <aa...@thelastpickle.com> wrote:
> > I've also found it useful to disable memmapped file access until the scrub is complete by adding this to the yaml
> >
> > disk_access_mode: standard
> >
> > Cheers
> >
> > -----------------
> > Aaron Morton
> > Freelance Cassandra Developer
> > @aaronmorton
> > http://www.thelastpickle.com
> >
> > On 20/09/2011, at 6:55 AM, Jonathan Ellis wrote:
> >
> >> You should start with scrub.
> >>
> >> On Mon, Sep 19, 2011 at 1:04 PM, Eric Czech <er...@nextbigsound.com> wrote:
> >>> I'm getting a lot of errors that look something like "java.io.IOError:
> >>> java.io.IOException: mmap segment underflow; remaining is 348268797
> >>> but 892417075 requested" on one node in a 10 node cluster. I'm
> >>> currently running version 0.8.4 but this is data that was carried over
> >>> from much earlier versions. Should I try to run scrub or are there
> >>> any other general guidelines for dealing with this sort of error?
> >>>
> >>> Thanks everyone!
> >>>
> >>
> >>
> >>
> >> --
> >> Jonathan Ellis
> >> Project Chair, Apache Cassandra
> >> co-founder of DataStax, the source for professional Cassandra support
> >> http://www.datastax.com
> >
> >
>
Re: mmap segment underflow
Posted by Eric Czech <er...@nextbigsound.com>.
Scrub seems to have worked. Thanks again!
Will a major compaction delete the "tmp" sstables genereated though? Scrub
seems to have generated a lot of them and they're taking up an unnerving
amount of disk space.
On Mon, Sep 19, 2011 at 5:34 PM, Eric Czech <er...@nextbigsound.com> wrote:
> Ok then I'll shutdown the server, change the access mode, restart, and
> run scrub (and then change the access mode back).
>
> Thanks for the pointers and I'll let you know how it goes one way or the
> other.
>
> On Tue, Sep 20, 2011 at 12:29 AM, aaron morton <aa...@thelastpickle.com>
> wrote:
> > I've also found it useful to disable memmapped file access until the
> scrub is complete by adding this to the yaml
> >
> > disk_access_mode: standard
> >
> > Cheers
> >
> > -----------------
> > Aaron Morton
> > Freelance Cassandra Developer
> > @aaronmorton
> > http://www.thelastpickle.com
> >
> > On 20/09/2011, at 6:55 AM, Jonathan Ellis wrote:
> >
> >> You should start with scrub.
> >>
> >> On Mon, Sep 19, 2011 at 1:04 PM, Eric Czech <er...@nextbigsound.com>
> wrote:
> >>> I'm getting a lot of errors that look something like "java.io.IOError:
> >>> java.io.IOException: mmap segment underflow; remaining is 348268797
> >>> but 892417075 requested" on one node in a 10 node cluster. I'm
> >>> currently running version 0.8.4 but this is data that was carried over
> >>> from much earlier versions. Should I try to run scrub or are there
> >>> any other general guidelines for dealing with this sort of error?
> >>>
> >>> Thanks everyone!
> >>>
> >>
> >>
> >>
> >> --
> >> Jonathan Ellis
> >> Project Chair, Apache Cassandra
> >> co-founder of DataStax, the source for professional Cassandra support
> >> http://www.datastax.com
> >
> >
>
Re: mmap segment underflow
Posted by Eric Czech <er...@nextbigsound.com>.
Ok then I'll shutdown the server, change the access mode, restart, and
run scrub (and then change the access mode back).
Thanks for the pointers and I'll let you know how it goes one way or the other.
On Tue, Sep 20, 2011 at 12:29 AM, aaron morton <aa...@thelastpickle.com> wrote:
> I've also found it useful to disable memmapped file access until the scrub is complete by adding this to the yaml
>
> disk_access_mode: standard
>
> Cheers
>
> -----------------
> Aaron Morton
> Freelance Cassandra Developer
> @aaronmorton
> http://www.thelastpickle.com
>
> On 20/09/2011, at 6:55 AM, Jonathan Ellis wrote:
>
>> You should start with scrub.
>>
>> On Mon, Sep 19, 2011 at 1:04 PM, Eric Czech <er...@nextbigsound.com> wrote:
>>> I'm getting a lot of errors that look something like "java.io.IOError:
>>> java.io.IOException: mmap segment underflow; remaining is 348268797
>>> but 892417075 requested" on one node in a 10 node cluster. I'm
>>> currently running version 0.8.4 but this is data that was carried over
>>> from much earlier versions. Should I try to run scrub or are there
>>> any other general guidelines for dealing with this sort of error?
>>>
>>> Thanks everyone!
>>>
>>
>>
>>
>> --
>> Jonathan Ellis
>> Project Chair, Apache Cassandra
>> co-founder of DataStax, the source for professional Cassandra support
>> http://www.datastax.com
>
>
Re: mmap segment underflow
Posted by aaron morton <aa...@thelastpickle.com>.
I've also found it useful to disable memmapped file access until the scrub is complete by adding this to the yaml
disk_access_mode: standard
Cheers
-----------------
Aaron Morton
Freelance Cassandra Developer
@aaronmorton
http://www.thelastpickle.com
On 20/09/2011, at 6:55 AM, Jonathan Ellis wrote:
> You should start with scrub.
>
> On Mon, Sep 19, 2011 at 1:04 PM, Eric Czech <er...@nextbigsound.com> wrote:
>> I'm getting a lot of errors that look something like "java.io.IOError:
>> java.io.IOException: mmap segment underflow; remaining is 348268797
>> but 892417075 requested" on one node in a 10 node cluster. I'm
>> currently running version 0.8.4 but this is data that was carried over
>> from much earlier versions. Should I try to run scrub or are there
>> any other general guidelines for dealing with this sort of error?
>>
>> Thanks everyone!
>>
>
>
>
> --
> Jonathan Ellis
> Project Chair, Apache Cassandra
> co-founder of DataStax, the source for professional Cassandra support
> http://www.datastax.com
Re: mmap segment underflow
Posted by Jonathan Ellis <jb...@gmail.com>.
You should start with scrub.
On Mon, Sep 19, 2011 at 1:04 PM, Eric Czech <er...@nextbigsound.com> wrote:
> I'm getting a lot of errors that look something like "java.io.IOError:
> java.io.IOException: mmap segment underflow; remaining is 348268797
> but 892417075 requested" on one node in a 10 node cluster. I'm
> currently running version 0.8.4 but this is data that was carried over
> from much earlier versions. Should I try to run scrub or are there
> any other general guidelines for dealing with this sort of error?
>
> Thanks everyone!
>
--
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of DataStax, the source for professional Cassandra support
http://www.datastax.com