You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@accumulo.apache.org by "Ligade, Shailesh [USA]" <Li...@bah.com> on 2022/04/14 13:14:18 UTC
minor compaction same as flush
Hello just wanted to some clarification,
Is the flush same as minor compaction? Is flush better (performance wise) than running say range compaction?
Having flush often, will it help major compaction performance or no difference??
Thanks
-S
RE: [External] Re: minor compaction same as flush
Posted by "Ligade, Shailesh [USA]" <Li...@bah.com>.
Thanks Christopher for detailed explanation.
Appreciated your response!
-S
-----Original Message-----
From: Christopher <ct...@apache.org>
Sent: Thursday, April 14, 2022 3:33 PM
To: accumulo-user <us...@accumulo.apache.org>
Subject: [External] Re: minor compaction same as flush
Ed's description is slightly wrong:
Yes, flush is the same as a minor compaction, writing in-memory data to an RFile. The shell calls this a "flush", because it's a more intuitive name than "minor compaction". In the technical documentation, it could be referred to as either, and some of our configuration options and APIs may have "minc" for minor compaction.
If you just say "compaction", without qualification, it implies "major compaction", which combines multiple files into one replacement file.
The shell has a "compact" command for forcing one to occur.
Furthermore, major compactions come in a few varieties: partial and full. A partial one is when only a subset of a tablet's file set are combined. Ed incorrectly referred to these partial major comapctions as a minor compaction, but these are still major compactions... they just don't include all files. A full major compaction is when all the tablet's files are combined. There are some operations (like dropping delete markers) that can only occur if the major compaction is of the full variety.
There is also something called a "merging minor compaction", which is a kind of flush/minor compaction that also picks up a small file to combine with. It was overly complex and not really necessary, so we removed it in 2.1 (not yet released).
There's also some special terms that refer to major compactions that occur under certain circumstances: chop compactions are partial major compactions that occur to truncate files in different tablets before those tablets are merged; idle compactions are major compactions that happen periodically when a tablet is not busy. Most compactions are system-initiated, but a user compaction is one that is user-initiated, such as from the shell's 'compact' command.
I hope that helps.
Re: minor compaction same as flush
Posted by Christopher <ct...@apache.org>.
Ed's description is slightly wrong:
Yes, flush is the same as a minor compaction, writing in-memory data
to an RFile. The shell calls this a "flush", because it's a more
intuitive name than "minor compaction". In the technical
documentation, it could be referred to as either, and some of our
configuration options and APIs may have "minc" for minor compaction.
If you just say "compaction", without qualification, it implies "major
compaction", which combines multiple files into one replacement file.
The shell has a "compact" command for forcing one to occur.
Furthermore, major compactions come in a few varieties: partial and
full. A partial one is when only a subset of a tablet's file set are
combined. Ed incorrectly referred to these partial major comapctions
as a minor compaction, but these are still major compactions... they
just don't include all files. A full major compaction is when all the
tablet's files are combined. There are some operations (like dropping
delete markers) that can only occur if the major compaction is of the
full variety.
There is also something called a "merging minor compaction", which is
a kind of flush/minor compaction that also picks up a small file to
combine with. It was overly complex and not really necessary, so we
removed it in 2.1 (not yet released).
There's also some special terms that refer to major compactions that
occur under certain circumstances: chop compactions are partial major
compactions that occur to truncate files in different tablets before
those tablets are merged; idle compactions are major compactions that
happen periodically when a tablet is not busy. Most compactions are
system-initiated, but a user compaction is one that is user-initiated,
such as from the shell's 'compact' command.
I hope that helps.
Re: minor compaction same as flush
Posted by Brian Loss <br...@gmail.com>.
Just a small correction re:
> Major compactions combine all files into a single file. Minor compactions select a subset of files can combines them into a file.
Minor compactions are when in-memory data is written to files. Major compactions are when files are combined. Major compactions can involve a subset of the files assigned to a tablet, or all of them (usually referred to as a full major compaction).
Flush forces minor compaction(s) to happen immediately vs when the in-memory tables fill up.
See:
https://accumulo.apache.org/1.10/accumulo_user_manual.html#_tablet_service <https://accumulo.apache.org/1.10/accumulo_user_manual.html#_tablet_service>
https://accumulo.apache.org/1.10/accumulo_user_manual.html#_compactions <https://accumulo.apache.org/1.10/accumulo_user_manual.html#_compactions>
> On Apr 14, 2022, at 9:36 AM, dev1 <de...@etcoleman.com> wrote:
>
> Flush and compactions are different actions.
>
> Flush – sorts and writes current, in-memory changes to a file. This can reduce the amount of recovery in case of a failure because the flushed entries do not need to be processed from the WAL.
>
> Compactions combine multiple files into a single file. Major compactions combine all files into a single file. Minor compactions select a subset of files can combines them into a file.
>
> See: https://accumulo.apache.org/1.10/accumulo_user_manual.html#_compaction <https://accumulo.apache.org/1.10/accumulo_user_manual.html#_compaction>
>
> Flushing will increase the number of files generated, this will potentially increase the number of compactions. There are tradeoffs. If you are asking will frequent flushes reduce the time required to perform a major compaction? Probably not much, if at all.
>
> Ed Coleman
>
> From: Ligade, Shailesh [USA] <Ligade_Shailesh@bah.com <ma...@bah.com>>
> Sent: Thursday, April 14, 2022 9:14 AM
> To: user@accumulo.apache.org <ma...@accumulo.apache.org>
> Subject: minor compaction same as flush
>
> Hello just wanted to some clarification,
>
> Is the flush same as minor compaction? Is flush better (performance wise) than running say range compaction?
> Having flush often, will it help major compaction performance or no difference??
>
> Thanks
>
> -S
RE: minor compaction same as flush
Posted by "Ligade, Shailesh [USA]" <Li...@bah.com>.
Thanks Ed,
Appreciated your clarification!
-S
From: dev1 <de...@etcoleman.com>
Sent: Thursday, April 14, 2022 9:36 AM
To: 'user@accumulo.apache.org' <us...@accumulo.apache.org>
Subject: [External] RE: minor compaction same as flush
Flush and compactions are different actions.
Flush - sorts and writes current, in-memory changes to a file. This can reduce the amount of recovery in case of a failure because the flushed entries do not need to be processed from the WAL.
Compactions combine multiple files into a single file. Major compactions combine all files into a single file. Minor compactions select a subset of files can combines them into a file.
See: https://accumulo.apache.org/1.10/accumulo_user_manual.html#_compaction<https://urldefense.com/v3/__https:/accumulo.apache.org/1.10/accumulo_user_manual.html*_compaction__;Iw!!May37g!JtX1JIc70ShpH3ztLGbjVuBPxlur4g_WBBG-NrAM5Ax5VOPy_41RN4hS2YuaHh9kaQaG5pPpAbdz8m2C$>
Flushing will increase the number of files generated, this will potentially increase the number of compactions. There are tradeoffs. If you are asking will frequent flushes reduce the time required to perform a major compaction? Probably not much, if at all.
Ed Coleman
From: Ligade, Shailesh [USA] <Li...@bah.com>>
Sent: Thursday, April 14, 2022 9:14 AM
To: user@accumulo.apache.org<ma...@accumulo.apache.org>
Subject: minor compaction same as flush
Hello just wanted to some clarification,
Is the flush same as minor compaction? Is flush better (performance wise) than running say range compaction?
Having flush often, will it help major compaction performance or no difference??
Thanks
-S
RE: minor compaction same as flush
Posted by dev1 <de...@etcoleman.com>.
Flush and compactions are different actions.
Flush - sorts and writes current, in-memory changes to a file. This can reduce the amount of recovery in case of a failure because the flushed entries do not need to be processed from the WAL.
Compactions combine multiple files into a single file. Major compactions combine all files into a single file. Minor compactions select a subset of files can combines them into a file.
See: https://accumulo.apache.org/1.10/accumulo_user_manual.html#_compaction
Flushing will increase the number of files generated, this will potentially increase the number of compactions. There are tradeoffs. If you are asking will frequent flushes reduce the time required to perform a major compaction? Probably not much, if at all.
Ed Coleman
From: Ligade, Shailesh [USA] <Li...@bah.com>
Sent: Thursday, April 14, 2022 9:14 AM
To: user@accumulo.apache.org
Subject: minor compaction same as flush
Hello just wanted to some clarification,
Is the flush same as minor compaction? Is flush better (performance wise) than running say range compaction?
Having flush often, will it help major compaction performance or no difference??
Thanks
-S