You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@carbondata.apache.org by Kunal Kapoor <ku...@knoldus.in> on 2017/04/13 06:33:40 UTC
[DISCUSSION] Forceful minor Compaction
Hi all,
I was looking into compaction and had a query regarding the same.
If you have auto compaction turned on and the threshold level is 4,3
Now try loading data 7 times which will make 7 segments. Now as the auto
compaction was on therefore 4 segments will be merged to 0.1.
The segments visible will be 0.1, 4, 5, 6.
When i try to run the compaction command nothing happens because the
threshold level is not reached for compaction to happen.
What if i want to merge the 3 segments i.e (4, 5, 6) and make a level-1
compacted segment.
The proposed solution would be to add a new option to the compaction
command which contains what level of compaction the user would like to do
Example:- alter table carbon_table compact 'minor' level '1'.
This would forcefully combine the segments(4, 5, 6) in to a level 1
compacted segment called 4.1 giving me 2 level-1 compacted segment (0.1 and
4.1)
Similar operation can be done with level-1 compacted segments.
Re: [DISCUSSION] Forceful minor Compaction
Posted by Venkata Gollamudi <g....@gmail.com>.
Forceful compaction might be required when user really wants control and
wanted to force trigger compaction on required segments.
How ever need to identify cases and its behavior , so that it can co exist
with current compaction model, with out disturbing or imposing any
limitations.
I feel forceful major compaction specifying segments will be better option
than forceful minor.
Regards,
Venkata Ramana Gollamudi
On Apr 19, 2017 14:50, "Kunal Kapoor" <ku...@knoldus.in> wrote:
> Hi Liang,
> Currently the user cannot choose which segments should be merged or not.
> We should give a DDL which supports both forceful minor and major
> compaction separately.
>
> Another point would be if the user can specify the segments which he needs
> merged. Obviously the segments that can be given should meet some criteria.
> for example:-- no major compacted or 2nd Level compacted segments can be
> provided during forced minor compaction.
>
>
> --
> Regards
>
> *Kunal Kapoor*
> Software Consultant
>
> *Knoldus Software LLP <http://www.knoldus.com/home.knol>*
> Twitter <http://twitter/kunal642> | LinkedIn
> <https://in.linkedin.com/in/kunal642> | FB
> <https://www.facebook.com/kunalkapoor642>
> Mobile: +91-9711013430
>
> On Wed, Apr 19, 2017 at 2:33 PM, Liang Chen <ch...@gmail.com>
> wrote:
>
> > Hi Kunal
> >
> > Thank you for taking the good topic for discussion.
> > First , let us think about : why users want to do forceful minor
> > compaction,
> > which cases?
> > Current "MAJOR compaction" whether can cover "forceful MINOR compaction"
> > scenarios ?
> >
> > As we know, compaction is mainly for optimizing index effectiveness by
> > merging for multiple times data loading(segments), so current system
> > provides two options for users to choose :"soft compaction(minor) and
> > strong
> > compaction(major)"。
> >
> > So i could not find the typical cases to add a new strong
> > compaction(forceful minor)。
> >
> > Regards
> > Liang
> >
> >
> > Kunal Kapoor wrote
> > > Hi all,
> > > I was looking into compaction and had a query regarding the same.
> > > If you have auto compaction turned on and the threshold level is 4,3
> > > Now try loading data 7 times which will make 7 segments. Now as the
> auto
> > > compaction was on therefore 4 segments will be merged to 0.1.
> > > The segments visible will be 0.1, 4, 5, 6.
> > >
> > > When i try to run the compaction command nothing happens because the
> > > threshold level is not reached for compaction to happen.
> > > What if i want to merge the 3 segments i.e (4, 5, 6) and make a level-1
> > > compacted segment.
> > >
> > > The proposed solution would be to add a new option to the compaction
> > > command which contains what level of compaction the user would like to
> do
> > >
> > > Example:- alter table carbon_table compact 'minor' level '1'.
> > > This would forcefully combine the segments(4, 5, 6) in to a level 1
> > > compacted segment called 4.1 giving me 2 level-1 compacted segment (0.1
> > > and
> > > 4.1)
> > > Similar operation can be done with level-1 compacted segments.
> >
> >
> >
> >
> >
> > --
> > View this message in context: http://apache-carbondata-
> > mailing-list-archive.1130556.n5.nabble.com/DISCUSSION-
> > Forceful-minor-Compaction-tp10924p11319.html
> > Sent from the Apache CarbonData Mailing List archive mailing list archive
> > at Nabble.com.
> >
>
Re: [DISCUSSION] Forceful minor Compaction
Posted by Kunal Kapoor <ku...@knoldus.in>.
Hi Liang,
Currently the user cannot choose which segments should be merged or not.
We should give a DDL which supports both forceful minor and major
compaction separately.
Another point would be if the user can specify the segments which he needs
merged. Obviously the segments that can be given should meet some criteria.
for example:-- no major compacted or 2nd Level compacted segments can be
provided during forced minor compaction.
--
Regards
*Kunal Kapoor*
Software Consultant
*Knoldus Software LLP <http://www.knoldus.com/home.knol>*
Twitter <http://twitter/kunal642> | LinkedIn
<https://in.linkedin.com/in/kunal642> | FB
<https://www.facebook.com/kunalkapoor642>
Mobile: +91-9711013430
On Wed, Apr 19, 2017 at 2:33 PM, Liang Chen <ch...@gmail.com> wrote:
> Hi Kunal
>
> Thank you for taking the good topic for discussion.
> First , let us think about : why users want to do forceful minor
> compaction,
> which cases?
> Current "MAJOR compaction" whether can cover "forceful MINOR compaction"
> scenarios ?
>
> As we know, compaction is mainly for optimizing index effectiveness by
> merging for multiple times data loading(segments), so current system
> provides two options for users to choose :"soft compaction(minor) and
> strong
> compaction(major)"。
>
> So i could not find the typical cases to add a new strong
> compaction(forceful minor)。
>
> Regards
> Liang
>
>
> Kunal Kapoor wrote
> > Hi all,
> > I was looking into compaction and had a query regarding the same.
> > If you have auto compaction turned on and the threshold level is 4,3
> > Now try loading data 7 times which will make 7 segments. Now as the auto
> > compaction was on therefore 4 segments will be merged to 0.1.
> > The segments visible will be 0.1, 4, 5, 6.
> >
> > When i try to run the compaction command nothing happens because the
> > threshold level is not reached for compaction to happen.
> > What if i want to merge the 3 segments i.e (4, 5, 6) and make a level-1
> > compacted segment.
> >
> > The proposed solution would be to add a new option to the compaction
> > command which contains what level of compaction the user would like to do
> >
> > Example:- alter table carbon_table compact 'minor' level '1'.
> > This would forcefully combine the segments(4, 5, 6) in to a level 1
> > compacted segment called 4.1 giving me 2 level-1 compacted segment (0.1
> > and
> > 4.1)
> > Similar operation can be done with level-1 compacted segments.
>
>
>
>
>
> --
> View this message in context: http://apache-carbondata-
> mailing-list-archive.1130556.n5.nabble.com/DISCUSSION-
> Forceful-minor-Compaction-tp10924p11319.html
> Sent from the Apache CarbonData Mailing List archive mailing list archive
> at Nabble.com.
>
Re: [DISCUSSION] Forceful minor Compaction
Posted by Liang Chen <ch...@gmail.com>.
Hi Kunal
Thank you for taking the good topic for discussion.
First , let us think about : why users want to do forceful minor compaction,
which cases?
Current "MAJOR compaction" whether can cover "forceful MINOR compaction"
scenarios ?
As we know, compaction is mainly for optimizing index effectiveness by
merging for multiple times data loading(segments), so current system
provides two options for users to choose :"soft compaction(minor) and strong
compaction(major)"。
So i could not find the typical cases to add a new strong
compaction(forceful minor)。
Regards
Liang
Kunal Kapoor wrote
> Hi all,
> I was looking into compaction and had a query regarding the same.
> If you have auto compaction turned on and the threshold level is 4,3
> Now try loading data 7 times which will make 7 segments. Now as the auto
> compaction was on therefore 4 segments will be merged to 0.1.
> The segments visible will be 0.1, 4, 5, 6.
>
> When i try to run the compaction command nothing happens because the
> threshold level is not reached for compaction to happen.
> What if i want to merge the 3 segments i.e (4, 5, 6) and make a level-1
> compacted segment.
>
> The proposed solution would be to add a new option to the compaction
> command which contains what level of compaction the user would like to do
>
> Example:- alter table carbon_table compact 'minor' level '1'.
> This would forcefully combine the segments(4, 5, 6) in to a level 1
> compacted segment called 4.1 giving me 2 level-1 compacted segment (0.1
> and
> 4.1)
> Similar operation can be done with level-1 compacted segments.
--
View this message in context: http://apache-carbondata-mailing-list-archive.1130556.n5.nabble.com/DISCUSSION-Forceful-minor-Compaction-tp10924p11319.html
Sent from the Apache CarbonData Mailing List archive mailing list archive at Nabble.com.