Should compaction be triggered even if User periodically flushes

[ramkrishna vasudevan <ra...@gmail.com>]

Hi devs

If user periodically flushes the memstore we keep creating Store files.
 Now if the number of store files  satisfies the compaction policy we dont
do compaction.

We print the msg saying compaction_requested=true but do not include into
the compactsplitthread.

But when the RS goes down and the region gets opened in another RS as part
of postOpenDeployTasks we do compaction because the store file count has
satisfied the compaction criteria.

What could be the right approach here?

Regards
Ram

Re: Should compaction be triggered even if User periodically flushes

[ramkrishna vasudevan <ra...@gmail.com>]

Raised HBASE-8282 with log snippets.

Regards
Ram


On Fri, Apr 5, 2013 at 11:14 PM, ramkrishna vasudevan <
ramkrishna.s.vasudevan@gmail.com> wrote:

> I will raise a JIRA and attach the logs.  I saw that in 0.94 also the
> behaviour remains the same.
> I did not change any of the default configruations.
>
> Regards
> Ram
>
>
> On Fri, Apr 5, 2013 at 10:55 PM, Sergey Shelukhin <se...@hortonworks.com>wrote:
>
>> +1... logs would be interesting to see, unless there was configuration
>> difference it should be the same.
>> The only thing that comes to mind if off-peak hours starting maybe?
>>
>> On Fri, Apr 5, 2013 at 9:57 AM, Jean-Daniel Cryans <jdcryans@apache.org
>> >wrote:
>>
>> > That sounds like a bug to me. Can you find why it's not being compacted?
>> > Was this behavior changed at some point?
>> >
>> > Thx,
>> >
>> > J-D
>> >
>> >
>> > On Fri, Apr 5, 2013 at 9:50 AM, ramkrishna vasudevan <
>> > ramkrishna.s.vasudevan@gmail.com> wrote:
>> >
>> > > Hi JM
>> > > Thanks for your reply.  My concern is not with respect to compaction
>> > > happening when a region is opening in another RS, my point was though
>> the
>> > > flushes were happening and more storefiles are created
>> > > by user initiated flushing, the compaction were not getting triggered.
>> > >
>> > >
>> > > On Fri, Apr 5, 2013 at 4:50 PM, Jean-Marc Spaggiari <
>> > > jean-marc@spaggiari.org
>> > > > wrote:
>> > >
>> > > > Hi Ram,
>> > > >
>> > > > What do you think is wrong with the approach below?
>> > > >
>> > > > Doing the compaction when the region is opened in another RS will
>> > > > improve data locality.
>> > > >
>> > > > JM
>> > > >
>> > > > 2013/4/5 ramkrishna vasudevan <ra...@gmail.com>:
>> > > > > Hi devs
>> > > > >
>> > > > > If user periodically flushes the memstore we keep creating Store
>> > files.
>> > > > >  Now if the number of store files  satisfies the compaction
>> policy we
>> > > > dont
>> > > > > do compaction.
>> > > > >
>> > > > > We print the msg saying compaction_requested=true but do not
>> include
>> > > into
>> > > > > the compactsplitthread.
>> > > > >
>> > > > > But when the RS goes down and the region gets opened in another
>> RS as
>> > > > part
>> > > > > of postOpenDeployTasks we do compaction because the store file
>> count
>> > > has
>> > > > > satisfied the compaction criteria.
>> > > > >
>> > > > > What could be the right approach here?
>> > > > >
>> > > > > Regards
>> > > > > Ram
>> > > >
>> > >
>> >
>>
>
>

Re: Should compaction be triggered even if User periodically flushes

[ramkrishna vasudevan <ra...@gmail.com>]

I will raise a JIRA and attach the logs.  I saw that in 0.94 also the
behaviour remains the same.
I did not change any of the default configruations.

Regards
Ram


On Fri, Apr 5, 2013 at 10:55 PM, Sergey Shelukhin <se...@hortonworks.com>wrote:

> +1... logs would be interesting to see, unless there was configuration
> difference it should be the same.
> The only thing that comes to mind if off-peak hours starting maybe?
>
> On Fri, Apr 5, 2013 at 9:57 AM, Jean-Daniel Cryans <jdcryans@apache.org
> >wrote:
>
> > That sounds like a bug to me. Can you find why it's not being compacted?
> > Was this behavior changed at some point?
> >
> > Thx,
> >
> > J-D
> >
> >
> > On Fri, Apr 5, 2013 at 9:50 AM, ramkrishna vasudevan <
> > ramkrishna.s.vasudevan@gmail.com> wrote:
> >
> > > Hi JM
> > > Thanks for your reply.  My concern is not with respect to compaction
> > > happening when a region is opening in another RS, my point was though
> the
> > > flushes were happening and more storefiles are created
> > > by user initiated flushing, the compaction were not getting triggered.
> > >
> > >
> > > On Fri, Apr 5, 2013 at 4:50 PM, Jean-Marc Spaggiari <
> > > jean-marc@spaggiari.org
> > > > wrote:
> > >
> > > > Hi Ram,
> > > >
> > > > What do you think is wrong with the approach below?
> > > >
> > > > Doing the compaction when the region is opened in another RS will
> > > > improve data locality.
> > > >
> > > > JM
> > > >
> > > > 2013/4/5 ramkrishna vasudevan <ra...@gmail.com>:
> > > > > Hi devs
> > > > >
> > > > > If user periodically flushes the memstore we keep creating Store
> > files.
> > > > >  Now if the number of store files  satisfies the compaction policy
> we
> > > > dont
> > > > > do compaction.
> > > > >
> > > > > We print the msg saying compaction_requested=true but do not
> include
> > > into
> > > > > the compactsplitthread.
> > > > >
> > > > > But when the RS goes down and the region gets opened in another RS
> as
> > > > part
> > > > > of postOpenDeployTasks we do compaction because the store file
> count
> > > has
> > > > > satisfied the compaction criteria.
> > > > >
> > > > > What could be the right approach here?
> > > > >
> > > > > Regards
> > > > > Ram
> > > >
> > >
> >
>

Re: Should compaction be triggered even if User periodically flushes

[Sergey Shelukhin <se...@hortonworks.com>]

+1... logs would be interesting to see, unless there was configuration
difference it should be the same.
The only thing that comes to mind if off-peak hours starting maybe?

On Fri, Apr 5, 2013 at 9:57 AM, Jean-Daniel Cryans <jd...@apache.org>wrote:

> That sounds like a bug to me. Can you find why it's not being compacted?
> Was this behavior changed at some point?
>
> Thx,
>
> J-D
>
>
> On Fri, Apr 5, 2013 at 9:50 AM, ramkrishna vasudevan <
> ramkrishna.s.vasudevan@gmail.com> wrote:
>
> > Hi JM
> > Thanks for your reply.  My concern is not with respect to compaction
> > happening when a region is opening in another RS, my point was though the
> > flushes were happening and more storefiles are created
> > by user initiated flushing, the compaction were not getting triggered.
> >
> >
> > On Fri, Apr 5, 2013 at 4:50 PM, Jean-Marc Spaggiari <
> > jean-marc@spaggiari.org
> > > wrote:
> >
> > > Hi Ram,
> > >
> > > What do you think is wrong with the approach below?
> > >
> > > Doing the compaction when the region is opened in another RS will
> > > improve data locality.
> > >
> > > JM
> > >
> > > 2013/4/5 ramkrishna vasudevan <ra...@gmail.com>:
> > > > Hi devs
> > > >
> > > > If user periodically flushes the memstore we keep creating Store
> files.
> > > >  Now if the number of store files  satisfies the compaction policy we
> > > dont
> > > > do compaction.
> > > >
> > > > We print the msg saying compaction_requested=true but do not include
> > into
> > > > the compactsplitthread.
> > > >
> > > > But when the RS goes down and the region gets opened in another RS as
> > > part
> > > > of postOpenDeployTasks we do compaction because the store file count
> > has
> > > > satisfied the compaction criteria.
> > > >
> > > > What could be the right approach here?
> > > >
> > > > Regards
> > > > Ram
> > >
> >
>

Re: Should compaction be triggered even if User periodically flushes

[Jean-Daniel Cryans <jd...@apache.org>]

That sounds like a bug to me. Can you find why it's not being compacted?
Was this behavior changed at some point?

Thx,

J-D


On Fri, Apr 5, 2013 at 9:50 AM, ramkrishna vasudevan <
ramkrishna.s.vasudevan@gmail.com> wrote:

> Hi JM
> Thanks for your reply.  My concern is not with respect to compaction
> happening when a region is opening in another RS, my point was though the
> flushes were happening and more storefiles are created
> by user initiated flushing, the compaction were not getting triggered.
>
>
> On Fri, Apr 5, 2013 at 4:50 PM, Jean-Marc Spaggiari <
> jean-marc@spaggiari.org
> > wrote:
>
> > Hi Ram,
> >
> > What do you think is wrong with the approach below?
> >
> > Doing the compaction when the region is opened in another RS will
> > improve data locality.
> >
> > JM
> >
> > 2013/4/5 ramkrishna vasudevan <ra...@gmail.com>:
> > > Hi devs
> > >
> > > If user periodically flushes the memstore we keep creating Store files.
> > >  Now if the number of store files  satisfies the compaction policy we
> > dont
> > > do compaction.
> > >
> > > We print the msg saying compaction_requested=true but do not include
> into
> > > the compactsplitthread.
> > >
> > > But when the RS goes down and the region gets opened in another RS as
> > part
> > > of postOpenDeployTasks we do compaction because the store file count
> has
> > > satisfied the compaction criteria.
> > >
> > > What could be the right approach here?
> > >
> > > Regards
> > > Ram
> >
>

Re: Should compaction be triggered even if User periodically flushes

[ramkrishna vasudevan <ra...@gmail.com>]

Hi JM
Thanks for your reply.  My concern is not with respect to compaction
happening when a region is opening in another RS, my point was though the
flushes were happening and more storefiles are created
by user initiated flushing, the compaction were not getting triggered.


On Fri, Apr 5, 2013 at 4:50 PM, Jean-Marc Spaggiari <jean-marc@spaggiari.org
> wrote:

> Hi Ram,
>
> What do you think is wrong with the approach below?
>
> Doing the compaction when the region is opened in another RS will
> improve data locality.
>
> JM
>
> 2013/4/5 ramkrishna vasudevan <ra...@gmail.com>:
> > Hi devs
> >
> > If user periodically flushes the memstore we keep creating Store files.
> >  Now if the number of store files  satisfies the compaction policy we
> dont
> > do compaction.
> >
> > We print the msg saying compaction_requested=true but do not include into
> > the compactsplitthread.
> >
> > But when the RS goes down and the region gets opened in another RS as
> part
> > of postOpenDeployTasks we do compaction because the store file count has
> > satisfied the compaction criteria.
> >
> > What could be the right approach here?
> >
> > Regards
> > Ram
>

Re: Should compaction be triggered even if User periodically flushes

[Jean-Marc Spaggiari <je...@spaggiari.org>]

Hi Ram,

What do you think is wrong with the approach below?

Doing the compaction when the region is opened in another RS will
improve data locality.

JM

2013/4/5 ramkrishna vasudevan <ra...@gmail.com>:
> Hi devs
>
> If user periodically flushes the memstore we keep creating Store files.
>  Now if the number of store files  satisfies the compaction policy we dont
> do compaction.
>
> We print the msg saying compaction_requested=true but do not include into
> the compactsplitthread.
>
> But when the RS goes down and the region gets opened in another RS as part
> of postOpenDeployTasks we do compaction because the store file count has
> satisfied the compaction criteria.
>
> What could be the right approach here?
>
> Regards
> Ram