You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@accumulo.apache.org by William Slacum <ws...@gmail.com> on 2015/07/31 02:10:21 UTC

Re: How to control Minor Compaction by programming

See
http://accumulo.apache.org/1.5/apidocs/org/apache/accumulo/core/client/admin/TableOperations.html#flush%28java.lang.String,%20org.apache.hadoop.io.Text,%20org.apache.hadoop.io.Text,%20boolean%29
for minor compacting (aka "flushing") a table via the API.


On Thu, Jul 30, 2015 at 5:52 PM, Hai Pham <ht...@tigermail.auburn.edu>
wrote:

> Hi,
>
>
> Please share with me is there any way that we can init / control the Minor
> Compaction by programming (not from the shell). My situation is when I
> ingest a large data using the BatchWriter, the minor compaction is
> triggered uncontrollably. The flush() command in BatchWriter seems not for
> this purpose.
>
> I also tried to play around with parameters in documentation but seems not
> much helpful.
>
>
> Also, can you please explain the number 0, 1.0, 2.0, ... in charts (web
> monitoring) denoting the level of Minor Compaction and Major Compaction?
>
>
> Thank you!
>
> Hai Pham
>
>
>
>
>

Re: How to control Minor Compaction by programming

Posted by Hai Pham <ht...@tigermail.auburn.edu>.
Hi,


Yes, in fact I really want to avoid the minor compaction as much as possible, because during a long Ingest, any minor compaction largely blocks the speed of ingest.


But since the memory is limited, compaction is unavoidable, thus my desire is to control it as much as possible to harmonize the code accordingly.


Thanks,

Hai

________________________________
From: dlmarion@comcast.net <dl...@comcast.net>
Sent: Thursday, July 30, 2015 7:12 PM
To: user@accumulo.apache.org
Subject: RE: How to control Minor Compaction by programming


It sounds like you want to try and not minor compact during your ingest of your data. Is that correct?



From: William Slacum [mailto:wslacum@gmail.com]
Sent: Thursday, July 30, 2015 8:10 PM
To: user@accumulo.apache.org
Subject: Re: How to control Minor Compaction by programming



See http://accumulo.apache.org/1.5/apidocs/org/apache/accumulo/core/client/admin/TableOperations.html#flush%28java.lang.String,%20org.apache.hadoop.io.Text,%20org.apache.hadoop.io.Text,%20boolean%29 for minor compacting (aka "flushing") a table via the API.



On Thu, Jul 30, 2015 at 5:52 PM, Hai Pham <ht...@tigermail.auburn.edu>> wrote:

Hi,



Please share with me is there any way that we can init / control the Minor Compaction by programming (not from the shell). My situation is when I ingest a large data using the BatchWriter, the minor compaction is triggered uncontrollably. The flush() command in BatchWriter seems not for this purpose.

I also tried to play around with parameters in documentation but seems not much helpful.



Also, can you please explain the number 0, 1.0, 2.0, ... in charts (web monitoring) denoting the level of Minor Compaction and Major Compaction?



Thank you!

Hai Pham

RE: How to control Minor Compaction by programming

Posted by dl...@comcast.net.
It sounds like you want to try and not minor compact during your ingest of your data. Is that correct?

 

From: William Slacum [mailto:wslacum@gmail.com] 
Sent: Thursday, July 30, 2015 8:10 PM
To: user@accumulo.apache.org
Subject: Re: How to control Minor Compaction by programming

 

See http://accumulo.apache.org/1.5/apidocs/org/apache/accumulo/core/client/admin/TableOperations.html#flush%28java.lang.String,%20org.apache.hadoop.io.Text,%20org.apache.hadoop.io.Text,%20boolean%29 for minor compacting (aka "flushing") a table via the API.

 

On Thu, Jul 30, 2015 at 5:52 PM, Hai Pham <ht...@tigermail.auburn.edu> wrote:

Hi, 

 

Please share with me is there any way that we can init / control the Minor Compaction by programming (not from the shell). My situation is when I ingest a large data using the BatchWriter, the minor compaction is triggered uncontrollably. The flush() command in BatchWriter seems not for this purpose. 

I also tried to play around with parameters in documentation but seems not much helpful. 

 

Also, can you please explain the number 0, 1.0, 2.0, ... in charts (web monitoring) denoting the level of Minor Compaction and Major Compaction? 

 

Thank you! 

Hai Pham

Re: How to control Minor Compaction by programming

Posted by Hai Pham <ht...@tigermail.auburn.edu>.
Hi William,


It's probably what I have looked for, thus I will try soon. Thank you!


Hai

________________________________
From: William Slacum <ws...@gmail.com>
Sent: Thursday, July 30, 2015 7:11 PM
To: user@accumulo.apache.org
Subject: Re: How to control Minor Compaction by programming

Swap out 1.5 in the previous link for the version you're probably using.

Which charts are you looking at for the compactions? Usually it's just the number of compactions currently running for the system.

On Thu, Jul 30, 2015 at 7:10 PM, William Slacum <ws...@gmail.com>> wrote:
See http://accumulo.apache.org/1.5/apidocs/org/apache/accumulo/core/client/admin/TableOperations.html#flush%28java.lang.String,%20org.apache.hadoop.io.Text,%20org.apache.hadoop.io.Text,%20boolean%29 for minor compacting (aka "flushing") a table via the API.


On Thu, Jul 30, 2015 at 5:52 PM, Hai Pham <ht...@tigermail.auburn.edu>> wrote:

Hi,


Please share with me is there any way that we can init / control the Minor Compaction by programming (not from the shell). My situation is when I ingest a large data using the BatchWriter, the minor compaction is triggered uncontrollably. The flush() command in BatchWriter seems not for this purpose.

I also tried to play around with parameters in documentation but seems not much helpful.


Also, can you please explain the number 0, 1.0, 2.0, ... in charts (web monitoring) denoting the level of Minor Compaction and Major Compaction?


Thank you!

Hai Pham

Re: How to control Minor Compaction by programming

Posted by William Slacum <ws...@gmail.com>.
Swap out 1.5 in the previous link for the version you're probably using.

Which charts are you looking at for the compactions? Usually it's just the
number of compactions currently running for the system.

On Thu, Jul 30, 2015 at 7:10 PM, William Slacum <ws...@gmail.com> wrote:

> See
> http://accumulo.apache.org/1.5/apidocs/org/apache/accumulo/core/client/admin/TableOperations.html#flush%28java.lang.String,%20org.apache.hadoop.io.Text,%20org.apache.hadoop.io.Text,%20boolean%29
> for minor compacting (aka "flushing") a table via the API.
>
>
> On Thu, Jul 30, 2015 at 5:52 PM, Hai Pham <ht...@tigermail.auburn.edu>
> wrote:
>
>> Hi,
>>
>>
>> Please share with me is there any way that we can init / control the
>> Minor Compaction by programming (not from the shell). My situation is when
>> I ingest a large data using the BatchWriter, the minor compaction is
>> triggered uncontrollably. The flush() command in BatchWriter seems not for
>> this purpose.
>>
>> I also tried to play around with parameters in documentation but seems
>> not much helpful.
>>
>>
>> Also, can you please explain the number 0, 1.0, 2.0, ... in charts (web
>> monitoring) denoting the level of Minor Compaction and Major Compaction?
>>
>>
>> Thank you!
>>
>> Hai Pham
>>
>>
>>
>>
>>
>