You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@accumulo.apache.org by Massimilian Mattetti <MA...@il.ibm.com> on 2017/03/23 11:54:53 UTC
tserver.compaction.*.concurrent.max behavior in Accumulo 1.8.1
Hi All,
I am running some heavy ingestion process on a 3 nodes cluster of Accumulo
1.8.1, using the following configuration:
table.compaction.minor.logs.threshold=10
table.durability=flush
table.file.max=30
tserver.wal.blocksize=2G
tserver.walog.max.size=4G
tserver.mutation.queue.max=2M
tserver.memory.maps.max=4G
tserver.compaction.minor.concurrent.max=50
tserver.compaction.major.concurrent.max=8
My problem is that both the minor and major compactions do not overcome
their default max values. I checked the config from the shell and it looks
fine to me:
default | tserver.compaction.major.concurrent.max ................ | 3
system | @override
........................................... | 8
default | tserver.compaction.minor.concurrent.max ............... | 4
system | @override
........................................... | 50
Is something changed from 1.8.0? I haven't seen such behavior with the
previous version.
Thanks.
Regards,
Max
Re: tserver.compaction.*.concurrent.max behavior in Accumulo 1.8.1
Posted by Massimilian Mattetti <MA...@il.ibm.com>.
The tserver.compaction.major.thread.files.open.max=10 and I have a maximum
of 30 files per tablet (table.file.max=30). What property are you
referring to for the max open files?
Thanks,
Max
From: "Marc P." <ma...@gmail.com>
To: user@accumulo.apache.org
Date: 23/03/2017 16:37
Subject: Re: tserver.compaction.*.concurrent.max behavior in
Accumulo 1.8.1
Max,
Additionally, what are your max open files? If you are under heavy
ingest you may exceed this with compactions if they cannot keep up.
Additionally, how many files per compaction are you allowing?
On Thu, Mar 23, 2017 at 10:27 AM, Michael Wall <mj...@gmail.com> wrote:
Max,
On you 3 node cluster, how many tables are you ingesting into? How many
tablets are in each table? Are the tablets equally spread amongst the 3
tablet servers?
Mike
On Thu, Mar 23, 2017 at 10:13 AM Massimilian Mattetti <MASSIMIL@il.ibm.com
> wrote:
With the configuration I presented before the concurrent major compactions
are never more than 3 per tablet server while the minor are under the 4
per node. Can one of the other configurations be the cause of this
behavior?
Regards,
Max
From: Dave Marion <dl...@comcast.net>
To: user@accumulo.apache.org, Massimilian Mattetti/Haifa/IBM@IBMIL
Date: 23/03/2017 14:55
Subject: Re: tserver.compaction.*.concurrent.max behavior in
Accumulo 1.8.1
Can you explain more what you mean by "My problem is that both the minor
and major compactions do not overcome their default max values?" I have
done some testing with 1.8.1 and specifically modifying
tserver.compaction.major.concurrent.max to a higher number and I have seen
it take effect.
On March 23, 2017 at 7:54 AM Massimilian Mattetti <MA...@il.ibm.com>
wrote:
Hi All,
I am running some heavy ingestion process on a 3 nodes cluster of Accumulo
1.8.1, using the following configuration:
table.compaction.minor.logs.threshold=10
table.durability=flush
table.file.max=30
tserver.wal.blocksize=2G
tserver.walog.max.size=4G
tserver.mutation.queue.max=2M
tserver.memory.maps.max=4G
tserver.compaction.minor.concurrent.max=50
tserver.compaction.major.concurrent.max=8
My problem is that both the minor and major compactions do not overcome
their default max values. I checked the config from the shell and it looks
fine to me:
default | tserver.compaction.major.concurrent.max ................ | 3
system | @override
........................................... | 8
default | tserver.compaction.minor.concurrent.max ............... | 4
system | @override
........................................... | 50
Is something changed from 1.8.0? I haven't seen such behavior with the
previous version.
Thanks.
Regards,
Max
Re: tserver.compaction.*.concurrent.max behavior in Accumulo 1.8.1
Posted by "Marc P." <ma...@gmail.com>.
Max,
Additionally, what are your max open files? If you are under heavy ingest
you may exceed this with compactions if they cannot keep up. Additionally,
how many files per compaction are you allowing?
On Thu, Mar 23, 2017 at 10:27 AM, Michael Wall <mj...@gmail.com> wrote:
> Max,
>
> On you 3 node cluster, how many tables are you ingesting into? How many
> tablets are in each table? Are the tablets equally spread amongst the 3
> tablet servers?
>
> Mike
>
> On Thu, Mar 23, 2017 at 10:13 AM Massimilian Mattetti <MA...@il.ibm.com>
> wrote:
>
>> With the configuration I presented before the concurrent major
>> compactions are never more than 3 per tablet server while the minor are
>> under the 4 per node. Can one of the other configurations be the cause of
>> this behavior?
>>
>> Regards,
>> Max
>>
>>
>>
>> From: Dave Marion <dl...@comcast.net>
>> To: user@accumulo.apache.org, Massimilian Mattetti/Haifa/IBM@IBMIL
>> Date: 23/03/2017 14:55
>> Subject: Re: tserver.compaction.*.concurrent.max behavior in
>> Accumulo 1.8.1
>> ------------------------------
>>
>>
>>
>> Can you explain more what you mean by "My problem is that both the minor
>> and major compactions do not overcome their default max values?" I have
>> done some testing with 1.8.1 and specifically modifying
>> tserver.compaction.major.concurrent.max to a higher number and I have
>> seen it take effect.
>>
>> On March 23, 2017 at 7:54 AM Massimilian Mattetti <MA...@il.ibm.com>
>> wrote:
>>
>> Hi All,
>>
>> I am running some heavy ingestion process on a 3 nodes cluster of
>> Accumulo 1.8.1, using the following configuration:
>>
>> table.compaction.minor.logs.threshold=10
>> table.durability=flush
>> table.file.max=30
>>
>> tserver.wal.blocksize=2G
>> tserver.walog.max.size=4G
>> tserver.mutation.queue.max=2M
>> tserver.memory.maps.max=4G
>> tserver.compaction.minor.concurrent.max=50
>> tserver.compaction.major.concurrent.max=8
>>
>> My problem is that both the minor and major compactions do not overcome
>> their default max values. I checked the config from the shell and it looks
>> fine to me:
>>
>> default | tserver.compaction.major.concurrent.max ................ | 3
>> system | @override ...........................................
>> | 8
>>
>> default | tserver.compaction.minor.concurrent.max ............... | 4
>> system | @override ...........................................
>> | 50
>>
>> Is something changed from 1.8.0? I haven't seen such behavior with the
>> previous version.
>> Thanks.
>>
>> Regards,
>> Max
>>
>>
>>
>>
>>
Re: tserver.compaction.*.concurrent.max behavior in Accumulo 1.8.1
Posted by Keith Turner <ke...@deenlo.com>.
On Thu, Mar 23, 2017 at 12:37 PM, Massimilian Mattetti
<MA...@il.ibm.com> wrote:
> I just tested the ingestion with 10 tablets per Server and now the system
> achieve up to 24 concurrent major compactions.
>
> I have another issue with the tserver.memory.maps.max property. I am play
> with the size of the native map to track how its size affect the ingestion
> performance. I set the values of table.compaction.minor.logs.threshold and
> tserver.walog.max.size big enough so I can increase the map size up to 40GB
> without problems. At the beginning changing the size of the map (using the
> shell) didn't bring any benefit to the ingestion process. Monitoring the
> servers I noticed that their RAM consumption was constant. Eventually I
> tried to restart the tablet servers after changing the
> tserver.memory.maps.max and the RAM usage on each node increased as I
> expected. From the documentation I know that any change to the
> tserver.memory.maps.max should take effect without restarting the tablet
> servers, is that always true? (tserver.memory.maps.native.enabled has always
> been true and from the logs I see that the shared library has been correctly
> loaded).
> Thanks!
I took a quick look at the 1.6.5 code and it does not seem to update
after tserver start in that version.
>
> Best Regards,
> Max
>
>
>
> From: Michael Wall <mj...@gmail.com>
> To: user@accumulo.apache.org
> Date: 23/03/2017 17:49
>
> Subject: Re: tserver.compaction.*.concurrent.max behavior in Accumulo
> 1.8.1
> ________________________________
>
>
>
> Yes, until you hit another constraint like Marc and Dave were asking about.
>
> Mike
>
> On Thu, Mar 23, 2017 at 11:34 AM Massimilian Mattetti <MA...@il.ibm.com>
> wrote:
> I wasn't aware of such constrain. So I can just increase the number of
> tablets per server and it will perform more major compactions.
> Thanks,
> Max
>
>
>
>
> From: Michael Wall <mj...@gmail.com>
> To: user@accumulo.apache.org
> Date: 23/03/2017 17:09
> Subject: Re: tserver.compaction.*.concurrent.max behavior in Accumulo
> 1.8.1
> ________________________________
>
>
>
> Max,
>
> So your max major compactions will be 3 per tablet server. Accumulo will
> not run 2 majors on the same tablet concurrently.
>
> Mike
>
> On Thu, Mar 23, 2017 at 10:37 AM Massimilian Mattetti <MA...@il.ibm.com>
> wrote:
> One table, 9 pre-split tablets, 3 tablets per server and the data is uniform
> distributed among each tablet.
> Max
>
>
>
>
> From: Michael Wall <mj...@gmail.com>
> To: user@accumulo.apache.org
> Date: 23/03/2017 16:28
> Subject: Re: tserver.compaction.*.concurrent.max behavior in Accumulo
> 1.8.1
> ________________________________
>
>
>
> Max,
>
> On you 3 node cluster, how many tables are you ingesting into? How many
> tablets are in each table? Are the tablets equally spread amongst the 3
> tablet servers?
>
> Mike
>
> On Thu, Mar 23, 2017 at 10:13 AM Massimilian Mattetti <MA...@il.ibm.com>
> wrote:
> With the configuration I presented before the concurrent major compactions
> are never more than 3 per tablet server while the minor are under the 4 per
> node. Can one of the other configurations be the cause of this behavior?
>
> Regards,
> Max
>
>
>
> From: Dave Marion <dl...@comcast.net>
> To: user@accumulo.apache.org, Massimilian Mattetti/Haifa/IBM@IBMIL
> Date: 23/03/2017 14:55
> Subject: Re: tserver.compaction.*.concurrent.max behavior in Accumulo
> 1.8.1
> ________________________________
>
>
>
> Can you explain more what you mean by "My problem is that both the minor and
> major compactions do not overcome their default max values?" I have done
> some testing with 1.8.1 and specifically modifying
> tserver.compaction.major.concurrent.max to a higher number and I have seen
> it take effect.
>
> On March 23, 2017 at 7:54 AM Massimilian Mattetti <MA...@il.ibm.com>
> wrote:
>
> Hi All,
>
> I am running some heavy ingestion process on a 3 nodes cluster of Accumulo
> 1.8.1, using the following configuration:
>
> table.compaction.minor.logs.threshold=10
> table.durability=flush
> table.file.max=30
>
> tserver.wal.blocksize=2G
> tserver.walog.max.size=4G
> tserver.mutation.queue.max=2M
> tserver.memory.maps.max=4G
> tserver.compaction.minor.concurrent.max=50
> tserver.compaction.major.concurrent.max=8
>
> My problem is that both the minor and major compactions do not overcome
> their default max values. I checked the config from the shell and it looks
> fine to me:
>
> default | tserver.compaction.major.concurrent.max ................ | 3
> system | @override
> ........................................... | 8
>
> default | tserver.compaction.minor.concurrent.max ............... | 4
> system | @override
> ........................................... | 50
>
> Is something changed from 1.8.0? I haven't seen such behavior with the
> previous version.
> Thanks.
>
> Regards,
> Max
>
>
>
>
>
>
>
>
>
>
Re: tserver.compaction.*.concurrent.max behavior in Accumulo 1.8.1
Posted by Massimilian Mattetti <MA...@il.ibm.com>.
I just tested the ingestion with 10 tablets per Server and now the system
achieve up to 24 concurrent major compactions.
I have another issue with the tserver.memory.maps.max property. I am play
with the size of the native map to track how its size affect the ingestion
performance. I set the values of table.compaction.minor.logs.threshold and
tserver.walog.max.size big enough so I can increase the map size up to
40GB without problems. At the beginning changing the size of the map
(using the shell) didn't bring any benefit to the ingestion process.
Monitoring the servers I noticed that their RAM consumption was constant.
Eventually I tried to restart the tablet servers after changing the
tserver.memory.maps.max and the RAM usage on each node increased as I
expected. From the documentation I know that any change to the
tserver.memory.maps.max should take effect without restarting the tablet
servers, is that always true? (tserver.memory.maps.native.enabled has
always been true and from the logs I see that the shared library has been
correctly loaded).
Thanks!
Best Regards,
Max
From: Michael Wall <mj...@gmail.com>
To: user@accumulo.apache.org
Date: 23/03/2017 17:49
Subject: Re: tserver.compaction.*.concurrent.max behavior in
Accumulo 1.8.1
Yes, until you hit another constraint like Marc and Dave were asking
about.
Mike
On Thu, Mar 23, 2017 at 11:34 AM Massimilian Mattetti <MASSIMIL@il.ibm.com
> wrote:
I wasn't aware of such constrain. So I can just increase the number of
tablets per server and it will perform more major compactions.
Thanks,
Max
From: Michael Wall <mj...@gmail.com>
To: user@accumulo.apache.org
Date: 23/03/2017 17:09
Subject: Re: tserver.compaction.*.concurrent.max behavior in
Accumulo 1.8.1
Max,
So your max major compactions will be 3 per tablet server. Accumulo will
not run 2 majors on the same tablet concurrently.
Mike
On Thu, Mar 23, 2017 at 10:37 AM Massimilian Mattetti <MASSIMIL@il.ibm.com
> wrote:
One table, 9 pre-split tablets, 3 tablets per server and the data is
uniform distributed among each tablet.
Max
From: Michael Wall <mj...@gmail.com>
To: user@accumulo.apache.org
Date: 23/03/2017 16:28
Subject: Re: tserver.compaction.*.concurrent.max behavior in
Accumulo 1.8.1
Max,
On you 3 node cluster, how many tables are you ingesting into? How many
tablets are in each table? Are the tablets equally spread amongst the 3
tablet servers?
Mike
On Thu, Mar 23, 2017 at 10:13 AM Massimilian Mattetti <MASSIMIL@il.ibm.com
> wrote:
With the configuration I presented before the concurrent major compactions
are never more than 3 per tablet server while the minor are under the 4
per node. Can one of the other configurations be the cause of this
behavior?
Regards,
Max
From: Dave Marion <dl...@comcast.net>
To: user@accumulo.apache.org, Massimilian Mattetti/Haifa/IBM@IBMIL
Date: 23/03/2017 14:55
Subject: Re: tserver.compaction.*.concurrent.max behavior in
Accumulo 1.8.1
Can you explain more what you mean by "My problem is that both the minor
and major compactions do not overcome their default max values?" I have
done some testing with 1.8.1 and specifically modifying
tserver.compaction.major.concurrent.max to a higher number and I have seen
it take effect.
On March 23, 2017 at 7:54 AM Massimilian Mattetti <MA...@il.ibm.com>
wrote:
Hi All,
I am running some heavy ingestion process on a 3 nodes cluster of Accumulo
1.8.1, using the following configuration:
table.compaction.minor.logs.threshold=10
table.durability=flush
table.file.max=30
tserver.wal.blocksize=2G
tserver.walog.max.size=4G
tserver.mutation.queue.max=2M
tserver.memory.maps.max=4G
tserver.compaction.minor.concurrent.max=50
tserver.compaction.major.concurrent.max=8
My problem is that both the minor and major compactions do not overcome
their default max values. I checked the config from the shell and it looks
fine to me:
default | tserver.compaction.major.concurrent.max ................ | 3
system | @override
........................................... | 8
default | tserver.compaction.minor.concurrent.max ............... | 4
system | @override
........................................... | 50
Is something changed from 1.8.0? I haven't seen such behavior with the
previous version.
Thanks.
Regards,
Max
Re: tserver.compaction.*.concurrent.max behavior in Accumulo 1.8.1
Posted by Michael Wall <mj...@gmail.com>.
Yes, until you hit another constraint like Marc and Dave were asking about.
Mike
On Thu, Mar 23, 2017 at 11:34 AM Massimilian Mattetti <MA...@il.ibm.com>
wrote:
> I wasn't aware of such constrain. So I can just increase the number of
> tablets per server and it will perform more major compactions.
> Thanks,
> Max
>
>
>
>
> From: Michael Wall <mj...@gmail.com>
> To: user@accumulo.apache.org
> Date: 23/03/2017 17:09
> Subject: Re: tserver.compaction.*.concurrent.max behavior in
> Accumulo 1.8.1
> ------------------------------
>
>
>
> Max,
>
> So your max major compactions will be 3 per tablet server. Accumulo will
> not run 2 majors on the same tablet concurrently.
>
> Mike
>
> On Thu, Mar 23, 2017 at 10:37 AM Massimilian Mattetti <
> *MASSIMIL@il.ibm.com* <MA...@il.ibm.com>> wrote:
> One table, 9 pre-split tablets, 3 tablets per server and the data is
> uniform distributed among each tablet.
> Max
>
>
>
>
> From: Michael Wall <*mjwall@gmail.com* <mj...@gmail.com>>
> To: *user@accumulo.apache.org* <us...@accumulo.apache.org>
> Date: 23/03/2017 16:28
> Subject: Re: tserver.compaction.*.concurrent.max behavior in
> Accumulo 1.8.1
> ------------------------------
>
>
>
> Max,
>
> On you 3 node cluster, how many tables are you ingesting into? How many
> tablets are in each table? Are the tablets equally spread amongst the 3
> tablet servers?
>
> Mike
>
> On Thu, Mar 23, 2017 at 10:13 AM Massimilian Mattetti <
> *MASSIMIL@il.ibm.com* <MA...@il.ibm.com>> wrote:
> With the configuration I presented before the concurrent major compactions
> are never more than 3 per tablet server while the minor are under the 4 per
> node. Can one of the other configurations be the cause of this behavior?
>
> Regards,
> Max
>
>
>
> From: Dave Marion <*dlmarion@comcast.net* <dl...@comcast.net>>
> To: *user@accumulo.apache.org* <us...@accumulo.apache.org>,
> Massimilian Mattetti/Haifa/IBM@IBMIL
> Date: 23/03/2017 14:55
> Subject: Re: tserver.compaction.*.concurrent.max behavior in
> Accumulo 1.8.1
> ------------------------------
>
>
>
> Can you explain more what you mean by "My problem is that both the minor
> and major compactions do not overcome their default max values?" I have
> done some testing with 1.8.1 and specifically modifying
> tserver.compaction.major.concurrent.max to a higher number and I have seen
> it take effect.
>
> On March 23, 2017 at 7:54 AM Massimilian Mattetti <*MASSIMIL@il.ibm.com*
> <MA...@il.ibm.com>> wrote:
>
> Hi All,
>
> I am running some heavy ingestion process on a 3 nodes cluster of Accumulo
> 1.8.1, using the following configuration:
>
> table.compaction.minor.logs.threshold=10
> table.durability=flush
> table.file.max=30
>
> tserver.wal.blocksize=2G
> tserver.walog.max.size=4G
> tserver.mutation.queue.max=2M
> tserver.memory.maps.max=4G
> tserver.compaction.minor.concurrent.max=50
> tserver.compaction.major.concurrent.max=8
>
> My problem is that both the minor and major compactions do not overcome
> their default max values. I checked the config from the shell and it looks
> fine to me:
>
> default | tserver.compaction.major.concurrent.max ................ | 3
> system | @override
> ........................................... | 8
>
> default | tserver.compaction.minor.concurrent.max ............... | 4
> system | @override
> ........................................... | 50
>
> Is something changed from 1.8.0? I haven't seen such behavior with the
> previous version.
> Thanks.
>
> Regards,
> Max
>
>
>
>
>
>
>
>
>
Re: tserver.compaction.*.concurrent.max behavior in Accumulo 1.8.1
Posted by Massimilian Mattetti <MA...@il.ibm.com>.
I wasn't aware of such constrain. So I can just increase the number of
tablets per server and it will perform more major compactions.
Thanks,
Max
From: Michael Wall <mj...@gmail.com>
To: user@accumulo.apache.org
Date: 23/03/2017 17:09
Subject: Re: tserver.compaction.*.concurrent.max behavior in
Accumulo 1.8.1
Max,
So your max major compactions will be 3 per tablet server. Accumulo will
not run 2 majors on the same tablet concurrently.
Mike
On Thu, Mar 23, 2017 at 10:37 AM Massimilian Mattetti <MASSIMIL@il.ibm.com
> wrote:
One table, 9 pre-split tablets, 3 tablets per server and the data is
uniform distributed among each tablet.
Max
From: Michael Wall <mj...@gmail.com>
To: user@accumulo.apache.org
Date: 23/03/2017 16:28
Subject: Re: tserver.compaction.*.concurrent.max behavior in
Accumulo 1.8.1
Max,
On you 3 node cluster, how many tables are you ingesting into? How many
tablets are in each table? Are the tablets equally spread amongst the 3
tablet servers?
Mike
On Thu, Mar 23, 2017 at 10:13 AM Massimilian Mattetti <MASSIMIL@il.ibm.com
> wrote:
With the configuration I presented before the concurrent major compactions
are never more than 3 per tablet server while the minor are under the 4
per node. Can one of the other configurations be the cause of this
behavior?
Regards,
Max
From: Dave Marion <dl...@comcast.net>
To: user@accumulo.apache.org, Massimilian Mattetti/Haifa/IBM@IBMIL
Date: 23/03/2017 14:55
Subject: Re: tserver.compaction.*.concurrent.max behavior in
Accumulo 1.8.1
Can you explain more what you mean by "My problem is that both the minor
and major compactions do not overcome their default max values?" I have
done some testing with 1.8.1 and specifically modifying
tserver.compaction.major.concurrent.max to a higher number and I have seen
it take effect.
On March 23, 2017 at 7:54 AM Massimilian Mattetti <MA...@il.ibm.com>
wrote:
Hi All,
I am running some heavy ingestion process on a 3 nodes cluster of Accumulo
1.8.1, using the following configuration:
table.compaction.minor.logs.threshold=10
table.durability=flush
table.file.max=30
tserver.wal.blocksize=2G
tserver.walog.max.size=4G
tserver.mutation.queue.max=2M
tserver.memory.maps.max=4G
tserver.compaction.minor.concurrent.max=50
tserver.compaction.major.concurrent.max=8
My problem is that both the minor and major compactions do not overcome
their default max values. I checked the config from the shell and it looks
fine to me:
default | tserver.compaction.major.concurrent.max ................ | 3
system | @override
........................................... | 8
default | tserver.compaction.minor.concurrent.max ............... | 4
system | @override
........................................... | 50
Is something changed from 1.8.0? I haven't seen such behavior with the
previous version.
Thanks.
Regards,
Max
Re: tserver.compaction.*.concurrent.max behavior in Accumulo 1.8.1
Posted by Michael Wall <mj...@gmail.com>.
Max,
So your max major compactions will be 3 per tablet server. Accumulo will
not run 2 majors on the same tablet concurrently.
Mike
On Thu, Mar 23, 2017 at 10:37 AM Massimilian Mattetti <MA...@il.ibm.com>
wrote:
> One table, 9 pre-split tablets, 3 tablets per server and the data is
> uniform distributed among each tablet.
> Max
>
>
>
>
> From: Michael Wall <mj...@gmail.com>
> To: user@accumulo.apache.org
> Date: 23/03/2017 16:28
> Subject: Re: tserver.compaction.*.concurrent.max behavior in
> Accumulo 1.8.1
> ------------------------------
>
>
>
> Max,
>
> On you 3 node cluster, how many tables are you ingesting into? How many
> tablets are in each table? Are the tablets equally spread amongst the 3
> tablet servers?
>
> Mike
>
> On Thu, Mar 23, 2017 at 10:13 AM Massimilian Mattetti <
> *MASSIMIL@il.ibm.com* <MA...@il.ibm.com>> wrote:
> With the configuration I presented before the concurrent major compactions
> are never more than 3 per tablet server while the minor are under the 4 per
> node. Can one of the other configurations be the cause of this behavior?
>
> Regards,
> Max
>
>
>
> From: Dave Marion <*dlmarion@comcast.net* <dl...@comcast.net>>
> To: *user@accumulo.apache.org* <us...@accumulo.apache.org>,
> Massimilian Mattetti/Haifa/IBM@IBMIL
> Date: 23/03/2017 14:55
> Subject: Re: tserver.compaction.*.concurrent.max behavior in
> Accumulo 1.8.1
> ------------------------------
>
>
>
> Can you explain more what you mean by "My problem is that both the minor
> and major compactions do not overcome their default max values?" I have
> done some testing with 1.8.1 and specifically modifying
> tserver.compaction.major.concurrent.max to a higher number and I have seen
> it take effect.
>
> On March 23, 2017 at 7:54 AM Massimilian Mattetti <*MASSIMIL@il.ibm.com*
> <MA...@il.ibm.com>> wrote:
>
> Hi All,
>
> I am running some heavy ingestion process on a 3 nodes cluster of Accumulo
> 1.8.1, using the following configuration:
>
> table.compaction.minor.logs.threshold=10
> table.durability=flush
> table.file.max=30
>
> tserver.wal.blocksize=2G
> tserver.walog.max.size=4G
> tserver.mutation.queue.max=2M
> tserver.memory.maps.max=4G
> tserver.compaction.minor.concurrent.max=50
> tserver.compaction.major.concurrent.max=8
>
> My problem is that both the minor and major compactions do not overcome
> their default max values. I checked the config from the shell and it looks
> fine to me:
>
> default | tserver.compaction.major.concurrent.max ................ | 3
> system | @override
> ........................................... | 8
>
> default | tserver.compaction.minor.concurrent.max ............... | 4
> system | @override
> ........................................... | 50
>
> Is something changed from 1.8.0? I haven't seen such behavior with the
> previous version.
> Thanks.
>
> Regards,
> Max
>
>
>
>
>
>
>
Re: tserver.compaction.*.concurrent.max behavior in Accumulo 1.8.1
Posted by Massimilian Mattetti <MA...@il.ibm.com>.
One table, 9 pre-split tablets, 3 tablets per server and the data is
uniform distributed among each tablet.
Max
From: Michael Wall <mj...@gmail.com>
To: user@accumulo.apache.org
Date: 23/03/2017 16:28
Subject: Re: tserver.compaction.*.concurrent.max behavior in
Accumulo 1.8.1
Max,
On you 3 node cluster, how many tables are you ingesting into? How many
tablets are in each table? Are the tablets equally spread amongst the 3
tablet servers?
Mike
On Thu, Mar 23, 2017 at 10:13 AM Massimilian Mattetti <MASSIMIL@il.ibm.com
> wrote:
With the configuration I presented before the concurrent major compactions
are never more than 3 per tablet server while the minor are under the 4
per node. Can one of the other configurations be the cause of this
behavior?
Regards,
Max
From: Dave Marion <dl...@comcast.net>
To: user@accumulo.apache.org, Massimilian Mattetti/Haifa/IBM@IBMIL
Date: 23/03/2017 14:55
Subject: Re: tserver.compaction.*.concurrent.max behavior in
Accumulo 1.8.1
Can you explain more what you mean by "My problem is that both the minor
and major compactions do not overcome their default max values?" I have
done some testing with 1.8.1 and specifically modifying
tserver.compaction.major.concurrent.max to a higher number and I have seen
it take effect.
On March 23, 2017 at 7:54 AM Massimilian Mattetti <MA...@il.ibm.com>
wrote:
Hi All,
I am running some heavy ingestion process on a 3 nodes cluster of Accumulo
1.8.1, using the following configuration:
table.compaction.minor.logs.threshold=10
table.durability=flush
table.file.max=30
tserver.wal.blocksize=2G
tserver.walog.max.size=4G
tserver.mutation.queue.max=2M
tserver.memory.maps.max=4G
tserver.compaction.minor.concurrent.max=50
tserver.compaction.major.concurrent.max=8
My problem is that both the minor and major compactions do not overcome
their default max values. I checked the config from the shell and it looks
fine to me:
default | tserver.compaction.major.concurrent.max ................ | 3
system | @override
........................................... | 8
default | tserver.compaction.minor.concurrent.max ............... | 4
system | @override
........................................... | 50
Is something changed from 1.8.0? I haven't seen such behavior with the
previous version.
Thanks.
Regards,
Max
Re: tserver.compaction.*.concurrent.max behavior in Accumulo 1.8.1
Posted by Michael Wall <mj...@gmail.com>.
Max,
On you 3 node cluster, how many tables are you ingesting into? How many
tablets are in each table? Are the tablets equally spread amongst the 3
tablet servers?
Mike
On Thu, Mar 23, 2017 at 10:13 AM Massimilian Mattetti <MA...@il.ibm.com>
wrote:
> With the configuration I presented before the concurrent major compactions
> are never more than 3 per tablet server while the minor are under the 4 per
> node. Can one of the other configurations be the cause of this behavior?
>
> Regards,
> Max
>
>
>
> From: Dave Marion <dl...@comcast.net>
> To: user@accumulo.apache.org, Massimilian Mattetti/Haifa/IBM@IBMIL
> Date: 23/03/2017 14:55
> Subject: Re: tserver.compaction.*.concurrent.max behavior in
> Accumulo 1.8.1
> ------------------------------
>
>
>
> Can you explain more what you mean by "My problem is that both the minor
> and major compactions do not overcome their default max values?" I have
> done some testing with 1.8.1 and specifically modifying
> tserver.compaction.major.concurrent.max to a higher number and I have seen
> it take effect.
>
> On March 23, 2017 at 7:54 AM Massimilian Mattetti <MA...@il.ibm.com>
> wrote:
>
> Hi All,
>
> I am running some heavy ingestion process on a 3 nodes cluster of Accumulo
> 1.8.1, using the following configuration:
>
> table.compaction.minor.logs.threshold=10
> table.durability=flush
> table.file.max=30
>
> tserver.wal.blocksize=2G
> tserver.walog.max.size=4G
> tserver.mutation.queue.max=2M
> tserver.memory.maps.max=4G
> tserver.compaction.minor.concurrent.max=50
> tserver.compaction.major.concurrent.max=8
>
> My problem is that both the minor and major compactions do not overcome
> their default max values. I checked the config from the shell and it looks
> fine to me:
>
> default | tserver.compaction.major.concurrent.max ................ | 3
> system | @override
> ........................................... | 8
>
> default | tserver.compaction.minor.concurrent.max ............... | 4
> system | @override
> ........................................... | 50
>
> Is something changed from 1.8.0? I haven't seen such behavior with the
> previous version.
> Thanks.
>
> Regards,
> Max
>
>
>
>
>
Re: tserver.compaction.*.concurrent.max behavior in Accumulo 1.8.1
Posted by Dave Marion <dl...@comcast.net>.
The number of concurrent major compactions per tserver should be directly related to the number of tablets hosted on each tablet server. During normal operations, major compactions for a tablet should be initiated by the applying the value of `table.compaction.major.ratio` to `table.file.max` and the actual number of files. Additionally, the major compaction process in the tablet server is controlled by the `tserver.compaction.major.*` properties. If you want to test whether or not your `tserver.compaction.major.concurrent.max` property is being honored, go into the shell and compact the table manually using the compact command. This will perform a full compaction and you should see 8 concurrent MajC's occurring on each tserver. If this is the case, but you are not seeing 8 concurrent MajC's at ingest time, then I would think that you are not pushing it hard enough.
I am less familiar with the minor compaction initiation process, but in general if you are not seeing hold times, push harder. You could also test your `tserver.compaction.minor.concurrent.max` setting by performing a flush command on the table from the shell.
> On March 23, 2017 at 10:13 AM Massimilian Mattetti <MA...@il.ibm.com> wrote:
>
> With the configuration I presented before the concurrent major compactions are never more than 3 per tablet server while the minor are under the 4 per node. Can one of the other configurations be the cause of this behavior?
>
> Regards,
> Max
>
>
>
> From: Dave Marion <dl...@comcast.net>
> To: user@accumulo.apache.org, Massimilian Mattetti/Haifa/IBM@IBMIL
> Date: 23/03/2017 14:55
> Subject: Re: tserver.compaction.*.concurrent.max behavior in Accumulo 1.8.1
>
> ---------------------------------------------
>
>
>
> Can you explain more what you mean by "My problem is that both the minor and major compactions do not overcome their default max values?" I have done some testing with 1.8.1 and specifically modifying tserver.compaction.major.concurrent.max to a higher number and I have seen it take effect.
>
> On March 23, 2017 at 7:54 AM Massimilian Mattetti <MA...@il.ibm.com> wrote:
>
> Hi All,
>
> I am running some heavy ingestion process on a 3 nodes cluster of Accumulo 1.8.1, using the following configuration:
>
> table.compaction.minor.logs.threshold=10
> table.durability=flush
> table.file.max=30
>
> tserver.wal.blocksize=2G
> tserver.walog.max.size=4G
> tserver.mutation.queue.max=2M
> tserver.memory.maps.max=4G
> tserver.compaction.minor.concurrent.max=50
> tserver.compaction.major.concurrent.max=8
>
> My problem is that both the minor and major compactions do not overcome their default max values. I checked the config from the shell and it looks fine to me:
>
> default | tserver.compaction.major.concurrent.max ................ | 3
> system | @override ........................................... | 8
>
> default | tserver.compaction.minor.concurrent.max ............... | 4
> system | @override ........................................... | 50
>
> Is something changed from 1.8.0? I haven't seen such behavior with the previous version.
> Thanks.
>
> Regards,
> Max
>
>
>
>
>
Re: tserver.compaction.*.concurrent.max behavior in Accumulo 1.8.1
Posted by Massimilian Mattetti <MA...@il.ibm.com>.
With the configuration I presented before the concurrent major compactions
are never more than 3 per tablet server while the minor are under the 4
per node. Can one of the other configurations be the cause of this
behavior?
Regards,
Max
From: Dave Marion <dl...@comcast.net>
To: user@accumulo.apache.org, Massimilian Mattetti/Haifa/IBM@IBMIL
Date: 23/03/2017 14:55
Subject: Re: tserver.compaction.*.concurrent.max behavior in
Accumulo 1.8.1
Can you explain more what you mean by "My problem is that both the minor
and major compactions do not overcome their default max values?" I have
done some testing with 1.8.1 and specifically modifying
tserver.compaction.major.concurrent.max to a higher number and I have seen
it take effect.
On March 23, 2017 at 7:54 AM Massimilian Mattetti <MA...@il.ibm.com>
wrote:
Hi All,
I am running some heavy ingestion process on a 3 nodes cluster of Accumulo
1.8.1, using the following configuration:
table.compaction.minor.logs.threshold=10
table.durability=flush
table.file.max=30
tserver.wal.blocksize=2G
tserver.walog.max.size=4G
tserver.mutation.queue.max=2M
tserver.memory.maps.max=4G
tserver.compaction.minor.concurrent.max=50
tserver.compaction.major.concurrent.max=8
My problem is that both the minor and major compactions do not overcome
their default max values. I checked the config from the shell and it looks
fine to me:
default | tserver.compaction.major.concurrent.max ................ | 3
system | @override
........................................... | 8
default | tserver.compaction.minor.concurrent.max ............... | 4
system | @override
........................................... | 50
Is something changed from 1.8.0? I haven't seen such behavior with the
previous version.
Thanks.
Regards,
Max
Re: tserver.compaction.*.concurrent.max behavior in Accumulo 1.8.1
Posted by Dave Marion <dl...@comcast.net>.
Can you explain more what you mean by "My problem is that both the minor and major compactions do not overcome their default max values?" I have done some testing with 1.8.1 and specifically modifying tserver.compaction.major.concurrent.max to a higher number and I have seen it take effect.
> On March 23, 2017 at 7:54 AM Massimilian Mattetti <MA...@il.ibm.com> wrote:
>
> Hi All,
>
> I am running some heavy ingestion process on a 3 nodes cluster of Accumulo 1.8.1, using the following configuration:
>
> table.compaction.minor.logs.threshold=10
> table.durability=flush
> table.file.max=30
>
> tserver.wal.blocksize=2G
> tserver.walog.max.size=4G
> tserver.mutation.queue.max=2M
> tserver.memory.maps.max=4G
> tserver.compaction.minor.concurrent.max=50
> tserver.compaction.major.concurrent.max=8
>
> My problem is that both the minor and major compactions do not overcome their default max values. I checked the config from the shell and it looks fine to me:
>
> default | tserver.compaction.major.concurrent.max ................ | 3
> system | @override ........................................... | 8
>
> default | tserver.compaction.minor.concurrent.max ............... | 4
> system | @override ........................................... | 50
>
> Is something changed from 1.8.0? I haven't seen such behavior with the previous version.
> Thanks.
>
> Regards,
> Max
>