Advice on increasing ingest rate

[Mike Hugo <mi...@piragua.com>]

Hello,

We have an ingest process that operates via Map Reduce, processing a large
set of XML files and  inserting mutations based on that data into a set of
tables.

On a 5 node cluster (each node has 64G ram, 20 cores, and ~600GB SSD) I get
400k inserts per second with 20 mapper tasks running concurrently.
 Increasing the number of concurrent mapper tasks to 40 doesn't have any
effect (besides causing a little more backup in compactions).

I've increased the table.compaction.major.ratio and increased the number of
concurrent allowed compactions for both min and max compaction but each of
those only had negligible impact on ingest rates.

Any advice on other settings I can tweak to get things to move more
quickly?  Or is 400k/second a reasonable ingest rate?  Are we at a point
where we should consider generating r files like the bulk ingest example?

Thanks in advance for any advice.

Mike

Re: Advice on increasing ingest rate

[Mike Hugo <mi...@piragua.com>]

Currently using the default AccumuloOutputFormat settings which I believe
is 2 threads


On Tue, Apr 8, 2014 at 5:36 PM, <dl...@comcast.net> wrote:

>
>
> How many threads are you using in the AccumuloOutputFormat? What is your
> latency set to?
>
>
>
> *From:* Adam Fuchs [mailto:afuchs@apache.org]
> *Sent:* Tuesday, April 08, 2014 5:36 PM
> *To:* user@accumulo.apache.org
> *Subject:* Re: Advice on increasing ingest rate
>
>
>
> MIke,
>
>
>
> What version of Accumulo are you using, how many tablets do you have, and
> how many threads are you using for minor and major compaction pools? Also,
> how big are the keys and values that you are using?
>
>
>
> Here are a few settings that may help you:
>
> 1. WAL replication factor (tserver.wal.replication). This defaults to 3
> replicas (the HDFS default), but if you set it to 2 it will give you a
> performance boost without a huge hit to reliability.
>
> 2. Ingest buffer size (tserver.memory.maps.max), also known as the
> in-memory map size. Increasing this generally improves the efficiency of
> minor compactions and reduces the number of major compactions that will be
> required down the line. 4-8 GB is not unreasonable.
>
> 3. Make sure your WAL settings are such that the size of a log
> (tserver.walog.max.size) multiplied by the number of active logs
> (table.compaction.minor.logs.threshold) is greater than the in-memory map
> size. You probably want to accomplish this by bumping up the number of
> active logs.
>
> 4. Increase the buffer size on the BatchWriter that the clients use. This
> can be done with the setBatchWriterOptions method on the
> AccumuloOutputFormat.
>
>
>
> Cheers,
>
> Adam
>
>
>
>
>
> On Tue, Apr 8, 2014 at 4:47 PM, Mike Hugo <mi...@piragua.com> wrote:
>
> Hello,
>
>
>
> We have an ingest process that operates via Map Reduce, processing a large
> set of XML files and  inserting mutations based on that data into a set of
> tables.
>
>
>
> On a 5 node cluster (each node has 64G ram, 20 cores, and ~600GB SSD) I
> get 400k inserts per second with 20 mapper tasks running concurrently.
>  Increasing the number of concurrent mapper tasks to 40 doesn't have any
> effect (besides causing a little more backup in compactions).
>
>
>
> I've increased the table.compaction.major.ratio and increased the number
> of concurrent allowed compactions for both min and max compaction but each
> of those only had negligible impact on ingest rates.
>
>
>
> Any advice on other settings I can tweak to get things to move more
> quickly?  Or is 400k/second a reasonable ingest rate?  Are we at a point
> where we should consider generating r files like the bulk ingest example?
>
>
>
> Thanks in advance for any advice.
>
>
>
> Mike
>
>
>

RE: Advice on increasing ingest rate

[dl...@comcast.net]

 

How many threads are you using in the AccumuloOutputFormat? What is your
latency set to?

 

From: Adam Fuchs [mailto:afuchs@apache.org] 
Sent: Tuesday, April 08, 2014 5:36 PM
To: user@accumulo.apache.org
Subject: Re: Advice on increasing ingest rate

 

MIke,

 

What version of Accumulo are you using, how many tablets do you have, and
how many threads are you using for minor and major compaction pools? Also,
how big are the keys and values that you are using?

 

Here are a few settings that may help you:

1. WAL replication factor (tserver.wal.replication). This defaults to 3
replicas (the HDFS default), but if you set it to 2 it will give you a
performance boost without a huge hit to reliability.

2. Ingest buffer size (tserver.memory.maps.max), also known as the in-memory
map size. Increasing this generally improves the efficiency of minor
compactions and reduces the number of major compactions that will be
required down the line. 4-8 GB is not unreasonable.

3. Make sure your WAL settings are such that the size of a log
(tserver.walog.max.size) multiplied by the number of active logs
(table.compaction.minor.logs.threshold) is greater than the in-memory map
size. You probably want to accomplish this by bumping up the number of
active logs.

4. Increase the buffer size on the BatchWriter that the clients use. This
can be done with the setBatchWriterOptions method on the
AccumuloOutputFormat.

 

Cheers,

Adam

 

 

On Tue, Apr 8, 2014 at 4:47 PM, Mike Hugo <mi...@piragua.com> wrote:

Hello,

 

We have an ingest process that operates via Map Reduce, processing a large
set of XML files and  inserting mutations based on that data into a set of
tables.

 

On a 5 node cluster (each node has 64G ram, 20 cores, and ~600GB SSD) I get
400k inserts per second with 20 mapper tasks running concurrently.
Increasing the number of concurrent mapper tasks to 40 doesn't have any
effect (besides causing a little more backup in compactions).

 

I've increased the table.compaction.major.ratio and increased the number of
concurrent allowed compactions for both min and max compaction but each of
those only had negligible impact on ingest rates.

 

Any advice on other settings I can tweak to get things to move more quickly?
Or is 400k/second a reasonable ingest rate?  Are we at a point where we
should consider generating r files like the bulk ingest example?

 

Thanks in advance for any advice.

 

Mike

 

Re: Advice on increasing ingest rate

[Mike Hugo <mi...@piragua.com>]

I took a screen shot of the monitor page and attached it.  Currently we're
seeing ingest of about 350k entries per second and 60-90 MB/s


On Wed, Apr 9, 2014 at 5:33 PM, Adam Fuchs <af...@apache.org> wrote:

> If the average is around 1k per k/v entry, then I would say that 400MB/s
> is very good performance for incremental/streaming ingest into Accumulo on
> that cluster. However, I suspect that your entries are probably not that
> big on average. Do you have a measurement for MB/s ingest?
>
> Adam
> On Apr 9, 2014 4:42 PM, "Mike Hugo" <mi...@piragua.com> wrote:
>
>>
>>
>>
>> On Tue, Apr 8, 2014 at 4:35 PM, Adam Fuchs <af...@apache.org> wrote:
>>
>>> MIke,
>>>
>>> What version of Accumulo are you using, how many tablets do you have,
>>> and how many threads are you using for minor and major compaction pools?
>>> Also, how big are the keys and values that you are using?
>>>
>>>
>> 1.4.5
>> 6 threads each for min and major compaction
>> Keys and values are not that large, there may be a few outliers but I
>> would estimate that most of them are < 1k
>>
>>
>>
>>> Here are a few settings that may help you:
>>> 1. WAL replication factor (tserver.wal.replication). This defaults to 3
>>> replicas (the HDFS default), but if you set it to 2 it will give you a
>>> performance boost without a huge hit to reliability.
>>> 2. Ingest buffer size (tserver.memory.maps.max), also known as the
>>> in-memory map size. Increasing this generally improves the efficiency of
>>> minor compactions and reduces the number of major compactions that will be
>>> required down the line. 4-8 GB is not unreasonable.
>>> 3. Make sure your WAL settings are such that the size of a log
>>> (tserver.walog.max.size) multiplied by the number of active logs
>>> (table.compaction.minor.logs.threshold) is greater than the in-memory map
>>> size. You probably want to accomplish this by bumping up the number of
>>> active logs.
>>> 4. Increase the buffer size on the BatchWriter that the clients use.
>>> This can be done with the setBatchWriterOptions method on the
>>> AccumuloOutputFormat.
>>>
>>>
>> Thanks for the tips, I try these out
>>
>>
>>> Cheers,
>>> Adam
>>>
>>>
>>>
>>> On Tue, Apr 8, 2014 at 4:47 PM, Mike Hugo <mi...@piragua.com> wrote:
>>>
>>>> Hello,
>>>>
>>>> We have an ingest process that operates via Map Reduce, processing a
>>>> large set of XML files and  inserting mutations based on that data into a
>>>> set of tables.
>>>>
>>>> On a 5 node cluster (each node has 64G ram, 20 cores, and ~600GB SSD) I
>>>> get 400k inserts per second with 20 mapper tasks running concurrently.
>>>>  Increasing the number of concurrent mapper tasks to 40 doesn't have any
>>>> effect (besides causing a little more backup in compactions).
>>>>
>>>> I've increased the table.compaction.major.ratio and increased the
>>>> number of concurrent allowed compactions for both min and max compaction
>>>> but each of those only had negligible impact on ingest rates.
>>>>
>>>> Any advice on other settings I can tweak to get things to move more
>>>> quickly?  Or is 400k/second a reasonable ingest rate?  Are we at a point
>>>> where we should consider generating r files like the bulk ingest example?
>>>>
>>>> Thanks in advance for any advice.
>>>>
>>>> Mike
>>>>
>>>
>>>
>>

Re: Advice on increasing ingest rate

[Adam Fuchs <af...@apache.org>]

If the average is around 1k per k/v entry, then I would say that 400MB/s is
very good performance for incremental/streaming ingest into Accumulo on
that cluster. However, I suspect that your entries are probably not that
big on average. Do you have a measurement for MB/s ingest?

Adam
On Apr 9, 2014 4:42 PM, "Mike Hugo" <mi...@piragua.com> wrote:

>
>
>
> On Tue, Apr 8, 2014 at 4:35 PM, Adam Fuchs <af...@apache.org> wrote:
>
>> MIke,
>>
>> What version of Accumulo are you using, how many tablets do you have, and
>> how many threads are you using for minor and major compaction pools? Also,
>> how big are the keys and values that you are using?
>>
>>
> 1.4.5
> 6 threads each for min and major compaction
> Keys and values are not that large, there may be a few outliers but I
> would estimate that most of them are < 1k
>
>
>
>> Here are a few settings that may help you:
>> 1. WAL replication factor (tserver.wal.replication). This defaults to 3
>> replicas (the HDFS default), but if you set it to 2 it will give you a
>> performance boost without a huge hit to reliability.
>> 2. Ingest buffer size (tserver.memory.maps.max), also known as the
>> in-memory map size. Increasing this generally improves the efficiency of
>> minor compactions and reduces the number of major compactions that will be
>> required down the line. 4-8 GB is not unreasonable.
>> 3. Make sure your WAL settings are such that the size of a log
>> (tserver.walog.max.size) multiplied by the number of active logs
>> (table.compaction.minor.logs.threshold) is greater than the in-memory map
>> size. You probably want to accomplish this by bumping up the number of
>> active logs.
>> 4. Increase the buffer size on the BatchWriter that the clients use. This
>> can be done with the setBatchWriterOptions method on the
>> AccumuloOutputFormat.
>>
>>
> Thanks for the tips, I try these out
>
>
>> Cheers,
>> Adam
>>
>>
>>
>> On Tue, Apr 8, 2014 at 4:47 PM, Mike Hugo <mi...@piragua.com> wrote:
>>
>>> Hello,
>>>
>>> We have an ingest process that operates via Map Reduce, processing a
>>> large set of XML files and  inserting mutations based on that data into a
>>> set of tables.
>>>
>>> On a 5 node cluster (each node has 64G ram, 20 cores, and ~600GB SSD) I
>>> get 400k inserts per second with 20 mapper tasks running concurrently.
>>>  Increasing the number of concurrent mapper tasks to 40 doesn't have any
>>> effect (besides causing a little more backup in compactions).
>>>
>>> I've increased the table.compaction.major.ratio and increased the number
>>> of concurrent allowed compactions for both min and max compaction but each
>>> of those only had negligible impact on ingest rates.
>>>
>>> Any advice on other settings I can tweak to get things to move more
>>> quickly?  Or is 400k/second a reasonable ingest rate?  Are we at a point
>>> where we should consider generating r files like the bulk ingest example?
>>>
>>> Thanks in advance for any advice.
>>>
>>> Mike
>>>
>>
>>
>

Re: Advice on increasing ingest rate

[Mike Hugo <mi...@piragua.com>]

On Tue, Apr 8, 2014 at 4:35 PM, Adam Fuchs <af...@apache.org> wrote:

> MIke,
>
> What version of Accumulo are you using, how many tablets do you have, and
> how many threads are you using for minor and major compaction pools? Also,
> how big are the keys and values that you are using?
>
>
1.4.5
6 threads each for min and major compaction
Keys and values are not that large, there may be a few outliers but I would
estimate that most of them are < 1k



> Here are a few settings that may help you:
> 1. WAL replication factor (tserver.wal.replication). This defaults to 3
> replicas (the HDFS default), but if you set it to 2 it will give you a
> performance boost without a huge hit to reliability.
> 2. Ingest buffer size (tserver.memory.maps.max), also known as the
> in-memory map size. Increasing this generally improves the efficiency of
> minor compactions and reduces the number of major compactions that will be
> required down the line. 4-8 GB is not unreasonable.
> 3. Make sure your WAL settings are such that the size of a log
> (tserver.walog.max.size) multiplied by the number of active logs
> (table.compaction.minor.logs.threshold) is greater than the in-memory map
> size. You probably want to accomplish this by bumping up the number of
> active logs.
> 4. Increase the buffer size on the BatchWriter that the clients use. This
> can be done with the setBatchWriterOptions method on the
> AccumuloOutputFormat.
>
>
Thanks for the tips, I try these out


> Cheers,
> Adam
>
>
>
> On Tue, Apr 8, 2014 at 4:47 PM, Mike Hugo <mi...@piragua.com> wrote:
>
>> Hello,
>>
>> We have an ingest process that operates via Map Reduce, processing a
>> large set of XML files and  inserting mutations based on that data into a
>> set of tables.
>>
>> On a 5 node cluster (each node has 64G ram, 20 cores, and ~600GB SSD) I
>> get 400k inserts per second with 20 mapper tasks running concurrently.
>>  Increasing the number of concurrent mapper tasks to 40 doesn't have any
>> effect (besides causing a little more backup in compactions).
>>
>> I've increased the table.compaction.major.ratio and increased the number
>> of concurrent allowed compactions for both min and max compaction but each
>> of those only had negligible impact on ingest rates.
>>
>> Any advice on other settings I can tweak to get things to move more
>> quickly?  Or is 400k/second a reasonable ingest rate?  Are we at a point
>> where we should consider generating r files like the bulk ingest example?
>>
>> Thanks in advance for any advice.
>>
>> Mike
>>
>
>

Re: Advice on increasing ingest rate

[Mike Hugo <mi...@piragua.com>]

Sorry - my mistake I read tables not tablets.  The map reduce jobs insert
into 4 different tables, each has between 100 and 200 tablets


On Wed, Apr 9, 2014 at 4:03 PM, David Medinets <da...@gmail.com>wrote:

> Pre-split as much as possible. Accumulo will take of spreading the tablets
> across the cluster. You have five nodes but only four tablets. This means
> that one node is not getting any data during the ingest process. If you
> can't pre-split because you don't have enough variability in the row value,
> and you need a higher ingest rate, you'll need to change your row value
> structure.
>

Re: Advice on increasing ingest rate

[David Medinets <da...@gmail.com>]

Pre-split as much as possible. Accumulo will take of spreading the tablets
across the cluster. You have five nodes but only four tablets. This means
that one node is not getting any data during the ingest process. If you
can't pre-split because you don't have enough variability in the row value,
and you need a higher ingest rate, you'll need to change your row value
structure.

Re: Advice on increasing ingest rate

[Mike Hugo <mi...@piragua.com>]

On Tue, Apr 8, 2014 at 5:35 PM, David Medinets <da...@gmail.com>wrote:

> 20 cores and just one SSD? Is there a standard recommendation for a core
> to SSD ratio?
>
> Other questions:
>
> How are you sharding your data (i.e., what does your row look like)?
>

we do something kind of like the entity attribute / graph tables example
from the accumulo manual


> Are you pre-spliting the table?
>

no


> How many tablets are ingesting at the same time?
>

4


> Are you writing from the map-reduce directly to Accumulo or writing to
> rFiles first?
>

directly to accumulo


> Are the Accumulo nodes and the Hadoop nodes on the same servers?
>

yes


>  Do you see the server load spike during ingest?
>



> How much memory are you allocating to the tservers?
>

2GB


> How large are the entries on average?
>
What are the largest entries?
> Does the data skew towards large entries?
>

entries are small.  probably <1k for key/value combined in most instances


> Are you querying at the same time as ingesting?
>
> no


>
>
> On Tue, Apr 8, 2014 at 5:35 PM, Adam Fuchs <af...@apache.org> wrote:
>
>> MIke,
>>
>> What version of Accumulo are you using, how many tablets do you have, and
>> how many threads are you using for minor and major compaction pools? Also,
>> how big are the keys and values that you are using?
>>
>> Here are a few settings that may help you:
>> 1. WAL replication factor (tserver.wal.replication). This defaults to 3
>> replicas (the HDFS default), but if you set it to 2 it will give you a
>> performance boost without a huge hit to reliability.
>> 2. Ingest buffer size (tserver.memory.maps.max), also known as the
>> in-memory map size. Increasing this generally improves the efficiency of
>> minor compactions and reduces the number of major compactions that will be
>> required down the line. 4-8 GB is not unreasonable.
>> 3. Make sure your WAL settings are such that the size of a log
>> (tserver.walog.max.size) multiplied by the number of active logs
>> (table.compaction.minor.logs.threshold) is greater than the in-memory map
>> size. You probably want to accomplish this by bumping up the number of
>> active logs.
>> 4. Increase the buffer size on the BatchWriter that the clients use. This
>> can be done with the setBatchWriterOptions method on the
>> AccumuloOutputFormat.
>>
>> Cheers,
>> Adam
>>
>>
>>
>> On Tue, Apr 8, 2014 at 4:47 PM, Mike Hugo <mi...@piragua.com> wrote:
>>
>>> Hello,
>>>
>>> We have an ingest process that operates via Map Reduce, processing a
>>> large set of XML files and  inserting mutations based on that data into a
>>> set of tables.
>>>
>>> On a 5 node cluster (each node has 64G ram, 20 cores, and ~600GB SSD) I
>>> get 400k inserts per second with 20 mapper tasks running concurrently.
>>>  Increasing the number of concurrent mapper tasks to 40 doesn't have any
>>> effect (besides causing a little more backup in compactions).
>>>
>>> I've increased the table.compaction.major.ratio and increased the number
>>> of concurrent allowed compactions for both min and max compaction but each
>>> of those only had negligible impact on ingest rates.
>>>
>>> Any advice on other settings I can tweak to get things to move more
>>> quickly?  Or is 400k/second a reasonable ingest rate?  Are we at a point
>>> where we should consider generating r files like the bulk ingest example?
>>>
>>> Thanks in advance for any advice.
>>>
>>> Mike
>>>
>>
>>
>

Re: Advice on increasing ingest rate

[David Medinets <da...@gmail.com>]

20 cores and just one SSD? Is there a standard recommendation for a core to
SSD ratio?

Other questions:

How are you sharding your data (i.e., what does your row look like)?
Are you pre-spliting the table?
How many tablets are ingesting at the same time?
Are you writing from the map-reduce directly to Accumulo or writing to
rFiles first?
Are the Accumulo nodes and the Hadoop nodes on the same servers?
Do you see the server load spike during ingest?
How much memory are you allocating to the tservers?
How large are the entries on average?
What are the largest entries?
Does the data skew towards large entries?
Are you querying at the same time as ingesting?



On Tue, Apr 8, 2014 at 5:35 PM, Adam Fuchs <af...@apache.org> wrote:

> MIke,
>
> What version of Accumulo are you using, how many tablets do you have, and
> how many threads are you using for minor and major compaction pools? Also,
> how big are the keys and values that you are using?
>
> Here are a few settings that may help you:
> 1. WAL replication factor (tserver.wal.replication). This defaults to 3
> replicas (the HDFS default), but if you set it to 2 it will give you a
> performance boost without a huge hit to reliability.
> 2. Ingest buffer size (tserver.memory.maps.max), also known as the
> in-memory map size. Increasing this generally improves the efficiency of
> minor compactions and reduces the number of major compactions that will be
> required down the line. 4-8 GB is not unreasonable.
> 3. Make sure your WAL settings are such that the size of a log
> (tserver.walog.max.size) multiplied by the number of active logs
> (table.compaction.minor.logs.threshold) is greater than the in-memory map
> size. You probably want to accomplish this by bumping up the number of
> active logs.
> 4. Increase the buffer size on the BatchWriter that the clients use. This
> can be done with the setBatchWriterOptions method on the
> AccumuloOutputFormat.
>
> Cheers,
> Adam
>
>
>
> On Tue, Apr 8, 2014 at 4:47 PM, Mike Hugo <mi...@piragua.com> wrote:
>
>> Hello,
>>
>> We have an ingest process that operates via Map Reduce, processing a
>> large set of XML files and  inserting mutations based on that data into a
>> set of tables.
>>
>> On a 5 node cluster (each node has 64G ram, 20 cores, and ~600GB SSD) I
>> get 400k inserts per second with 20 mapper tasks running concurrently.
>>  Increasing the number of concurrent mapper tasks to 40 doesn't have any
>> effect (besides causing a little more backup in compactions).
>>
>> I've increased the table.compaction.major.ratio and increased the number
>> of concurrent allowed compactions for both min and max compaction but each
>> of those only had negligible impact on ingest rates.
>>
>> Any advice on other settings I can tweak to get things to move more
>> quickly?  Or is 400k/second a reasonable ingest rate?  Are we at a point
>> where we should consider generating r files like the bulk ingest example?
>>
>> Thanks in advance for any advice.
>>
>> Mike
>>
>
>

Re: Advice on increasing ingest rate

[Adam Fuchs <af...@apache.org>]

MIke,

What version of Accumulo are you using, how many tablets do you have, and
how many threads are you using for minor and major compaction pools? Also,
how big are the keys and values that you are using?

Here are a few settings that may help you:
1. WAL replication factor (tserver.wal.replication). This defaults to 3
replicas (the HDFS default), but if you set it to 2 it will give you a
performance boost without a huge hit to reliability.
2. Ingest buffer size (tserver.memory.maps.max), also known as the
in-memory map size. Increasing this generally improves the efficiency of
minor compactions and reduces the number of major compactions that will be
required down the line. 4-8 GB is not unreasonable.
3. Make sure your WAL settings are such that the size of a log
(tserver.walog.max.size) multiplied by the number of active logs
(table.compaction.minor.logs.threshold) is greater than the in-memory map
size. You probably want to accomplish this by bumping up the number of
active logs.
4. Increase the buffer size on the BatchWriter that the clients use. This
can be done with the setBatchWriterOptions method on the
AccumuloOutputFormat.

Cheers,
Adam



On Tue, Apr 8, 2014 at 4:47 PM, Mike Hugo <mi...@piragua.com> wrote:

> Hello,
>
> We have an ingest process that operates via Map Reduce, processing a large
> set of XML files and  inserting mutations based on that data into a set of
> tables.
>
> On a 5 node cluster (each node has 64G ram, 20 cores, and ~600GB SSD) I
> get 400k inserts per second with 20 mapper tasks running concurrently.
>  Increasing the number of concurrent mapper tasks to 40 doesn't have any
> effect (besides causing a little more backup in compactions).
>
> I've increased the table.compaction.major.ratio and increased the number
> of concurrent allowed compactions for both min and max compaction but each
> of those only had negligible impact on ingest rates.
>
> Any advice on other settings I can tweak to get things to move more
> quickly?  Or is 400k/second a reasonable ingest rate?  Are we at a point
> where we should consider generating r files like the bulk ingest example?
>
> Thanks in advance for any advice.
>
> Mike
>