You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@accumulo.apache.org by "Kepner, Jeremy - LLSC - MITLL" <ke...@ll.mit.edu> on 2021/06/05 22:11:35 UTC
Accumulo 1.10.1 performance
Has anyone benchmarked Accumulo 1.10.1? I have been looking into repeating the measurements we did in 2014 with Accumulo 1.5 (https://arxiv.org/abs/1406.4923) using Accumulo 1.10.1 on a bigger system with more modern hardware. Unfortunately, when I repeat the single node measurements, there is no performance improvement from having multiple ingestors inserting into different presplits of a table. I get 120K inserts/sec with one ingestor and 2x60K inserts/sec with two ingestors. In 2014 we got linear speedup to ~6 ingestors, providing ~600K inserts/sec on a single node.
Regards. -Jeremy
Re: Accumulo 1.10.1 performance
Posted by Mike Miller <mm...@apache.org>.
Hi Jeremy - Which specific version of 1.8 (1.8.0 or 1.8.1) did you test
that is still performing?
And I don't know of any benchmarking that was done for 1.10 or 1.9.
On Sat, Jun 5, 2021 at 10:04 PM Kepner, Jeremy - LLSC - MITLL <
kepner@ll.mit.edu> wrote:
> I did a quick check with Accumulo 1.8 and I get the expected single node
> performance scalability.
> So between Accumulo 1.8 and 1.10.1 something changed that significantly
> slowed the performance.
>
> > On Jun 5, 2021, at 8:48 PM, Kepner, Jeremy - LLSC - MITLL <
> kepner@ll.mit.edu> wrote:
> >
> > Hi Dave,
> > I am looking into the Accumulo/Hadoop configuration. Hopefully it is
> as simple as getting the settings the same. The hardware configurations is:
> Dual Xeon Platinum 8260 2.4 GHz 48 cores, DDR4 2.93 GHz 192 GB RAM. I am
> looking into the disk specs, but that shouldn't matter since the writes are
> only a few megabytes. I also just tested on some older hardware that is
> closer to what was used in the 2014 paper, and the single process ingest
> rate is ~8x slower.
> >
> > Has anyone done any recent benchmarking of Accumulo 1.10+?
> >
> > Regards. -Jeremy
> >
> >
> >> On Jun 5, 2021, at 7:08 PM, Dave Marion <dm...@gmail.com> wrote:
> >>
> >> Jeremy,
> >>
> >> Are you able to share any details about the hardware and the Accumulo
> >> configuration? Is the Accumulo/Hadoop configuration the same as the
> prior
> >> test (no replication, WAL turned off, batch writer configuration, etc.)
> >>
> >> Dave
> >>
> >> On Sat, Jun 5, 2021 at 6:12 PM Kepner, Jeremy - LLSC - MITLL <
> >> kepner@ll.mit.edu> wrote:
> >>
> >>> Has anyone benchmarked Accumulo 1.10.1? I have been looking into
> repeating
> >>> the measurements we did in 2014 with Accumulo 1.5 (
> >>> https://arxiv.org/abs/1406.4923) using Accumulo 1.10.1 on a bigger
> system
> >>> with more modern hardware. Unfortunately, when I repeat the single
> node
> >>> measurements, there is no performance improvement from having multiple
> >>> ingestors inserting into different presplits of a table. I get 120K
> >>> inserts/sec with one ingestor and 2x60K inserts/sec with two
> ingestors. In
> >>> 2014 we got linear speedup to ~6 ingestors, providing ~600K
> inserts/sec on
> >>> a single node.
> >>>
> >>> Regards. -Jeremy
> >
>
>
Re: Accumulo 1.10.1 performance
Posted by "Kepner, Jeremy - LLSC - MITLL" <ke...@ll.mit.edu>.
False alarm. Looks like we had some debugging flags on that severely reduced performance. Sorry about that.
Regards. -Jeremy
> On Jun 7, 2021, at 8:38 AM, Dave Marion <dl...@comcast.net> wrote:
>
> Jeremy,
>
> It seems that you have the ability to quickly run your test to determine if a release is "good". Testing 1.8 ruled out a lot of commits for us to look at. Would it be possible for you to test a few others so that we can try and narrow it down even more. The following releases are after 1.8.0:
>
> 1.8.1
> 1.9.0
> 1.9.1
> 1.9.2
> 1.9.3
> 1.10.0
> 1.10.1
>
> Could you test with the 1.9.2 release?
>
>
>> On 06/05/2021 10:04 PM Kepner, Jeremy - LLSC - MITLL <ke...@ll.mit.edu> wrote:
>>
>>
>> I did a quick check with Accumulo 1.8 and I get the expected single node performance scalability.
>> So between Accumulo 1.8 and 1.10.1 something changed that significantly slowed the performance.
>>
>>> On Jun 5, 2021, at 8:48 PM, Kepner, Jeremy - LLSC - MITLL <ke...@ll.mit.edu> wrote:
>>>
>>> Hi Dave,
>>> I am looking into the Accumulo/Hadoop configuration. Hopefully it is as simple as getting the settings the same. The hardware configurations is: Dual Xeon Platinum 8260 2.4 GHz 48 cores, DDR4 2.93 GHz 192 GB RAM. I am looking into the disk specs, but that shouldn't matter since the writes are only a few megabytes. I also just tested on some older hardware that is closer to what was used in the 2014 paper, and the single process ingest rate is ~8x slower.
>>>
>>> Has anyone done any recent benchmarking of Accumulo 1.10+?
>>>
>>> Regards. -Jeremy
>>>
>>>
>>>> On Jun 5, 2021, at 7:08 PM, Dave Marion <dm...@gmail.com> wrote:
>>>>
>>>> Jeremy,
>>>>
>>>> Are you able to share any details about the hardware and the Accumulo
>>>> configuration? Is the Accumulo/Hadoop configuration the same as the prior
>>>> test (no replication, WAL turned off, batch writer configuration, etc.)
>>>>
>>>> Dave
>>>>
>>>> On Sat, Jun 5, 2021 at 6:12 PM Kepner, Jeremy - LLSC - MITLL <
>>>> kepner@ll.mit.edu> wrote:
>>>>
>>>>> Has anyone benchmarked Accumulo 1.10.1? I have been looking into repeating
>>>>> the measurements we did in 2014 with Accumulo 1.5 (
>>>>> https://arxiv.org/abs/1406.4923) using Accumulo 1.10.1 on a bigger system
>>>>> with more modern hardware. Unfortunately, when I repeat the single node
>>>>> measurements, there is no performance improvement from having multiple
>>>>> ingestors inserting into different presplits of a table. I get 120K
>>>>> inserts/sec with one ingestor and 2x60K inserts/sec with two ingestors. In
>>>>> 2014 we got linear speedup to ~6 ingestors, providing ~600K inserts/sec on
>>>>> a single node.
>>>>>
>>>>> Regards. -Jeremy
>>>
Re: Accumulo 1.10.1 performance
Posted by Dave Marion <dl...@comcast.net>.
Jeremy,
It seems that you have the ability to quickly run your test to determine if a release is "good". Testing 1.8 ruled out a lot of commits for us to look at. Would it be possible for you to test a few others so that we can try and narrow it down even more. The following releases are after 1.8.0:
1.8.1
1.9.0
1.9.1
1.9.2
1.9.3
1.10.0
1.10.1
Could you test with the 1.9.2 release?
> On 06/05/2021 10:04 PM Kepner, Jeremy - LLSC - MITLL <ke...@ll.mit.edu> wrote:
>
>
> I did a quick check with Accumulo 1.8 and I get the expected single node performance scalability.
> So between Accumulo 1.8 and 1.10.1 something changed that significantly slowed the performance.
>
> > On Jun 5, 2021, at 8:48 PM, Kepner, Jeremy - LLSC - MITLL <ke...@ll.mit.edu> wrote:
> >
> > Hi Dave,
> > I am looking into the Accumulo/Hadoop configuration. Hopefully it is as simple as getting the settings the same. The hardware configurations is: Dual Xeon Platinum 8260 2.4 GHz 48 cores, DDR4 2.93 GHz 192 GB RAM. I am looking into the disk specs, but that shouldn't matter since the writes are only a few megabytes. I also just tested on some older hardware that is closer to what was used in the 2014 paper, and the single process ingest rate is ~8x slower.
> >
> > Has anyone done any recent benchmarking of Accumulo 1.10+?
> >
> > Regards. -Jeremy
> >
> >
> >> On Jun 5, 2021, at 7:08 PM, Dave Marion <dm...@gmail.com> wrote:
> >>
> >> Jeremy,
> >>
> >> Are you able to share any details about the hardware and the Accumulo
> >> configuration? Is the Accumulo/Hadoop configuration the same as the prior
> >> test (no replication, WAL turned off, batch writer configuration, etc.)
> >>
> >> Dave
> >>
> >> On Sat, Jun 5, 2021 at 6:12 PM Kepner, Jeremy - LLSC - MITLL <
> >> kepner@ll.mit.edu> wrote:
> >>
> >>> Has anyone benchmarked Accumulo 1.10.1? I have been looking into repeating
> >>> the measurements we did in 2014 with Accumulo 1.5 (
> >>> https://arxiv.org/abs/1406.4923) using Accumulo 1.10.1 on a bigger system
> >>> with more modern hardware. Unfortunately, when I repeat the single node
> >>> measurements, there is no performance improvement from having multiple
> >>> ingestors inserting into different presplits of a table. I get 120K
> >>> inserts/sec with one ingestor and 2x60K inserts/sec with two ingestors. In
> >>> 2014 we got linear speedup to ~6 ingestors, providing ~600K inserts/sec on
> >>> a single node.
> >>>
> >>> Regards. -Jeremy
> >
Re: Accumulo 1.10.1 performance
Posted by "Kepner, Jeremy - LLSC - MITLL" <ke...@ll.mit.edu>.
I did a quick check with Accumulo 1.8 and I get the expected single node performance scalability.
So between Accumulo 1.8 and 1.10.1 something changed that significantly slowed the performance.
> On Jun 5, 2021, at 8:48 PM, Kepner, Jeremy - LLSC - MITLL <ke...@ll.mit.edu> wrote:
>
> Hi Dave,
> I am looking into the Accumulo/Hadoop configuration. Hopefully it is as simple as getting the settings the same. The hardware configurations is: Dual Xeon Platinum 8260 2.4 GHz 48 cores, DDR4 2.93 GHz 192 GB RAM. I am looking into the disk specs, but that shouldn't matter since the writes are only a few megabytes. I also just tested on some older hardware that is closer to what was used in the 2014 paper, and the single process ingest rate is ~8x slower.
>
> Has anyone done any recent benchmarking of Accumulo 1.10+?
>
> Regards. -Jeremy
>
>
>> On Jun 5, 2021, at 7:08 PM, Dave Marion <dm...@gmail.com> wrote:
>>
>> Jeremy,
>>
>> Are you able to share any details about the hardware and the Accumulo
>> configuration? Is the Accumulo/Hadoop configuration the same as the prior
>> test (no replication, WAL turned off, batch writer configuration, etc.)
>>
>> Dave
>>
>> On Sat, Jun 5, 2021 at 6:12 PM Kepner, Jeremy - LLSC - MITLL <
>> kepner@ll.mit.edu> wrote:
>>
>>> Has anyone benchmarked Accumulo 1.10.1? I have been looking into repeating
>>> the measurements we did in 2014 with Accumulo 1.5 (
>>> https://arxiv.org/abs/1406.4923) using Accumulo 1.10.1 on a bigger system
>>> with more modern hardware. Unfortunately, when I repeat the single node
>>> measurements, there is no performance improvement from having multiple
>>> ingestors inserting into different presplits of a table. I get 120K
>>> inserts/sec with one ingestor and 2x60K inserts/sec with two ingestors. In
>>> 2014 we got linear speedup to ~6 ingestors, providing ~600K inserts/sec on
>>> a single node.
>>>
>>> Regards. -Jeremy
>
Re: Accumulo 1.10.1 performance
Posted by "Kepner, Jeremy - LLSC - MITLL" <ke...@ll.mit.edu>.
Hi Dave,
I am looking into the Accumulo/Hadoop configuration. Hopefully it is as simple as getting the settings the same. The hardware configurations is: Dual Xeon Platinum 8260 2.4 GHz 48 cores, DDR4 2.93 GHz 192 GB RAM. I am looking into the disk specs, but that shouldn't matter since the writes are only a few megabytes. I also just tested on some older hardware that is closer to what was used in the 2014 paper, and the single process ingest rate is ~8x slower.
Has anyone done any recent benchmarking of Accumulo 1.10+?
Regards. -Jeremy
> On Jun 5, 2021, at 7:08 PM, Dave Marion <dm...@gmail.com> wrote:
>
> Jeremy,
>
> Are you able to share any details about the hardware and the Accumulo
> configuration? Is the Accumulo/Hadoop configuration the same as the prior
> test (no replication, WAL turned off, batch writer configuration, etc.)
>
> Dave
>
> On Sat, Jun 5, 2021 at 6:12 PM Kepner, Jeremy - LLSC - MITLL <
> kepner@ll.mit.edu> wrote:
>
>> Has anyone benchmarked Accumulo 1.10.1? I have been looking into repeating
>> the measurements we did in 2014 with Accumulo 1.5 (
>> https://arxiv.org/abs/1406.4923) using Accumulo 1.10.1 on a bigger system
>> with more modern hardware. Unfortunately, when I repeat the single node
>> measurements, there is no performance improvement from having multiple
>> ingestors inserting into different presplits of a table. I get 120K
>> inserts/sec with one ingestor and 2x60K inserts/sec with two ingestors. In
>> 2014 we got linear speedup to ~6 ingestors, providing ~600K inserts/sec on
>> a single node.
>>
>> Regards. -Jeremy
Re: Accumulo 1.10.1 performance
Posted by Dave Marion <dm...@gmail.com>.
Jeremy,
Are you able to share any details about the hardware and the Accumulo
configuration? Is the Accumulo/Hadoop configuration the same as the prior
test (no replication, WAL turned off, batch writer configuration, etc.)
Dave
On Sat, Jun 5, 2021 at 6:12 PM Kepner, Jeremy - LLSC - MITLL <
kepner@ll.mit.edu> wrote:
> Has anyone benchmarked Accumulo 1.10.1? I have been looking into repeating
> the measurements we did in 2014 with Accumulo 1.5 (
> https://arxiv.org/abs/1406.4923) using Accumulo 1.10.1 on a bigger system
> with more modern hardware. Unfortunately, when I repeat the single node
> measurements, there is no performance improvement from having multiple
> ingestors inserting into different presplits of a table. I get 120K
> inserts/sec with one ingestor and 2x60K inserts/sec with two ingestors. In
> 2014 we got linear speedup to ~6 ingestors, providing ~600K inserts/sec on
> a single node.
>
> Regards. -Jeremy