You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@accumulo.apache.org by "Brown, Scott M (IS)" <Sc...@ngc.com> on 2016/10/20 15:55:22 UTC
RE: EXT :Re: 1.7 native library performance boost
We're testing identical systems, one with 1.6, and one with 1.7 since it was mentioned (and I can't find the link again) that using the native library instead of java library gave the gain.. we are testing my performing a large rya ingest thru accumulo, and also using Rya Webview and retrieving large queries.. the ingest is ~90 min, so a good run, and the query is ~20k record return.
-----Original Message-----
From: Josh Elser [mailto:josh.elser@gmail.com]
Sent: Thursday, October 20, 2016 10:02 AM
To: user@accumulo.apache.org
Subject: EXT :Re: 1.7 native library performance boost
Sorry, I'm not following. Was this a typo: "With *1.6*, ... native library can add up to 40%"?
What is the test you're running to evaluate performance? What are the numbers you're seeing for each scenario you're testing?
Brown, Scott M (IS) wrote:
> So with 1.7, we noticed the statement that utilizing the native library can add up to 40% performance increase. With two identical systems, one with 1.6 one with 1.7, we're not seeing any notable difference at all with either queries or large ingests.
>
> Any idea what operations can take advantage, or what circumstances the performance boost comes into play?
Re: EXT :Re: 1.7 native library performance boost
Posted by Josh Elser <jo...@gmail.com>.
Well, that's why you don't see any impact :). I'd recommend brushing up
on the Accumulo user manual[1] and maybe the BigTable architecture[2].
Bulk loading files into Accumulo avoids the write-ahead log and
in-memory map completely. Thus, a more efficient native map would have
next to no effect on the performance of your ingest task.
[1] http://accumulo.apache.org/1.8/accumulo_user_manual#_native_map
[2] http://research.google.com/archive/bigtable-osdi06.pdf
Brown, Scott M (IS) wrote:
> Yes, bulk .nt files with millions of triples..
>
> -----Original Message-----
> From: Josh Elser [mailto:josh.elser@gmail.com]
> Sent: Thursday, October 20, 2016 11:04 AM
> To: user@accumulo.apache.org
> Subject: Re: EXT :Re: 1.7 native library performance boost
>
> Ok, so the mention of using both 1.6 and 1.7 is what is confusing me.
> Yes, both versions of the software should equally benefit from using the Native library for live ingest.
>
> The Native Maps are going to benefit the write-path significantly more than the read-path. Is Rya maybe doing bulk-loads (creating Accumulo files)?
>
> Brown, Scott M (IS) wrote:
>> We're testing identical systems, one with 1.6, and one with 1.7 since it was mentioned (and I can't find the link again) that using the native library instead of java library gave the gain.. we are testing my performing a large rya ingest thru accumulo, and also using Rya Webview and retrieving large queries.. the ingest is ~90 min, so a good run, and the query is ~20k record return.
>>
>> -----Original Message-----
>> From: Josh Elser [mailto:josh.elser@gmail.com]
>> Sent: Thursday, October 20, 2016 10:02 AM
>> To: user@accumulo.apache.org
>> Subject: EXT :Re: 1.7 native library performance boost
>>
>> Sorry, I'm not following. Was this a typo: "With *1.6*, ... native library can add up to 40%"?
>>
>> What is the test you're running to evaluate performance? What are the numbers you're seeing for each scenario you're testing?
>>
>> Brown, Scott M (IS) wrote:
>>> So with 1.7, we noticed the statement that utilizing the native library can add up to 40% performance increase. With two identical systems, one with 1.6 one with 1.7, we're not seeing any notable difference at all with either queries or large ingests.
>>>
>>> Any idea what operations can take advantage, or what circumstances the performance boost comes into play?
>
RE: EXT :Re: 1.7 native library performance boost
Posted by "Brown, Scott M (IS)" <Sc...@ngc.com>.
Yes, bulk .nt files with millions of triples..
-----Original Message-----
From: Josh Elser [mailto:josh.elser@gmail.com]
Sent: Thursday, October 20, 2016 11:04 AM
To: user@accumulo.apache.org
Subject: Re: EXT :Re: 1.7 native library performance boost
Ok, so the mention of using both 1.6 and 1.7 is what is confusing me.
Yes, both versions of the software should equally benefit from using the Native library for live ingest.
The Native Maps are going to benefit the write-path significantly more than the read-path. Is Rya maybe doing bulk-loads (creating Accumulo files)?
Brown, Scott M (IS) wrote:
> We're testing identical systems, one with 1.6, and one with 1.7 since it was mentioned (and I can't find the link again) that using the native library instead of java library gave the gain.. we are testing my performing a large rya ingest thru accumulo, and also using Rya Webview and retrieving large queries.. the ingest is ~90 min, so a good run, and the query is ~20k record return.
>
> -----Original Message-----
> From: Josh Elser [mailto:josh.elser@gmail.com]
> Sent: Thursday, October 20, 2016 10:02 AM
> To: user@accumulo.apache.org
> Subject: EXT :Re: 1.7 native library performance boost
>
> Sorry, I'm not following. Was this a typo: "With *1.6*, ... native library can add up to 40%"?
>
> What is the test you're running to evaluate performance? What are the numbers you're seeing for each scenario you're testing?
>
> Brown, Scott M (IS) wrote:
>> So with 1.7, we noticed the statement that utilizing the native library can add up to 40% performance increase. With two identical systems, one with 1.6 one with 1.7, we're not seeing any notable difference at all with either queries or large ingests.
>>
>> Any idea what operations can take advantage, or what circumstances the performance boost comes into play?
>
Re: EXT :Re: 1.7 native library performance boost
Posted by Josh Elser <jo...@gmail.com>.
Ok, so the mention of using both 1.6 and 1.7 is what is confusing me.
Yes, both versions of the software should equally benefit from using the
Native library for live ingest.
The Native Maps are going to benefit the write-path significantly more
than the read-path. Is Rya maybe doing bulk-loads (creating Accumulo files)?
Brown, Scott M (IS) wrote:
> We're testing identical systems, one with 1.6, and one with 1.7 since it was mentioned (and I can't find the link again) that using the native library instead of java library gave the gain.. we are testing my performing a large rya ingest thru accumulo, and also using Rya Webview and retrieving large queries.. the ingest is ~90 min, so a good run, and the query is ~20k record return.
>
> -----Original Message-----
> From: Josh Elser [mailto:josh.elser@gmail.com]
> Sent: Thursday, October 20, 2016 10:02 AM
> To: user@accumulo.apache.org
> Subject: EXT :Re: 1.7 native library performance boost
>
> Sorry, I'm not following. Was this a typo: "With *1.6*, ... native library can add up to 40%"?
>
> What is the test you're running to evaluate performance? What are the numbers you're seeing for each scenario you're testing?
>
> Brown, Scott M (IS) wrote:
>> So with 1.7, we noticed the statement that utilizing the native library can add up to 40% performance increase. With two identical systems, one with 1.6 one with 1.7, we're not seeing any notable difference at all with either queries or large ingests.
>>
>> Any idea what operations can take advantage, or what circumstances the performance boost comes into play?
>