You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@jena.apache.org by Simon Helsen <sh...@ca.ibm.com> on 2012/06/13 16:06:47 UTC
concurrency and slower results
hi everyone, Andy especially,
it looks like some of the changes to TDB since May 15 have negatively
affected the concurrent behavior of Tx. To clarify, we are doing some
serious internal performance testing on a product which is based on
Jena/TDB. We initially did these runs to compare Tx (2.7.1 snapshot of May
15) with our old TDB implementation (which used a conservative locking
model). The results were good, especially with long-running/expensive
index update operations.
Now, I had asked the performance team to rerun their tests with a new
build of the product based on the release candidate (of last weekend) and
they are reporting a concerning degradation. Compared to the May 15 TDB,
they are reporting 12-15% decrease in query performance when doing a
"light update load" and about 40% query and 40% update performance
degradation when doing a "heave update load".
That is quite a serious regression. The question is what changed since May
15 that could have such a bad effect on performance. The tests included
only 25 concurrent users on about 15 million triples. Yet, I wonder if
some of the recent lock changes (see JENA-252) are responsible
thanks
Simon
Re: concurrency and slower results
Posted by Simon Helsen <sh...@ca.ibm.com>.
I was actually able to run a bunch of simpler tests with no concurrency
and compare the exact same implementation of our product with (tx) and
without transactions (vanilla) and it seems that the time is going to
write activities. In my suite, it took Tx about 70% longer to write the
same data. In these non concurrency tests, queries were running with
roughly the same speed. I assume that the slowdown on queries with 25
users is caused by lock contention on the writes which are taking much
longer for some reason
Simon
From:
Laurent Pellegrino <la...@gmail.com>
To:
dev@jena.apache.org
Date:
06/13/2012 11:10 AM
Subject:
Re: concurrency and slower results
Hello,
I also noticed the same thing by updating jena-tdb only. I switched
from version 0.9.0-incubating to snapshots on 6 June and I noticed
around ~50% decrease on one of our benchmark which is read/write
intensive.
Kind Regards,
Laurent
On Wed, Jun 13, 2012 at 4:33 PM, Simon Helsen <sh...@ca.ibm.com> wrote:
> before you ask, of course, we are looking into whether this is something
> on our end. We only made very few changes, but we should be able to
> isolate this. I'll report on that once I know for sure. My question was
> more whether anyone knew of any changes in tx that could affect
concurrent
> performance negatively since May 15
>
> thanks
>
> Simon
>
>
>
>
>
> From:
> Simon Helsen/Toronto/IBM@IBMCA
> To:
> dev@jena.apache.org
> Date:
> 06/13/2012 10:09 AM
> Subject:
> concurrency and slower results
>
>
>
> hi everyone, Andy especially,
>
> it looks like some of the changes to TDB since May 15 have negatively
> affected the concurrent behavior of Tx. To clarify, we are doing some
> serious internal performance testing on a product which is based on
> Jena/TDB. We initially did these runs to compare Tx (2.7.1 snapshot of
May
>
> 15) with our old TDB implementation (which used a conservative locking
> model). The results were good, especially with long-running/expensive
> index update operations.
>
> Now, I had asked the performance team to rerun their tests with a new
> build of the product based on the release candidate (of last weekend)
and
> they are reporting a concerning degradation. Compared to the May 15 TDB,
> they are reporting 12-15% decrease in query performance when doing a
> "light update load" and about 40% query and 40% update performance
> degradation when doing a "heave update load".
>
> That is quite a serious regression. The question is what changed since
May
>
> 15 that could have such a bad effect on performance. The tests included
> only 25 concurrent users on about 15 million triples. Yet, I wonder if
> some of the recent lock changes (see JENA-252) are responsible
>
> thanks
>
> Simon
>
>
>
Re: concurrency and slower results
Posted by Simon Helsen <sh...@ca.ibm.com>.
I replied to this, but not sure if the mailing list accepts as many
attachments. I created
https://issues.apache.org/jira/browse/JENA-256
which includes the test and data to demonstrate the problem
Simon
From:
Damian Steer <d....@bristol.ac.uk>
To:
dev@jena.apache.org
Date:
06/14/2012 10:16 AM
Subject:
Re: concurrency and slower results
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
On 14/06/12 14:45, Simon Helsen wrote:
> However, in order to test the regression, I'd need a 2.7.1 snapshot
> from May 15. I checked the snapshots pages and they are gone. How
> do you suggest to get the older build back? (I don't want to use
> 0.9.0 because there are too many differences -> based on our
> experience, the regression is very recent)
>
> Simon
You can check out the state of the repository at a particular date using:
svn update -r {2012-05-15}
Damian
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iEYEARECAAYFAk/Z8Z4ACgkQAyLCB+mTtykrzQCg1tn13P/tsIE52HCb9g1DPCoa
2E0AoIHTVf+mWRGzZuulZqoWc7JBrcqi
=L0yM
-----END PGP SIGNATURE-----
Re: concurrency and slower results
Posted by Simon Helsen <sh...@ca.ibm.com>.
thanks Damian
I was able to reproduce the observed problem with an isolated test
scenario. So I recreated the components ARQ, CORE, IRI, and TDB with the
SVN state of May 15 9 (so svn update -r {2012-05-15} and then svn clean
install)
I then created a simple test program (attached as
PerformanceRegressionTest.java) which I ran 4 times in a row for each
version of Jena. Note that I deleted the TDB directory after the first 4
runs before using the other Jena version. Attached are the files with the
output. The regression is obvious
Andy: could you please take a look at what would have caused this and if
the performance regression is unavoidable. Note that we are talking just
about write operations, no queries, no release store or anything like
that.
thanks
Simon
From:
Damian Steer <d....@bristol.ac.uk>
To:
dev@jena.apache.org
Date:
06/14/2012 10:16 AM
Subject:
Re: concurrency and slower results
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
On 14/06/12 14:45, Simon Helsen wrote:
> However, in order to test the regression, I'd need a 2.7.1 snapshot
> from May 15. I checked the snapshots pages and they are gone. How
> do you suggest to get the older build back? (I don't want to use
> 0.9.0 because there are too many differences -> based on our
> experience, the regression is very recent)
>
> Simon
You can check out the state of the repository at a particular date using:
svn update -r {2012-05-15}
Damian
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iEYEARECAAYFAk/Z8Z4ACgkQAyLCB+mTtykrzQCg1tn13P/tsIE52HCb9g1DPCoa
2E0AoIHTVf+mWRGzZuulZqoWc7JBrcqi
=L0yM
-----END PGP SIGNATURE-----
Re: concurrency and slower results
Posted by Damian Steer <d....@bristol.ac.uk>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
On 14/06/12 14:45, Simon Helsen wrote:
> However, in order to test the regression, I'd need a 2.7.1 snapshot
> from May 15. I checked the snapshots pages and they are gone. How
> do you suggest to get the older build back? (I don't want to use
> 0.9.0 because there are too many differences -> based on our
> experience, the regression is very recent)
>
> Simon
You can check out the state of the repository at a particular date using:
svn update -r {2012-05-15}
Damian
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iEYEARECAAYFAk/Z8Z4ACgkQAyLCB+mTtykrzQCg1tn13P/tsIE52HCb9g1DPCoa
2E0AoIHTVf+mWRGzZuulZqoWc7JBrcqi
=L0yM
-----END PGP SIGNATURE-----
Re: concurrency and slower results
Posted by Simon Helsen <sh...@ca.ibm.com>.
In general I agree
but as I have reported, it looks like in this case, even single-threaded
write operations are taking a considerable longer time. The load is small
and although my own junit uses store releasing, our client, in their
performance runs, don't ever use store release. I cannot prove this just
yet, but it smells like simple (single-threaded) write operations are
slower.
However, in order to test the regression, I'd need a 2.7.1 snapshot from
May 15. I checked the snapshots pages and they are gone. How do you
suggest to get the older build back? (I don't want to use 0.9.0 because
there are too many differences -> based on our experience, the regression
is very recent)
Simon
From:
Andy Seaborne <an...@apache.org>
To:
dev@jena.apache.org
Date:
06/14/2012 07:57 AM
Subject:
Re: concurrency and slower results
On 14/06/12 11:52, Laurent Pellegrino wrote:
> It is strongly linked to our application. I think it would be simpler
> to write new test cases to include them with others. Ideally, the
> results (e.g. the throughput) could be plotted automatically by
> Jenkins with the Plot plugin
> (https://wiki.jenkins-ci.org/display/JENKINS/Plot+Plugin) to easily
> check differences between builds.
>
> Laurent
What would be helpful is to know the queries, the characteristics of the
data and the use model (e.g. update / query mix, total load,
releasing stores etc). Otherwise, it's the proverbial hunting for a
needle in a haystack.
Were other benchmarks affected? If not, what's the key difference?
Andy
>
> On Wed, Jun 13, 2012 at 10:57 PM, Andy Seaborne<an...@apache.org> wrote:
>> On 13/06/12 16:06, Laurent Pellegrino wrote:
>>>
>>> Hello,
>>>
>>> I also noticed the same thing by updating jena-tdb only. I switched
>>> from version 0.9.0-incubating to snapshots on 6 June and I noticed
>>> around ~50% decrease on one of our benchmark which is read/write
>>> intensive.
>>>
>>> Kind Regards,
>>>
>>> Laurent
>>
>>
>> Can you share the benchmark system?
>>
>> Andy
>>
>>
>>>
>>> On Wed, Jun 13, 2012 at 4:33 PM, Simon Helsen<sh...@ca.ibm.com>
wrote:
>>>>
>>>> before you ask, of course, we are looking into whether this is
something
>>>> on our end. We only made very few changes, but we should be able to
>>>> isolate this. I'll report on that once I know for sure. My question
was
>>>> more whether anyone knew of any changes in tx that could affect
>>>> concurrent
>>>> performance negatively since May 15
>>>>
>>>> thanks
>>>>
>>>> Simon
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> From:
>>>> Simon Helsen/Toronto/IBM@IBMCA
>>>> To:
>>>> dev@jena.apache.org
>>>> Date:
>>>> 06/13/2012 10:09 AM
>>>> Subject:
>>>> concurrency and slower results
>>>>
>>>>
>>>>
>>>> hi everyone, Andy especially,
>>>>
>>>> it looks like some of the changes to TDB since May 15 have negatively
>>>> affected the concurrent behavior of Tx. To clarify, we are doing some
>>>> serious internal performance testing on a product which is based on
>>>> Jena/TDB. We initially did these runs to compare Tx (2.7.1 snapshot
of
>>>> May
>>>>
>>>> 15) with our old TDB implementation (which used a conservative
locking
>>>> model). The results were good, especially with long-running/expensive
>>>> index update operations.
>>>>
>>>> Now, I had asked the performance team to rerun their tests with a new
>>>> build of the product based on the release candidate (of last weekend)
and
>>>> they are reporting a concerning degradation. Compared to the May 15
TDB,
>>>> they are reporting 12-15% decrease in query performance when doing a
>>>> "light update load" and about 40% query and 40% update performance
>>>> degradation when doing a "heave update load".
>>>>
>>>> That is quite a serious regression. The question is what changed
since
>>>> May
>>>>
>>>> 15 that could have such a bad effect on performance. The tests
included
>>>> only 25 concurrent users on about 15 million triples. Yet, I wonder
if
>>>> some of the recent lock changes (see JENA-252) are responsible
>>>>
>>>> thanks
>>>>
>>>> Simon
>>>>
>>>>
>>>>
>>
Re: concurrency and slower results
Posted by Andy Seaborne <an...@apache.org>.
On 14/06/12 11:52, Laurent Pellegrino wrote:
> It is strongly linked to our application. I think it would be simpler
> to write new test cases to include them with others. Ideally, the
> results (e.g. the throughput) could be plotted automatically by
> Jenkins with the Plot plugin
> (https://wiki.jenkins-ci.org/display/JENKINS/Plot+Plugin) to easily
> check differences between builds.
>
> Laurent
What would be helpful is to know the queries, the characteristics of the
data and the use model (e.g. update / query mix, total load,
releasing stores etc). Otherwise, it's the proverbial hunting for a
needle in a haystack.
Were other benchmarks affected? If not, what's the key difference?
Andy
>
> On Wed, Jun 13, 2012 at 10:57 PM, Andy Seaborne<an...@apache.org> wrote:
>> On 13/06/12 16:06, Laurent Pellegrino wrote:
>>>
>>> Hello,
>>>
>>> I also noticed the same thing by updating jena-tdb only. I switched
>>> from version 0.9.0-incubating to snapshots on 6 June and I noticed
>>> around ~50% decrease on one of our benchmark which is read/write
>>> intensive.
>>>
>>> Kind Regards,
>>>
>>> Laurent
>>
>>
>> Can you share the benchmark system?
>>
>> Andy
>>
>>
>>>
>>> On Wed, Jun 13, 2012 at 4:33 PM, Simon Helsen<sh...@ca.ibm.com> wrote:
>>>>
>>>> before you ask, of course, we are looking into whether this is something
>>>> on our end. We only made very few changes, but we should be able to
>>>> isolate this. I'll report on that once I know for sure. My question was
>>>> more whether anyone knew of any changes in tx that could affect
>>>> concurrent
>>>> performance negatively since May 15
>>>>
>>>> thanks
>>>>
>>>> Simon
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> From:
>>>> Simon Helsen/Toronto/IBM@IBMCA
>>>> To:
>>>> dev@jena.apache.org
>>>> Date:
>>>> 06/13/2012 10:09 AM
>>>> Subject:
>>>> concurrency and slower results
>>>>
>>>>
>>>>
>>>> hi everyone, Andy especially,
>>>>
>>>> it looks like some of the changes to TDB since May 15 have negatively
>>>> affected the concurrent behavior of Tx. To clarify, we are doing some
>>>> serious internal performance testing on a product which is based on
>>>> Jena/TDB. We initially did these runs to compare Tx (2.7.1 snapshot of
>>>> May
>>>>
>>>> 15) with our old TDB implementation (which used a conservative locking
>>>> model). The results were good, especially with long-running/expensive
>>>> index update operations.
>>>>
>>>> Now, I had asked the performance team to rerun their tests with a new
>>>> build of the product based on the release candidate (of last weekend) and
>>>> they are reporting a concerning degradation. Compared to the May 15 TDB,
>>>> they are reporting 12-15% decrease in query performance when doing a
>>>> "light update load" and about 40% query and 40% update performance
>>>> degradation when doing a "heave update load".
>>>>
>>>> That is quite a serious regression. The question is what changed since
>>>> May
>>>>
>>>> 15 that could have such a bad effect on performance. The tests included
>>>> only 25 concurrent users on about 15 million triples. Yet, I wonder if
>>>> some of the recent lock changes (see JENA-252) are responsible
>>>>
>>>> thanks
>>>>
>>>> Simon
>>>>
>>>>
>>>>
>>
Re: concurrency and slower results
Posted by Simon Helsen <sh...@ca.ibm.com>.
Same here (on the bit that it is strongly linked to our application).
yes, an automatic performance comparison between builds would be the best
way to avoid performance regressions
Simon
From:
Laurent Pellegrino <la...@gmail.com>
To:
dev@jena.apache.org
Date:
06/14/2012 06:53 AM
Subject:
Re: concurrency and slower results
It is strongly linked to our application. I think it would be simpler
to write new test cases to include them with others. Ideally, the
results (e.g. the throughput) could be plotted automatically by
Jenkins with the Plot plugin
(https://wiki.jenkins-ci.org/display/JENKINS/Plot+Plugin) to easily
check differences between builds.
Laurent
On Wed, Jun 13, 2012 at 10:57 PM, Andy Seaborne <an...@apache.org> wrote:
> On 13/06/12 16:06, Laurent Pellegrino wrote:
>>
>> Hello,
>>
>> I also noticed the same thing by updating jena-tdb only. I switched
>> from version 0.9.0-incubating to snapshots on 6 June and I noticed
>> around ~50% decrease on one of our benchmark which is read/write
>> intensive.
>>
>> Kind Regards,
>>
>> Laurent
>
>
> Can you share the benchmark system?
>
> Andy
>
>
>>
>> On Wed, Jun 13, 2012 at 4:33 PM, Simon Helsen<sh...@ca.ibm.com>
wrote:
>>>
>>> before you ask, of course, we are looking into whether this is
something
>>> on our end. We only made very few changes, but we should be able to
>>> isolate this. I'll report on that once I know for sure. My question
was
>>> more whether anyone knew of any changes in tx that could affect
>>> concurrent
>>> performance negatively since May 15
>>>
>>> thanks
>>>
>>> Simon
>>>
>>>
>>>
>>>
>>>
>>> From:
>>> Simon Helsen/Toronto/IBM@IBMCA
>>> To:
>>> dev@jena.apache.org
>>> Date:
>>> 06/13/2012 10:09 AM
>>> Subject:
>>> concurrency and slower results
>>>
>>>
>>>
>>> hi everyone, Andy especially,
>>>
>>> it looks like some of the changes to TDB since May 15 have negatively
>>> affected the concurrent behavior of Tx. To clarify, we are doing some
>>> serious internal performance testing on a product which is based on
>>> Jena/TDB. We initially did these runs to compare Tx (2.7.1 snapshot of
>>> May
>>>
>>> 15) with our old TDB implementation (which used a conservative locking
>>> model). The results were good, especially with long-running/expensive
>>> index update operations.
>>>
>>> Now, I had asked the performance team to rerun their tests with a new
>>> build of the product based on the release candidate (of last weekend)
and
>>> they are reporting a concerning degradation. Compared to the May 15
TDB,
>>> they are reporting 12-15% decrease in query performance when doing a
>>> "light update load" and about 40% query and 40% update performance
>>> degradation when doing a "heave update load".
>>>
>>> That is quite a serious regression. The question is what changed since
>>> May
>>>
>>> 15 that could have such a bad effect on performance. The tests
included
>>> only 25 concurrent users on about 15 million triples. Yet, I wonder if
>>> some of the recent lock changes (see JENA-252) are responsible
>>>
>>> thanks
>>>
>>> Simon
>>>
>>>
>>>
>
Re: concurrency and slower results
Posted by Laurent Pellegrino <la...@gmail.com>.
It is strongly linked to our application. I think it would be simpler
to write new test cases to include them with others. Ideally, the
results (e.g. the throughput) could be plotted automatically by
Jenkins with the Plot plugin
(https://wiki.jenkins-ci.org/display/JENKINS/Plot+Plugin) to easily
check differences between builds.
Laurent
On Wed, Jun 13, 2012 at 10:57 PM, Andy Seaborne <an...@apache.org> wrote:
> On 13/06/12 16:06, Laurent Pellegrino wrote:
>>
>> Hello,
>>
>> I also noticed the same thing by updating jena-tdb only. I switched
>> from version 0.9.0-incubating to snapshots on 6 June and I noticed
>> around ~50% decrease on one of our benchmark which is read/write
>> intensive.
>>
>> Kind Regards,
>>
>> Laurent
>
>
> Can you share the benchmark system?
>
> Andy
>
>
>>
>> On Wed, Jun 13, 2012 at 4:33 PM, Simon Helsen<sh...@ca.ibm.com> wrote:
>>>
>>> before you ask, of course, we are looking into whether this is something
>>> on our end. We only made very few changes, but we should be able to
>>> isolate this. I'll report on that once I know for sure. My question was
>>> more whether anyone knew of any changes in tx that could affect
>>> concurrent
>>> performance negatively since May 15
>>>
>>> thanks
>>>
>>> Simon
>>>
>>>
>>>
>>>
>>>
>>> From:
>>> Simon Helsen/Toronto/IBM@IBMCA
>>> To:
>>> dev@jena.apache.org
>>> Date:
>>> 06/13/2012 10:09 AM
>>> Subject:
>>> concurrency and slower results
>>>
>>>
>>>
>>> hi everyone, Andy especially,
>>>
>>> it looks like some of the changes to TDB since May 15 have negatively
>>> affected the concurrent behavior of Tx. To clarify, we are doing some
>>> serious internal performance testing on a product which is based on
>>> Jena/TDB. We initially did these runs to compare Tx (2.7.1 snapshot of
>>> May
>>>
>>> 15) with our old TDB implementation (which used a conservative locking
>>> model). The results were good, especially with long-running/expensive
>>> index update operations.
>>>
>>> Now, I had asked the performance team to rerun their tests with a new
>>> build of the product based on the release candidate (of last weekend) and
>>> they are reporting a concerning degradation. Compared to the May 15 TDB,
>>> they are reporting 12-15% decrease in query performance when doing a
>>> "light update load" and about 40% query and 40% update performance
>>> degradation when doing a "heave update load".
>>>
>>> That is quite a serious regression. The question is what changed since
>>> May
>>>
>>> 15 that could have such a bad effect on performance. The tests included
>>> only 25 concurrent users on about 15 million triples. Yet, I wonder if
>>> some of the recent lock changes (see JENA-252) are responsible
>>>
>>> thanks
>>>
>>> Simon
>>>
>>>
>>>
>
Re: concurrency and slower results
Posted by Andy Seaborne <an...@apache.org>.
On 13/06/12 16:06, Laurent Pellegrino wrote:
> Hello,
>
> I also noticed the same thing by updating jena-tdb only. I switched
> from version 0.9.0-incubating to snapshots on 6 June and I noticed
> around ~50% decrease on one of our benchmark which is read/write
> intensive.
>
> Kind Regards,
>
> Laurent
Can you share the benchmark system?
Andy
>
> On Wed, Jun 13, 2012 at 4:33 PM, Simon Helsen<sh...@ca.ibm.com> wrote:
>> before you ask, of course, we are looking into whether this is something
>> on our end. We only made very few changes, but we should be able to
>> isolate this. I'll report on that once I know for sure. My question was
>> more whether anyone knew of any changes in tx that could affect concurrent
>> performance negatively since May 15
>>
>> thanks
>>
>> Simon
>>
>>
>>
>>
>>
>> From:
>> Simon Helsen/Toronto/IBM@IBMCA
>> To:
>> dev@jena.apache.org
>> Date:
>> 06/13/2012 10:09 AM
>> Subject:
>> concurrency and slower results
>>
>>
>>
>> hi everyone, Andy especially,
>>
>> it looks like some of the changes to TDB since May 15 have negatively
>> affected the concurrent behavior of Tx. To clarify, we are doing some
>> serious internal performance testing on a product which is based on
>> Jena/TDB. We initially did these runs to compare Tx (2.7.1 snapshot of May
>>
>> 15) with our old TDB implementation (which used a conservative locking
>> model). The results were good, especially with long-running/expensive
>> index update operations.
>>
>> Now, I had asked the performance team to rerun their tests with a new
>> build of the product based on the release candidate (of last weekend) and
>> they are reporting a concerning degradation. Compared to the May 15 TDB,
>> they are reporting 12-15% decrease in query performance when doing a
>> "light update load" and about 40% query and 40% update performance
>> degradation when doing a "heave update load".
>>
>> That is quite a serious regression. The question is what changed since May
>>
>> 15 that could have such a bad effect on performance. The tests included
>> only 25 concurrent users on about 15 million triples. Yet, I wonder if
>> some of the recent lock changes (see JENA-252) are responsible
>>
>> thanks
>>
>> Simon
>>
>>
>>
Re: Voting on releases (WAS concurrency and slower results)
Posted by Rob Vesse <rv...@yarcdata.com>.
On 6/14/12 12:58 AM, "Andy Seaborne" <an...@apache.org> wrote:
>On 13/06/12 23:46, Rob Vesse wrote:
>> Yes particularly in the context of components like ARQ there are a lot
>>of
>> bug and performance fixes plus other minor enhancements that I and
>>others
>> have worked on.
>>
>> For the record ARQ hasn't received a new release in 6 months so while
>> there are clearly some TDB issues to be resolved it is best to get a
>> release out the door to get the fixes to that and other components into
>> the hands of users.
>
>And even for TDB, it's been getting on for 4 months and the last version
>was a major change with the introduction of transactions. Ditto Fuseki
>(3 months) which incorporated TDB.
>
>> More frequent releases should hopefully be a lot easier now we are a TLP
>> and don't have quite so many hurdles to jump through to get our release
>> done (we now only have to hold a successful release vote ourselves
>>rather
>> than also having to pass through the Incubator process)
>
>All true though the biggest time-sink has been migration. The Apache
>process added a few days of latency but much of the time/resource has
>gone in set-up; change of infrastructure, much (much, much) better
>licensing clarity on the codebase.
>
>Releases will get increasingly easier ... and PMC members know how to
>check and vote (hint, hint!)
Yes I am going to take a look at it today hopefully
Still snowed under with work and the joys of filling out US government
forms which are the most impressive achievement in bureaucracy that the
western world has ever made ;-)
Rob
>
> Andy
Re: concurrency and slower results
Posted by Andy Seaborne <an...@apache.org>.
On 13/06/12 23:46, Rob Vesse wrote:
> Yes particularly in the context of components like ARQ there are a lot of
> bug and performance fixes plus other minor enhancements that I and others
> have worked on.
>
> For the record ARQ hasn't received a new release in 6 months so while
> there are clearly some TDB issues to be resolved it is best to get a
> release out the door to get the fixes to that and other components into
> the hands of users.
And even for TDB, it's been getting on for 4 months and the last version
was a major change with the introduction of transactions. Ditto Fuseki
(3 months) which incorporated TDB.
> More frequent releases should hopefully be a lot easier now we are a TLP
> and don't have quite so many hurdles to jump through to get our release
> done (we now only have to hold a successful release vote ourselves rather
> than also having to pass through the Incubator process)
All true though the biggest time-sink has been migration. The Apache
process added a few days of latency but much of the time/resource has
gone in set-up; change of infrastructure, much (much, much) better
licensing clarity on the codebase.
Releases will get increasingly easier ... and PMC members know how to
check and vote (hint, hint!)
Andy
Re: concurrency and slower results
Posted by Rob Vesse <rv...@yarcdata.com>.
Yes particularly in the context of components like ARQ there are a lot of
bug and performance fixes plus other minor enhancements that I and others
have worked on.
For the record ARQ hasn't received a new release in 6 months so while
there are clearly some TDB issues to be resolved it is best to get a
release out the door to get the fixes to that and other components into
the hands of users.
More frequent releases should hopefully be a lot easier now we are a TLP
and don't have quite so many hurdles to jump through to get our release
done (we now only have to hold a successful release vote ourselves rather
than also having to pass through the Incubator process)
Rob
Rob Vesse -- YarcData.com -- A Division of Cray Inc
Software Engineer, Bay Area
m: 925.960.3941 | o: 925.264.4729 | @: rvesse@yarcdata.com | Skype:
rvesse
6210 Stoneridge Mall Rd | Suite 120 | Pleasanton CA, 94588
On 6/13/12 3:34 PM, "Andy Seaborne" <an...@apache.org> wrote:
>> PS: Is this concern not a good reason to delay the release, at least
>> until it is understood?
>
>See http://s.apache.org/pt and also the fact there may well be another
>release quite soon for, if nothing else, a better NodeTableTrans.
>
>There are many things in the release - non-TDB bug fixes and
>improvements, close tracking of SPARQL 1.1, etc etc. And for TDB,
>better protection of data due to fixes since 0.9.0-incubating.
>
>And the release doesn't say "-incubating"
>
>So, in the spirit of continuous improvement, I think the release should
>go ahead, and I have sent the VOTE message.
>
>Release early, release often.
>
> Andy
>
>
Re: concurrency and slower results
Posted by Andy Seaborne <an...@apache.org>.
On 14/06/12 02:47, Simon Helsen wrote:
> when you say there may well be another release quite soon", how soon is
> soon? Is there a plan?
"Soon" is specifically not a timescale.
http://mail-archives.apache.org/mod_mbox/jena-dev/201206.mbox/%3C4FD4B8E4.8080709%40apache.org%3E
Where you can help is in characterising the performance issue. At the
moment, from my point of view, you have reported a slow down but there
is no model of what is slower. You, and possibly Laurent, use store
releasing - that has changed, maybe it's the cause, maybe it's not
relevant. Laurent's case may not even be TDB as he jumped quite a few
ARQ changes as well (e.g. your report slower query, now fixed).
If you get a additional test or tests into the suite that reflect your
usage, it will be tested in the development cycle. If we get enough
systematic tests and a framework to run and notice significant
slowdowns, we can add it to the nightly build cycle.
So currently, there is NodeTableTrans and the performance investigation.
Timescale? Depends on inputs.
Andy
Re: concurrency and slower results
Posted by Simon Helsen <sh...@ca.ibm.com>.
when you say there may well be another release quite soon", how soon is
soon? Is there a plan?
Simon
From:
Andy Seaborne <an...@apache.org>
To:
Simon Helsen/Toronto/IBM@IBMCA
Cc:
dev@jena.apache.org, Andy Seaborne <an...@gmail.com>
Date:
06/13/2012 06:35 PM
Subject:
Re: concurrency and slower results
> PS: Is this concern not a good reason to delay the release, at least
> until it is understood?
See http://s.apache.org/pt and also the fact there may well be another
release quite soon for, if nothing else, a better NodeTableTrans.
There are many things in the release - non-TDB bug fixes and
improvements, close tracking of SPARQL 1.1, etc etc. And for TDB,
better protection of data due to fixes since 0.9.0-incubating.
And the release doesn't say "-incubating"
So, in the spirit of continuous improvement, I think the release should
go ahead, and I have sent the VOTE message.
Release early, release often.
Andy
Re: concurrency and slower results
Posted by Andy Seaborne <an...@apache.org>.
> PS: Is this concern not a good reason to delay the release, at least
> until it is understood?
See http://s.apache.org/pt and also the fact there may well be another
release quite soon for, if nothing else, a better NodeTableTrans.
There are many things in the release - non-TDB bug fixes and
improvements, close tracking of SPARQL 1.1, etc etc. And for TDB,
better protection of data due to fixes since 0.9.0-incubating.
And the release doesn't say "-incubating"
So, in the spirit of continuous improvement, I think the release should
go ahead, and I have sent the VOTE message.
Release early, release often.
Andy
Re: concurrency and slower results
Posted by Simon Helsen <sh...@ca.ibm.com>.
Andy,
I was probably not all that clear. I admit that I was initially suspecting
increased lock contention, but I don't think those are the cause here
because our internal regular (functionally-oriented and mostly
single-threaded) junit suite slowed down by 25%. When I looked at the
underlying stats (which we always collect in our framework), I noticed
that the total suite index write time took 3 times as long. I did this by
comparing Tx and nonTx because I know that only last week with the May 15
TDB build, this junit suite was running at about the same speed in both tx
and non-tx. (I am not claiming tx is as fast as non-tx in all
circumstances, but we had observed significantly faster index write times
under almost all circumstances)
Let me stress that these are not even performance tests. I only executed
this experiment because our internal client adopted a new build of ours
with the latest TDB build (last weekend, the release candidate) and
suddenly reported the 40% slowdown on both index and query compared to the
previous TDB (tx) build of May 15 for 25 concurrent users in a write-heavy
test scenario.
So, the question is how we can identify the culprit. Perhaps there is even
a legitimate reason, but Laurent seems to have observed the same with
respect to the 0.9.0 build. Could we not use some of the existing Jena
unit tests which verify concurrent query/write behavior in Tx? Perhaps
increasing the load to get more useful execution times and then run them
on the SVN state of May 15 and HEAD and compare. I am not sure if it would
narrow down the problem, but at least it would demonstrate the regression
Simon
PS: Is this concern not a good reason to delay the release, at least until
it is understood?
From:
Andy Seaborne <an...@apache.org>
To:
dev@jena.apache.org
Date:
06/13/2012 06:08 PM
Subject:
Re: concurrency and slower results
On 13/06/12 18:16, Simon Helsen wrote:
> I have to correct my statement below, I was misreading my own numbers.
> The overall write time was about 3 times slower, not just 70%. This is
> comparing the latest build, once with and once without transactions
> (single user). The numbers (40%) I mentioned before came from 25
> concurrent users on a build using transactions from May 15 compared to a
> build using transactions from last weekend. That is not quite the same
> kind of comparison of course. However, I do recall (I don't have that on
> paper) that my own test suite with the May 15 build was running at a
> comparable speed whether I was using transactions or not. So I dare to
> conclude that write transactions are now considerable more expensive
> than around May 15
Is there some way we can turn this into a publicly usable example?
The recent changes to fix problems (JENA-250, JENA-252) might change the
concurrency but the changes also added a potential file system sync the
dataset when transitioning at first use transactionally. Depending on
how you use StoreConenction.release, that might have an effect but in
normal use it's not supposed to (and also syncs should only go to the OS
if they are needed as components keep a flag as to whether a raw update
has been done.
I'm a little surprised that tx and non-tx were the same speed in the
first place.
> by lock contention on the writes
For clarity:
TDB does not use 2-phase locking - it uses a write-behind log.
It does use java locks to protect key datastructures.
Andy
Re: concurrency and slower results
Posted by Andy Seaborne <an...@apache.org>.
On 13/06/12 18:16, Simon Helsen wrote:
> I have to correct my statement below, I was misreading my own numbers.
> The overall write time was about 3 times slower, not just 70%. This is
> comparing the latest build, once with and once without transactions
> (single user). The numbers (40%) I mentioned before came from 25
> concurrent users on a build using transactions from May 15 compared to a
> build using transactions from last weekend. That is not quite the same
> kind of comparison of course. However, I do recall (I don't have that on
> paper) that my own test suite with the May 15 build was running at a
> comparable speed whether I was using transactions or not. So I dare to
> conclude that write transactions are now considerable more expensive
> than around May 15
Is there some way we can turn this into a publicly usable example?
The recent changes to fix problems (JENA-250, JENA-252) might change the
concurrency but the changes also added a potential file system sync the
dataset when transitioning at first use transactionally. Depending on
how you use StoreConenction.release, that might have an effect but in
normal use it's not supposed to (and also syncs should only go to the OS
if they are needed as components keep a flag as to whether a raw update
has been done.
I'm a little surprised that tx and non-tx were the same speed in the
first place.
> by lock contention on the writes
For clarity:
TDB does not use 2-phase locking - it uses a write-behind log.
It does use java locks to protect key datastructures.
Andy
Re: concurrency and slower results
Posted by Simon Helsen <sh...@ca.ibm.com>.
I have to correct my statement below, I was misreading my own numbers. The
overall write time was about 3 times slower, not just 70%. This is
comparing the latest build, once with and once without transactions (single
user). The numbers (40%) I mentioned before came from 25 concurrent users
on a build using transactions from May 15 compared to a build using
transactions from last weekend. That is not quite the same kind of
comparison of course. However, I do recall (I don't have that on paper)
that my own test suite with the May 15 build was running at a comparable
speed whether I was using transactions or not. So I dare to conclude that
write transactions are now considerable more expensive than around May 15
Simon
|------------>
| From: |
|------------>
>--------------------------------------------------------------------------------------------------------------------------------------------------|
|Simon Helsen/Toronto/IBM |
>--------------------------------------------------------------------------------------------------------------------------------------------------|
|------------>
| To: |
|------------>
>--------------------------------------------------------------------------------------------------------------------------------------------------|
|dev@jena.apache.org |
>--------------------------------------------------------------------------------------------------------------------------------------------------|
|------------>
| Cc: |
|------------>
>--------------------------------------------------------------------------------------------------------------------------------------------------|
|dev@jena.apache.org |
>--------------------------------------------------------------------------------------------------------------------------------------------------|
|------------>
| Date: |
|------------>
>--------------------------------------------------------------------------------------------------------------------------------------------------|
|06/13/2012 01:08 PM |
>--------------------------------------------------------------------------------------------------------------------------------------------------|
|------------>
| Subject: |
|------------>
>--------------------------------------------------------------------------------------------------------------------------------------------------|
|Re: concurrency and slower results |
>--------------------------------------------------------------------------------------------------------------------------------------------------|
I was actually able to run a bunch of simpler tests with no concurrency and
compare the exact same implementation of our product with (tx) and without
transactions (vanilla) and it seems that the time is going to write
activities. In my suite, it took Tx about 70% longer to write the same
data. In these non concurrency tests, queries were running with roughly the
same speed. I assume that the slowdown on queries with 25 users is caused
by lock contention on the writes which are taking much longer for some
reason
Simon
|------------>
| From: |
|------------>
>--------------------------------------------------------------------------------------------------------------------------------------------------|
|Laurent Pellegrino <la...@gmail.com> |
>--------------------------------------------------------------------------------------------------------------------------------------------------|
|------------>
| To: |
|------------>
>--------------------------------------------------------------------------------------------------------------------------------------------------|
|dev@jena.apache.org |
>--------------------------------------------------------------------------------------------------------------------------------------------------|
|------------>
| Date: |
|------------>
>--------------------------------------------------------------------------------------------------------------------------------------------------|
|06/13/2012 11:10 AM |
>--------------------------------------------------------------------------------------------------------------------------------------------------|
|------------>
| Subject: |
|------------>
>--------------------------------------------------------------------------------------------------------------------------------------------------|
|Re: concurrency and slower results |
>--------------------------------------------------------------------------------------------------------------------------------------------------|
Hello,
I also noticed the same thing by updating jena-tdb only. I switched
from version 0.9.0-incubating to snapshots on 6 June and I noticed
around ~50% decrease on one of our benchmark which is read/write
intensive.
Kind Regards,
Laurent
On Wed, Jun 13, 2012 at 4:33 PM, Simon Helsen <sh...@ca.ibm.com> wrote:
> before you ask, of course, we are looking into whether this is something
> on our end. We only made very few changes, but we should be able to
> isolate this. I'll report on that once I know for sure. My question was
> more whether anyone knew of any changes in tx that could affect
concurrent
> performance negatively since May 15
>
> thanks
>
> Simon
>
>
>
>
>
> From:
> Simon Helsen/Toronto/IBM@IBMCA
> To:
> dev@jena.apache.org
> Date:
> 06/13/2012 10:09 AM
> Subject:
> concurrency and slower results
>
>
>
> hi everyone, Andy especially,
>
> it looks like some of the changes to TDB since May 15 have negatively
> affected the concurrent behavior of Tx. To clarify, we are doing some
> serious internal performance testing on a product which is based on
> Jena/TDB. We initially did these runs to compare Tx (2.7.1 snapshot of
May
>
> 15) with our old TDB implementation (which used a conservative locking
> model). The results were good, especially with long-running/expensive
> index update operations.
>
> Now, I had asked the performance team to rerun their tests with a new
> build of the product based on the release candidate (of last weekend) and
> they are reporting a concerning degradation. Compared to the May 15 TDB,
> they are reporting 12-15% decrease in query performance when doing a
> "light update load" and about 40% query and 40% update performance
> degradation when doing a "heave update load".
>
> That is quite a serious regression. The question is what changed since
May
>
> 15 that could have such a bad effect on performance. The tests included
> only 25 concurrent users on about 15 million triples. Yet, I wonder if
> some of the recent lock changes (see JENA-252) are responsible
>
> thanks
>
> Simon
>
>
>
Re: concurrency and slower results
Posted by Laurent Pellegrino <la...@gmail.com>.
Hello,
I also noticed the same thing by updating jena-tdb only. I switched
from version 0.9.0-incubating to snapshots on 6 June and I noticed
around ~50% decrease on one of our benchmark which is read/write
intensive.
Kind Regards,
Laurent
On Wed, Jun 13, 2012 at 4:33 PM, Simon Helsen <sh...@ca.ibm.com> wrote:
> before you ask, of course, we are looking into whether this is something
> on our end. We only made very few changes, but we should be able to
> isolate this. I'll report on that once I know for sure. My question was
> more whether anyone knew of any changes in tx that could affect concurrent
> performance negatively since May 15
>
> thanks
>
> Simon
>
>
>
>
>
> From:
> Simon Helsen/Toronto/IBM@IBMCA
> To:
> dev@jena.apache.org
> Date:
> 06/13/2012 10:09 AM
> Subject:
> concurrency and slower results
>
>
>
> hi everyone, Andy especially,
>
> it looks like some of the changes to TDB since May 15 have negatively
> affected the concurrent behavior of Tx. To clarify, we are doing some
> serious internal performance testing on a product which is based on
> Jena/TDB. We initially did these runs to compare Tx (2.7.1 snapshot of May
>
> 15) with our old TDB implementation (which used a conservative locking
> model). The results were good, especially with long-running/expensive
> index update operations.
>
> Now, I had asked the performance team to rerun their tests with a new
> build of the product based on the release candidate (of last weekend) and
> they are reporting a concerning degradation. Compared to the May 15 TDB,
> they are reporting 12-15% decrease in query performance when doing a
> "light update load" and about 40% query and 40% update performance
> degradation when doing a "heave update load".
>
> That is quite a serious regression. The question is what changed since May
>
> 15 that could have such a bad effect on performance. The tests included
> only 25 concurrent users on about 15 million triples. Yet, I wonder if
> some of the recent lock changes (see JENA-252) are responsible
>
> thanks
>
> Simon
>
>
>
Re: concurrency and slower results
Posted by Simon Helsen <sh...@ca.ibm.com>.
before you ask, of course, we are looking into whether this is something
on our end. We only made very few changes, but we should be able to
isolate this. I'll report on that once I know for sure. My question was
more whether anyone knew of any changes in tx that could affect concurrent
performance negatively since May 15
thanks
Simon
From:
Simon Helsen/Toronto/IBM@IBMCA
To:
dev@jena.apache.org
Date:
06/13/2012 10:09 AM
Subject:
concurrency and slower results
hi everyone, Andy especially,
it looks like some of the changes to TDB since May 15 have negatively
affected the concurrent behavior of Tx. To clarify, we are doing some
serious internal performance testing on a product which is based on
Jena/TDB. We initially did these runs to compare Tx (2.7.1 snapshot of May
15) with our old TDB implementation (which used a conservative locking
model). The results were good, especially with long-running/expensive
index update operations.
Now, I had asked the performance team to rerun their tests with a new
build of the product based on the release candidate (of last weekend) and
they are reporting a concerning degradation. Compared to the May 15 TDB,
they are reporting 12-15% decrease in query performance when doing a
"light update load" and about 40% query and 40% update performance
degradation when doing a "heave update load".
That is quite a serious regression. The question is what changed since May
15 that could have such a bad effect on performance. The tests included
only 25 concurrent users on about 15 million triples. Yet, I wonder if
some of the recent lock changes (see JENA-252) are responsible
thanks
Simon