You are viewing a plain text version of this content. The canonical link for it is here.
Posted to derby-dev@db.apache.org by Rick Hillegas <ri...@oracle.com> on 2011/04/26 14:48:56 UTC
10.8.1 testing
So far, the regression tests have not uncovered any showstoppers for the
release: http://wiki.apache.org/db-derby/TenEightOnePlatformTesting
It would be interesting to know the status of the other release tests
listed here: http://wiki.apache.org/db-derby/TenEightOneChecklist
Thanks,
-Rick
Re: 10.8.1 testing
Posted by Tiago Espinha <ti...@espinhas.net>.
Ok, I will try this myself tomorrow.
Off the top of my head, if I recall correctly, before the UTF8 CCSID,
conversions were being done byte by byte as each character was always
guaranteed to be 1 byte at the most. If memory serves well, this was
actually just a cast into char, which I would expect to be fairly
inexpensive in terms of CPU.
With the UTF8 CCSID, we rely on the getBytes() (again, off the top of my
head) to actually convert Strings to and from UTF8. I'm guessing this has
the potential to cause some slowdown. If that's the case though, then the
scenario is even darker as it will in fact affect both client AND server.
Tiago
On Wed, Apr 27, 2011 at 6:22 PM, Knut Anders Hatlen
<kn...@oracle.com>wrote:
> Tiago Espinha <ti...@espinhas.net> writes:
>
> > How exactly did you notice the performance change with DERBY-5068? I
> > would like to be able to replicate this myself. Was it just during
> > the test runs? Or do you have a concrete scenario where it's
> > noticeable?
>
> Hi Tiago,
>
> I've only seen this in the nightly performance regression tests. (That's
> the only place I've been looking for it so far...) The reported
> throughput from the tests didn't change much, that's why it wasn't
> noticed before, but the CPU per transaction seems to have increased on
> the client side.
>
> In all of these tests, the clients have lots of free CPU cycles, they're
> spending much of their time waiting on a socket read call, so an
> increase in CPU usage doesn't necessarily cause lower throughput.
>
> We may be able to get these tests to show a difference in throughput if
> we run the network server on the same machine as the clients. In that
> case, less time would be spent waiting on network I/O, and the clients
> would have to compete with the server process for the CPU cycles, so
> increased CPU should mean that the system is able to process fewer
> transactions per time unit.
>
> One quick way to test it would be to start a network server on localhost
> (with insane jars) and then run the test client with this command:
>
> java org.apache.derbyTesting.perf.clients.Runner \
> -load sr_select -driver org.apache.derby.jdbc.ClientDriver \
> -url 'jdbc:derby://localhost/db;create=true' -threads 10 -init
>
> We could try a couple times with 10.6.2.1 and 10.7.1.1 and see if they
> report different results.
>
> --
> Knut Anders
>
Re: 10.8.1 testing
Posted by Knut Anders Hatlen <kn...@oracle.com>.
Tiago Espinha <ti...@espinhas.net> writes:
> How exactly did you notice the performance change with DERBY-5068? I
> would like to be able to replicate this myself. Was it just during
> the test runs? Or do you have a concrete scenario where it's
> noticeable?
Hi Tiago,
I've only seen this in the nightly performance regression tests. (That's
the only place I've been looking for it so far...) The reported
throughput from the tests didn't change much, that's why it wasn't
noticed before, but the CPU per transaction seems to have increased on
the client side.
In all of these tests, the clients have lots of free CPU cycles, they're
spending much of their time waiting on a socket read call, so an
increase in CPU usage doesn't necessarily cause lower throughput.
We may be able to get these tests to show a difference in throughput if
we run the network server on the same machine as the clients. In that
case, less time would be spent waiting on network I/O, and the clients
would have to compete with the server process for the CPU cycles, so
increased CPU should mean that the system is able to process fewer
transactions per time unit.
One quick way to test it would be to start a network server on localhost
(with insane jars) and then run the test client with this command:
java org.apache.derbyTesting.perf.clients.Runner \
-load sr_select -driver org.apache.derby.jdbc.ClientDriver \
-url 'jdbc:derby://localhost/db;create=true' -threads 10 -init
We could try a couple times with 10.6.2.1 and 10.7.1.1 and see if they
report different results.
--
Knut Anders
Re: 10.8.1 testing
Posted by Tiago Espinha <ti...@espinhas.net>.
Hi Knut,
How exactly did you notice the performance change with DERBY-5068? I would
like to be able to replicate this myself. Was it just during the test runs?
Or do you have a concrete scenario where it's noticeable?
I'm very interested in finding the issue on this one and I'm available to
put some work into making it better. It's no good having a convenience
feature if it's slowing down Derby, I think.
@ Rick,
I would like to do some platform tests myself (it seems they've not yet been
ran on OS X 16.* on a x64 JVM) but unfortunately my CPU time lately is
precious for the research I'm doing :-( . Everything else looks good though!
Tiago
On Wed, Apr 27, 2011 at 12:43 PM, Knut Anders Hatlen <knut.hatlen@oracle.com
> wrote:
> Myrna van Lunteren <m....@gmail.com> writes:
>
> > I don't intend to do any performance tests at this point - is anyone
> > else doing any?
>
> I've been keeping an eye on the nightly performance tests on trunk
> (http://home.online.no/~olmsan/derby/perf/) and haven't noticed any big
> changes after 10.7. I'd expect the performance of 10.8 to be very close
> to trunk, but I've started some of the test clients on the RC just in
> case.
>
> The performance changes I have noticed lately are:
>
> 1) DERBY-5068 - higher CPU usage on the client after the introduction of
> UTF-8 CcsidManager. I'm planning to look into it to see where the extra
> CPU is spent. But this happened before 10.7.1.1 and shouldn't be a
> blocker for the 10.8 release.
>
> 2) The full table scans in the nightly test, as well as the TPC-B like
> transactions, seem to have been running slightly slower the last two
> months (see http://home.online.no/~olmsan/derby/perf/tpcb_1y.html and
> http://home.online.no/~olmsan/derby/perf/select_sec_noindex_1y.html). It
> may be worth looking into if this is an actual degradation or just
> noise. But even if it turns out that it is a real performance
> degradation, it appears to be so small that it wouldn't be worth
> blocking the release.
>
> --
> Knut Anders
>
Re: 10.8.1 testing
Posted by Knut Anders Hatlen <kn...@oracle.com>.
Myrna van Lunteren <m....@gmail.com> writes:
> I don't intend to do any performance tests at this point - is anyone
> else doing any?
I've been keeping an eye on the nightly performance tests on trunk
(http://home.online.no/~olmsan/derby/perf/) and haven't noticed any big
changes after 10.7. I'd expect the performance of 10.8 to be very close
to trunk, but I've started some of the test clients on the RC just in
case.
The performance changes I have noticed lately are:
1) DERBY-5068 - higher CPU usage on the client after the introduction of
UTF-8 CcsidManager. I'm planning to look into it to see where the extra
CPU is spent. But this happened before 10.7.1.1 and shouldn't be a
blocker for the 10.8 release.
2) The full table scans in the nightly test, as well as the TPC-B like
transactions, seem to have been running slightly slower the last two
months (see http://home.online.no/~olmsan/derby/perf/tpcb_1y.html and
http://home.online.no/~olmsan/derby/perf/select_sec_noindex_1y.html). It
may be worth looking into if this is an actual degradation or just
noise. But even if it turns out that it is a real performance
degradation, it appears to be so small that it wouldn't be worth
blocking the release.
--
Knut Anders
Re: 10.8.1 testing
Posted by Rick Hillegas <ri...@oracle.com>.
On 4/28/11 2:36 PM, Myrna van Lunteren wrote:
> On Tue, Apr 26, 2011 at 10:52 AM, Rick Hillegas
> <ri...@oracle.com> wrote:
>> On 4/26/11 10:38 AM, Myrna van Lunteren wrote:
>>> On Tue, Apr 26, 2011 at 5:48 AM, Rick Hillegas<ri...@oracle.com>
>>> wrote:
>>>> So far, the regression tests have not uncovered any showstoppers for the
>>>> release: http://wiki.apache.org/db-derby/TenEightOnePlatformTesting
>>>>
>>>> It would be interesting to know the status of the other release tests
>>>> listed
>>>> here: http://wiki.apache.org/db-derby/TenEightOneChecklist
>>>>
...
> I noticed only now (...?) that there's also ODBC testing on this list.
> I don't know of any ODBC driver currently officially supporting derby;
> perhaps this should come off the list? Or did I miss something?
>
> Myrna
>
Hi Myrna,
I don't know what this means either. Maybe some testing of the ODBC
metadata procedures beyond what's done by DatabaseMetaDataTest? There
are still a number of ODBC drivers listed on the Uses of Derby page (
http://wiki.apache.org/db-derby/UsesOfDerby#Products_By_Type ), but none
of those companies participate in our release testing. I am puzzled too.
Thanks,
-Rick
Re: 10.8.1 testing
Posted by Myrna van Lunteren <m....@gmail.com>.
On Tue, Apr 26, 2011 at 10:52 AM, Rick Hillegas
<ri...@oracle.com> wrote:
> On 4/26/11 10:38 AM, Myrna van Lunteren wrote:
>>
>> On Tue, Apr 26, 2011 at 5:48 AM, Rick Hillegas<ri...@oracle.com>
>> wrote:
>>>
>>> So far, the regression tests have not uncovered any showstoppers for the
>>> release: http://wiki.apache.org/db-derby/TenEightOnePlatformTesting
>>>
>>> It would be interesting to know the status of the other release tests
>>> listed
>>> here: http://wiki.apache.org/db-derby/TenEightOneChecklist
>>>
>>> Thanks,
>>> -Rick
>>>
>> I've got 5 long running tests in progress, so far so good (or at
>> least, no worse than the previous runs for 10.7* and 10.8 release
>> candidates). I'll update the platform testing page with these results
>> in 2 days. I also kicked off the largeData test suite with ibm 1.6.
>> And I'll do the eclipse plugin testing in the next few days.
>>
>> I don't intend to do any performance tests at this point - is anyone
>> else doing any?
>>
>> I'd like to see the j2EE CTS tests get added to the vetting list - I'm
>> glad to see Knut already took care of them for RC3 (Thanks Knut!).
>
> Thanks, Myrna. I have added this item to the 10.8.1 checklist and noted that
> Knut ran the tests. We can use the 10.8.1 checklist as a template for the
> next release.
>
> Thanks,
> -Rick
>>
>> Myrna
>>
>
>
I noticed only now (...?) that there's also ODBC testing on this list.
I don't know of any ODBC driver currently officially supporting derby;
perhaps this should come off the list? Or did I miss something?
Myrna
Re: 10.8.1 testing
Posted by Rick Hillegas <ri...@oracle.com>.
On 4/26/11 10:38 AM, Myrna van Lunteren wrote:
> On Tue, Apr 26, 2011 at 5:48 AM, Rick Hillegas<ri...@oracle.com> wrote:
>> So far, the regression tests have not uncovered any showstoppers for the
>> release: http://wiki.apache.org/db-derby/TenEightOnePlatformTesting
>>
>> It would be interesting to know the status of the other release tests listed
>> here: http://wiki.apache.org/db-derby/TenEightOneChecklist
>>
>> Thanks,
>> -Rick
>>
> I've got 5 long running tests in progress, so far so good (or at
> least, no worse than the previous runs for 10.7* and 10.8 release
> candidates). I'll update the platform testing page with these results
> in 2 days. I also kicked off the largeData test suite with ibm 1.6.
> And I'll do the eclipse plugin testing in the next few days.
>
> I don't intend to do any performance tests at this point - is anyone
> else doing any?
>
> I'd like to see the j2EE CTS tests get added to the vetting list - I'm
> glad to see Knut already took care of them for RC3 (Thanks Knut!).
Thanks, Myrna. I have added this item to the 10.8.1 checklist and noted
that Knut ran the tests. We can use the 10.8.1 checklist as a template
for the next release.
Thanks,
-Rick
> Myrna
>
Re: 10.8.1 testing
Posted by Myrna van Lunteren <m....@gmail.com>.
On Tue, Apr 26, 2011 at 5:48 AM, Rick Hillegas <ri...@oracle.com> wrote:
> So far, the regression tests have not uncovered any showstoppers for the
> release: http://wiki.apache.org/db-derby/TenEightOnePlatformTesting
>
> It would be interesting to know the status of the other release tests listed
> here: http://wiki.apache.org/db-derby/TenEightOneChecklist
>
> Thanks,
> -Rick
>
I've got 5 long running tests in progress, so far so good (or at
least, no worse than the previous runs for 10.7* and 10.8 release
candidates). I'll update the platform testing page with these results
in 2 days. I also kicked off the largeData test suite with ibm 1.6.
And I'll do the eclipse plugin testing in the next few days.
I don't intend to do any performance tests at this point - is anyone
else doing any?
I'd like to see the j2EE CTS tests get added to the vetting list - I'm
glad to see Knut already took care of them for RC3 (Thanks Knut!).
Myrna