Derbyall runtimes, 10.1, and Security Manager

[Bryan Pendleton <bp...@amberpoint.com>]

I had occasion recently to be porting a few bug fixes from the
trunk to 10.1, and so I happened to be running derbyall on 10.1.

I don't really want to re-ignite the debate over derbyall runtime,
but the difference in duration between a derbyall run on the
trunk, and a derbyall run on 10.1, was really remarkable.

Then, as part of working on DERBY-1229, I happened to be running
a lot of interactive experiments both with and without the
security manager. And I noticed that when I ran Derby code with
the security manager enabled, the runtime speed was noticeably
slower.

So I'm wondering, is it possible that a significant portion of
the derbyall slowdown in the trunk is due to running with the
security manager enabled, and, if so, is there anything we can
do with that knowledge?

thanks,

bryan

Re: Derbyall runtimes, 10.1, and Security Manager

[Daniel John Debrunner <dj...@apache.org>]

Daniel John Debrunner wrote:


> 
> Hmmm, it suprises me that the SecurityManager would have that much
> effect. I'll try running some tests and see what exactly is happening.

I did some testing of this today. There is a slowdown due the security
manager and it's not due to additional code in the test harness.

Disabling the security manager by either of:

 - forcing the noSecurityManager=true for every test
 - setting Xjava.security.manager instead of the real property

had similar speed-ups.

The overhead was suprisingly large from around 14-16mins to 12mins to
for a set of ~40 tests from derbylang. Some guesses are:

 - tests have a lot of compilation, leading to a lot of class loading
which may have a higher cost with a security manager
 - tests create & drop tables, indexes etc. which leads to files being
created and dropped, all requiring security manager checks
 - harness/derby startup reads a lot of system properties requiring
security checks
 - jvm setup with a security manager has a large overhead - would
disappaer if we ran all tests in the same jvm with junit
   (will test theory with nist suite)

Dan.

Re: Derbyall runtimes, 10.1, and Security Manager

[Daniel John Debrunner <dj...@apache.org>]

Rick Hillegas wrote:
> Hi Bryan,
> 
> When playing around with individual tests--enabling and disabling the
> SecurityManager--I have noticed that our tests run considerably slower
> when launched under the SecurityManager. I don't have any sense of how
> much of the problem is just a tax we have to pay for security. Sounds
> like your experiments may have confirmed that the problem is not
> isolated to our test environment. It's definitely worth profiling this
> drag so that
> 
> 1) we can factor security calls to the outer loop
> 2) we can appropriately set our customers' expectations

Hmmm, it suprises me that the SecurityManager would have that much
effect. I'll try running some tests and see what exactly is happening.

Dan.

Re: Derbyall runtimes, 10.1, and Security Manager

["David W. Van Couvering" <Da...@Sun.COM>]

Wouldn't it be reasonable for developers to run without security manager 
and for the regression tests to run with?  We are not expecting regular 
breaks of security with our checkins, are we (although it can and does 
occur)?

Thanks,

David

Rick Hillegas wrote:
> Hi Bryan,
> 
> When playing around with individual tests--enabling and disabling the 
> SecurityManager--I have noticed that our tests run considerably slower 
> when launched under the SecurityManager. I don't have any sense of how 
> much of the problem is just a tax we have to pay for security. Sounds 
> like your experiments may have confirmed that the problem is not 
> isolated to our test environment. It's definitely worth profiling this 
> drag so that
> 
> 1) we can factor security calls to the outer loop
> 2) we can appropriately set our customers' expectations
> 
> Regards,
> -Rick
> 
> Bryan Pendleton wrote:
> 
>> I had occasion recently to be porting a few bug fixes from the
>> trunk to 10.1, and so I happened to be running derbyall on 10.1.
>>
>> I don't really want to re-ignite the debate over derbyall runtime,
>> but the difference in duration between a derbyall run on the
>> trunk, and a derbyall run on 10.1, was really remarkable.
>>
>> Then, as part of working on DERBY-1229, I happened to be running
>> a lot of interactive experiments both with and without the
>> security manager. And I noticed that when I ran Derby code with
>> the security manager enabled, the runtime speed was noticeably
>> slower.
>>
>> So I'm wondering, is it possible that a significant portion of
>> the derbyall slowdown in the trunk is due to running with the
>> security manager enabled, and, if so, is there anything we can
>> do with that knowledge?
>>
>> thanks,
>>
>> bryan
>>
> 

Re: Derbyall runtimes, 10.1, and Security Manager

[Daniel John Debrunner <dj...@apache.org>]

Rick Hillegas wrote:

> Hi Bryan,
> 
> When playing around with individual tests--enabling and disabling the
> SecurityManager--I have noticed that our tests run considerably slower
> when launched under the SecurityManager. I don't have any sense of how
> much of the problem is just a tax we have to pay for security. Sounds
> like your experiments may have confirmed that the problem is not
> isolated to our test environment. It's definitely worth profiling this
> drag so that
> 
> 1) we can factor security calls to the outer loop
> 2) we can appropriately set our customers' expectations

I agree it's worth running benchmarks with and without the security
manager. I would not take the tests and test harness slowdown as any
real indication of application performance. Most applications will reach
a steady state where they do not make calls that require a JVM security
check, e.g. reading a file does not, only opening it.

Dan.

Re: Derbyall runtimes, 10.1, and Security Manager

[Rick Hillegas <Ri...@Sun.COM>]

Hi Bryan,

When playing around with individual tests--enabling and disabling the 
SecurityManager--I have noticed that our tests run considerably slower 
when launched under the SecurityManager. I don't have any sense of how 
much of the problem is just a tax we have to pay for security. Sounds 
like your experiments may have confirmed that the problem is not 
isolated to our test environment. It's definitely worth profiling this 
drag so that

1) we can factor security calls to the outer loop
2) we can appropriately set our customers' expectations

Regards,
-Rick

Bryan Pendleton wrote:

> I had occasion recently to be porting a few bug fixes from the
> trunk to 10.1, and so I happened to be running derbyall on 10.1.
>
> I don't really want to re-ignite the debate over derbyall runtime,
> but the difference in duration between a derbyall run on the
> trunk, and a derbyall run on 10.1, was really remarkable.
>
> Then, as part of working on DERBY-1229, I happened to be running
> a lot of interactive experiments both with and without the
> security manager. And I noticed that when I ran Derby code with
> the security manager enabled, the runtime speed was noticeably
> slower.
>
> So I'm wondering, is it possible that a significant portion of
> the derbyall slowdown in the trunk is due to running with the
> security manager enabled, and, if so, is there anything we can
> do with that knowledge?
>
> thanks,
>
> bryan
>