testing using remote host

[Daniel John Debrunner <dj...@apache.org>]

Sorry that the changes to run useProcess under the security manager
broke the remote testing, I was not aware of the implications. Looking
at section 4.14 or the testing README, I couldn't understand why when
running in this mode useProcess=false is choosen. Why not continue to
run the clients as they did before?

Dan.

Re: testing using remote host

[Myrna van Lunteren <m....@gmail.com>]

After some (re)experimenting and dabbing in the permissions I am revisiting
this...

As I understand it, there is no way (would violate some basic security
concepts) for the networkserver to drop databases, or create subdirectories.
(correct?)

Thus, when running against a networkserver on another system/IP address, all
databases get created where the networkserver is running (possibly
redirected, but it is still a fixed location). Thus databases are reused. In
this, it is *like* running with useprocess false.
And yes, alas, this means a horrible mess catching up because test changes
leave stuff behind. Thus, as I said in email to the list in December, I
tried to make tests pass with useprocess=false as much as possible.

I think, also, that when I first ran with useprocess=false, various security
permissions error went away, and I thought that I was doing the right thing.
However, after I saw the securitymanager/useprocess=false checkin, I became
suspicious of that conclusion, and thus I had to revisit. I felt it was my
responsibility rather than simply complain.

I apologize for providing inaccurate information in my first reply on this
thread, I hadn't thought it all through, and had forgotten some of the
behaviour I had seen while experimenting in the month or so between my
initial check-in and the question starting this thread. So, no, there is no
problem on the serverside with permissions or creation of databases. Rather,
there were problems because of reusing of databases and security trouble on
the clientside.

The clientside securitymanager troubles are essentially resolved by making
the hostName test property automatically getting passed into the ${
derbyTesting.serverhost} (renamed from $csinfo.serverhost) property in
jvm.java which resolves the property in derby_tests.policy. (I did this in
my inital patch for DERBY-871).

Now, I would still like to pursue the useprocess=false angle a little.
In my thinking, it would kill 2 birds with one stone - speed up processing
and make every developer responsible for cleaning up their tests instead of
putting that burden on myself or whomever actually tries to test with remote
host. In theory everyone *should* cleanup the tests objects they create
anyways, so I didn't think it an unreasonable hardship. As you might have
noticed, I did take it on to get things a fair ways.

I had jdbc20 suite almost running with useprocess=false...except for
- jdbcapi/getCurConnJdbc20.java when running with db2jcc in the classpath.
   It hits:
  ERROR XJ001: Java exception: 'access denied ( java.lang.RuntimePermission
loadLibrary.db2jcct2): java.security.AccessControlException'
   (as I reported to the list).
   I think this problem will be resolved by granting db2jcc*.jar all.
I haven't tested this. But would it be ok? Is there a better approach?

- all tests run with either JCC or with test property securityManager=false
after an initial test has securityManager installed fail because the message
'security Manager not installed' pops up in the test output, instead of the
*.sum (or to the console).
  This is because it gets generated after the redirection to the stream
(from calling the method installManager.
  I understand from Dan's reply in this thread and re DERBY-871 that this
was because otherwise additional permissions would be needed. I never saw
errors relating to the stream, but then I was running with the classes -
could that make a difference? What would be the behavio(u)r if you'd get the
permission error? Maybe I saw it and misunderstood...
- still...would it be possible to save the installedSecurityManager state
and set things back for every test, i.e. uninstall security Manager somehow?
Would that be a slowing down things?

Myrna



>
>

Re: testing using remote host

[Daniel John Debrunner <dj...@apache.org>]

Kathey Marsden wrote:

> Daniel John Debrunner wrote:

>>I don't see why database creation would fail, the policy file has
>>permissions to create databases under ${derby.system.home} and ${user.dir}.
>>

> I think the network server knows nothing about the test it is running or
> what directories to create.
> Consider this URL.
> jdbc:derby:net://my.remotehost.com:1527/wombat;user=howardR;password=takeItEasy
> 
> Network server just has the derby.system.home it was started with and
> gets a database name "wombat", so all the tests that use that database
> name share a database.  In local mode network server is restarted for
> each test in the appropriate directory.

I think all that you say is true, but I don't see how it relates to not
having the permission to create a database. As far as I understand when
running the tests in the normal fashion a test will create its database in:

  ${derby.system.home} when useProcess not set (process mode)
  ${user.dir} when useProcess=false

Now in the process mode the value of derby.system.home changes for each
test (e.g. it's equal to ${user.dir}/insert when running the single test
lang/insert.sql), but that doesn't affect the permissions granted,
because they are granted relative to ${derby.system.home}.
The as far as Derby is concerned the location of the test wombat
database is the same in remote and process mode, it's
${derby.system.home}/wombat.

Dan.

Re: testing using remote host

[Kathey Marsden <km...@sbcglobal.net>]

Daniel John Debrunner wrote:

>The text in 4.14 scared me as it seems the setup for test when running
>in remote mode is very different to running normally, thus increasing
>the chance for some change to a test to break it in remote mode. Thus it
>may be a never ending battle to keep the remote tests running.
>
>  
>
It seems like all of the conditions are a function of either
restrictions on remote access or useprocess itself, e.g. the need for
database cleanup and the location of the extin files.   It seems like
ideally we would want those tests that run remotely to always run with
useprocess=false when running with network server (or embedded for that
manner). Then the needed test maintenance will be caught as a matter of 
normal development.
[snip]

>I don't see why database creation would fail, the policy file has
>permissions to create databases under ${derby.system.home} and ${user.dir}.
>
>  
>
I think the network server knows nothing about the test it is running or
what directories to create.
Consider this URL.
jdbc:derby:net://my.remotehost.com:1527/wombat;user=howardR;password=takeItEasy

Network server just has the derby.system.home it was started with and
gets a database name "wombat", so all the tests that use that database
name share a database.  In local mode network server is restarted for
each test in the appropriate directory.

Kathey

Re: testing using remote host

[Daniel John Debrunner <dj...@apache.org>]

The text in 4.14 scared me as it seems the setup for test when running
in remote mode is very different to running normally, thus increasing
the chance for some change to a test to break it in remote mode. Thus it
may be a never ending battle to keep the remote tests running.

Myrna van Lunteren wrote:

> The current test harness will try to create databases in subdirectories,
> with the name of the test. Also, the derby_tests.policy file gets copied
> to the correct test dir and accessed through there. I envisaged that
> without useprocess=false, creation of databases would have failed.

I don't see why database creation would fail, the policy file has
permissions to create databases under ${derby.system.home} and ${user.dir}.

> Even
> if we'd have directories created by hand beforehand, we'd still have a
> problem because the derby_tests.policy file normally gets copied to a
> proper place in the test directories and as this doesn't happen on the
> remote host, we'd have incorrect file permissions...

I don't understand what you are trying to say here. The location of the
policy file (I think) doesn't matter, as long as it is pointed to by
java.security.policy.


> Also, it seemed to me that as the actual network interaction was bigger,
> it made sense to speed up the process by having as much as possible
> reuse of databases; but that was more of a side-issue.

Right, that's a separate issue, if we could speed up tests by running
them with useProcess=false than that should be done independent of
running remote or not.

Thanks for the info.
Dan.

Re: testing using remote host

[Myrna van Lunteren <m....@gmail.com>]

The current test harness will try to create databases in subdirectories,
with the name of the test. Also, the derby_tests.policy file gets copied to
the correct test dir and accessed through there. I envisaged that without
useprocess=false, creation of databases would have failed. Even if we'd
have directories created by hand beforehand, we'd still have a problem
because the derby_tests.policy file normally gets copied to a proper place
in the test directories and as this doesn't happen on the remote host, we'd
have incorrect file permissions...
Also, it seemed to me that as the actual network interaction was bigger, it
made sense to speed up the process by having as much as possible reuse of
databases; but that was more of a side-issue.

I'll try to think of a way to explain that succinctly in the section 4.14 if
you think all this makes sense.

Myrna

On 1/29/06, Daniel John Debrunner <dj...@apache.org> wrote:
>
> Sorry that the changes to run useProcess under the security manager
> broke the remote testing, I was not aware of the implications. Looking
> at section 4.14 or the testing README, I couldn't understand why when
> running in this mode useProcess=false is choosen. Why not continue to
> run the clients as they did before?
>
> Dan.
>
>