You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@river.apache.org by Peter Firmstone <ji...@zeus.net.au> on 2012/08/19 14:09:28 UTC
Develop new spec for RMIClassLoader replacement
As you know Gregg's contributed his CodebaseAccessClassLoader
alternative to RMIClassLoader.
I figure we need to review MarshalInputStream and MarshalOutputStream, a
final interface for CodebaseAccessClassLoader and expected behaviour of
Class loading.
I'm also considering class loading for classes other than preferred.
You might have other ideas and opinions, please speak up.
How do we find a suitable parent class loader?
1. In a modular system, ClassLoaders don't necessarily follow a tree
hierarchy.
2. If the class belongs to a smart proxy, the parent ClassLoader is
the ClassLoader that contains the Service API.
1. The smart proxy may contain several classes.
2. Smart proxy classes should generally be preferred.
3. If we know the name of the service API interface, we can find the
parent ClassLoader in any system.
1. Annotate all smart proxy classes with the Service API
interface name (in addition to the codebase annotation).
2. Only annotate proxy classes with the proxy Codebase annotation.
3. During marshaling annotate the stream with the Service API
interface name and whether the class is preferred (this
removes the need to download the preferred list from
codebase URL to confirm if the class is preferred or not.
4. During unmarshalling, walk the call stack, find the first
non system ClassLoader and call findClass(String name), keep
traversing the stack until the Service API class is found.
5. The ClassLoader of the Service API is the parent
ClassLoader, this will work as expected in any modular system.
4. Having found the parent ClassLoader we can now lookup our Map to
find if a PreferredClassLoader has already been assigned to the
unmarshaling object class. This eliminates the need to use the
problematic Thread context ClassLoader, which may not be set or
may be set incorrectly.
Lacking codebase annotations, ObjectInputStream usually just tries to
resolve classes using the first non null ClassLoader in the call stack
(little more to it than that see the spec for details). Standard
serialisation doesn't play well with OSGi, in fact exceptions are
commonplace during deserialisation with OSGi, there is an easy way to
solve this, also annotate the package version, with this information, we
can find the correct classloader in a system where more than one
ClassLoader may contain different versions of the same classes. In a
system where only one version of a package can be found and this package
uses a different version, we'll have to accept it, or provide another
way for the client to resolve the package, this needs to be a standard
interface that can be implemented at the client (or server for objects
traveling the other way). Then maven, osgi or some other means can be
used to resolve the missing dependency.
Thoughts?
Cheers,
Peter.
Serialisation and Modularity
Posted by Peter Firmstone <ji...@zeus.net.au>.
This is relevant to OSGi and jboss as well as Jigsaw, now delayed to Java 9.
https://community.jboss.org/wiki/ModularSerialization
I strongly suggest anyone interested in River read this.
Regards,
Peter.
Re: Serialisation and Modularity
Posted by Peter Firmstone <ji...@zeus.net.au>.
Dan, I like having you around, your a good detective, always picking up
on important details.
Yes, this could mean that an object (and all other serializable fields
it contains) may have an equal serialized form, but be unequal in object
form after unmarshalling.
Object unmarshalled equality would be determined by the implementor of
CodebaseAccessClassLoader. We need to provide a default implementation,
with similar behaviour to PreferredClassProvider.
The question is of course, what to do in the case of version conflicts?
1. Use an already downloaded earlier version (with identical codebase
annotations when present)?
2. Ask the local environment to decide if it wants to use the latest
codebase, or try using the local codebase instead? (after
checking the serial version UID is compatible). This allows the
client to force object equality if it's required. This couldn't
be set in a modular environment, in that case the client would
have to live with inequality.
3. Another option might be to use reflective proxy's to refer to
smart proxy instances and keep a weak register of smart proxy's
and their version information and codebase annotations (a @proxy
annotation might be required to help us identify the smart proxy),
and persist all smart proxy's that share the codebase to disk and
reload them with the new codebase version, again allowing the
client to decide if object equality should be enforced, as this
wouldn't be free, the update operation would have to block. We'd
also have to make sure the local reflective proxy was replaced by
the smart proxy in any object streams. Only after all local smart
proxy's are updated is is possible to check for boomerangs. If
object equality is not enforced, then the newer version would be
loaded into a different ClassLoader and not be equal, this
includes boomerangs.
4. A class might not belong to a proxy, other classes are transmitted
in the object tree, these may be platform classes or
implementation classes. So we need a way for the client
environment to resolve a class if we can't work it out. For
example maven codebase provisioning or OSGi etc. This is where it
might be useful to provide a CodebaseAccessClassLoader
implementation with a method the client can override, to resolve
classes our implementation can't, after we first attempt to but
can't find the right class, since most clients don't want to
reinvent the wheel, they can accept the default behaviour and
override it for corner cases.
This doesn't look as hard to solve as it used to.
I think I'm leaning towards 3 and 4, although this would cause some
backward compatibility issues, developers would have to recompile with
@proxy annotations. Remember the smart proxy's public API will likely
not implement Remote, unless someone's got a better idea?
So the client can fully implement CodebaseAccessClassLoader if they so
desire, or they can use a default implementation that does most of the
leg work and only asks for help when it gets stuck, reducing the
implementation burden. If a developer is in charge of both the service
and the clients, then they can develop their own object stream
properties they can utilise in this situation. I think if our
implementation detects stream properties it doesn't understand, then it
should probably delegate first to the protected method overridable by
the user developer before attempting to resolve the class itself.
Cheers,
Peter.
On 23/08/2012 6:43 PM, Dan Creswell wrote:
> I haven't thought about this in any depth at all so I'm unsure whether this
> matters but:
>
> JavaSpace implementations have, obviously, an equality rule for the fields
> in the Entry's they handle. The basics of this are that content is compared
> whilst codebase annotations etc are excluded.
>
> The act of splitting out the annotations and so on is trivial but dependent
> upon a good understanding of codebases, serialization etc.
>
> And I would expect that changes such as the below will impact both space
> matching and (de)serializing of Entry's given the above.
>
> We'd best account for the impact there in this design effort?
>
> On 23 August 2012 08:46, Peter Firmstone<ji...@zeus.net.au> wrote:
>
>> Yes, however the ObjectStream used, also annotates other information, like
>> superclasses and version metadata to ensure the correct ClassLoader is
>> selected locally.
>>
>> Object Streams are not easily changed, MarshalOutputStream annotates a
>> codebase string, however this could be much more flexible, if we could
>> annotate a second string that contained key value pairs separated by pipe
>> symbols.
>>
>> This would allow any future changes to be easily accommodated, such as
>> version information and superclasses, this is similar to system properties.
>>
>> CodebaseAccessClassLoader could be passed a Map<String,String> that
>> contains additional information. Then we can standardise certain
>> properties that can be defined along with codebase annotations. This will
>> assist to accommodate future modular environments.
>>
>> We need to increment the stream version, then continue to use the old
>> stream for now, however when the time comes, the existing releases will be
>> compatible with the new stream version, so we could see a few releases
>> before we need the ability to use it.
>>
>> So a MarshalOutputStream would utilise the old stream version by default
>> for a few releases, while MarshalInputStream is capable of reading both
>> stream versions. Then at some time in the future, we set
>> MarshalOutputStream to use the new stream version by default and users have
>> to set a property if they want MarshalOutputStream to revert back to the
>> existing protocol.
>>
>> Obviously the stream properties will have to be standardised over time,
>> but it will enable us to do so without again needing to revise the stream
>> protocol, then CodebaseAccessClassLoader doesn't need to change again
>> either.
>>
>> Regards,
>>
>> Peter.
>>
>>
>> On 23/08/2012 1:17 AM, GREGG WONDERLY wrote:
>>
>>> So a quick look at this, tells me that the JBoss resolver is position
>>> with the duties that the CodebaseClassAccess interface is at. It allows
>>> the resolution of classes to be resolved to a class loader which is
>>> appropriate for the application environment.
>>>
>>> Gregg
>>>
>>> On Aug 22, 2012, at 7:36 AM, Peter Firmstone<ji...@zeus.net.au> wrote:
>>>
>>> This is relevant to OSGi and jboss as well as Jigsaw, now delayed to
>>>> Java 9.
>>>>
>>>> https://community.jboss.org/**wiki/ModularSerialization<https://community.jboss.org/wiki/ModularSerialization>
>>>>
>>>> I strongly suggest anyone interested in River read this.
>>>>
>>>> Regards,
>>>>
>>>> Peter.
>>>>
>>>>
Re: Serialisation and Modularity
Posted by Dan Creswell <da...@gmail.com>.
I haven't thought about this in any depth at all so I'm unsure whether this
matters but:
JavaSpace implementations have, obviously, an equality rule for the fields
in the Entry's they handle. The basics of this are that content is compared
whilst codebase annotations etc are excluded.
The act of splitting out the annotations and so on is trivial but dependent
upon a good understanding of codebases, serialization etc.
And I would expect that changes such as the below will impact both space
matching and (de)serializing of Entry's given the above.
We'd best account for the impact there in this design effort?
On 23 August 2012 08:46, Peter Firmstone <ji...@zeus.net.au> wrote:
> Yes, however the ObjectStream used, also annotates other information, like
> superclasses and version metadata to ensure the correct ClassLoader is
> selected locally.
>
> Object Streams are not easily changed, MarshalOutputStream annotates a
> codebase string, however this could be much more flexible, if we could
> annotate a second string that contained key value pairs separated by pipe
> symbols.
>
> This would allow any future changes to be easily accommodated, such as
> version information and superclasses, this is similar to system properties.
>
> CodebaseAccessClassLoader could be passed a Map<String,String> that
> contains additional information. Then we can standardise certain
> properties that can be defined along with codebase annotations. This will
> assist to accommodate future modular environments.
>
> We need to increment the stream version, then continue to use the old
> stream for now, however when the time comes, the existing releases will be
> compatible with the new stream version, so we could see a few releases
> before we need the ability to use it.
>
> So a MarshalOutputStream would utilise the old stream version by default
> for a few releases, while MarshalInputStream is capable of reading both
> stream versions. Then at some time in the future, we set
> MarshalOutputStream to use the new stream version by default and users have
> to set a property if they want MarshalOutputStream to revert back to the
> existing protocol.
>
> Obviously the stream properties will have to be standardised over time,
> but it will enable us to do so without again needing to revise the stream
> protocol, then CodebaseAccessClassLoader doesn't need to change again
> either.
>
> Regards,
>
> Peter.
>
>
> On 23/08/2012 1:17 AM, GREGG WONDERLY wrote:
>
>> So a quick look at this, tells me that the JBoss resolver is position
>> with the duties that the CodebaseClassAccess interface is at. It allows
>> the resolution of classes to be resolved to a class loader which is
>> appropriate for the application environment.
>>
>> Gregg
>>
>> On Aug 22, 2012, at 7:36 AM, Peter Firmstone<ji...@zeus.net.au> wrote:
>>
>> This is relevant to OSGi and jboss as well as Jigsaw, now delayed to
>>> Java 9.
>>>
>>> https://community.jboss.org/**wiki/ModularSerialization<https://community.jboss.org/wiki/ModularSerialization>
>>>
>>> I strongly suggest anyone interested in River read this.
>>>
>>> Regards,
>>>
>>> Peter.
>>>
>>>
>
Re: Serialisation and Modularity
Posted by Peter Firmstone <ji...@zeus.net.au>.
Yes, however the ObjectStream used, also annotates other information,
like superclasses and version metadata to ensure the correct ClassLoader
is selected locally.
Object Streams are not easily changed, MarshalOutputStream annotates a
codebase string, however this could be much more flexible, if we could
annotate a second string that contained key value pairs separated by
pipe symbols.
This would allow any future changes to be easily accommodated, such as
version information and superclasses, this is similar to system properties.
CodebaseAccessClassLoader could be passed a Map<String,String> that
contains additional information. Then we can standardise certain
properties that can be defined along with codebase annotations. This
will assist to accommodate future modular environments.
We need to increment the stream version, then continue to use the old
stream for now, however when the time comes, the existing releases will
be compatible with the new stream version, so we could see a few
releases before we need the ability to use it.
So a MarshalOutputStream would utilise the old stream version by default
for a few releases, while MarshalInputStream is capable of reading both
stream versions. Then at some time in the future, we set
MarshalOutputStream to use the new stream version by default and users
have to set a property if they want MarshalOutputStream to revert back
to the existing protocol.
Obviously the stream properties will have to be standardised over time,
but it will enable us to do so without again needing to revise the
stream protocol, then CodebaseAccessClassLoader doesn't need to change
again either.
Regards,
Peter.
On 23/08/2012 1:17 AM, GREGG WONDERLY wrote:
> So a quick look at this, tells me that the JBoss resolver is position with the duties that the CodebaseClassAccess interface is at. It allows the resolution of classes to be resolved to a class loader which is appropriate for the application environment.
>
> Gregg
>
> On Aug 22, 2012, at 7:36 AM, Peter Firmstone<ji...@zeus.net.au> wrote:
>
>> This is relevant to OSGi and jboss as well as Jigsaw, now delayed to Java 9.
>>
>> https://community.jboss.org/wiki/ModularSerialization
>>
>> I strongly suggest anyone interested in River read this.
>>
>> Regards,
>>
>> Peter.
>>
Re: Serialisation and Modularity
Posted by GREGG WONDERLY <gr...@gmail.com>.
So a quick look at this, tells me that the JBoss resolver is position with the duties that the CodebaseClassAccess interface is at. It allows the resolution of classes to be resolved to a class loader which is appropriate for the application environment.
Gregg
On Aug 22, 2012, at 7:36 AM, Peter Firmstone <ji...@zeus.net.au> wrote:
> This is relevant to OSGi and jboss as well as Jigsaw, now delayed to Java 9.
>
> https://community.jboss.org/wiki/ModularSerialization
>
> I strongly suggest anyone interested in River read this.
>
> Regards,
>
> Peter.
>
Serialisation and Modularity
Posted by Peter Firmstone <ji...@zeus.net.au>.
This is relevant to OSGi and jboss as well as Jigsaw, now delayed to Java 9.
https://community.jboss.org/wiki/ModularSerialization
I strongly suggest anyone interested in River read this.
Regards,
Peter.
Re: Develop new spec for RMIClassLoader replacement
Posted by Peter Firmstone <ji...@zeus.net.au>.
On 20/08/2012 10:07 AM, GREGG WONDERLY wrote:
> The point of my changes was to make it possible for arbitrary behaviors. What you describe below, could be done as a "version" of the SPI.
Thanks Gregg, glad you're still participating.
That's my intent, to determine what, if anything else, is required by
other SPI implementations at this moment, now that RMIClassLoader has
come up for replacement. (I have literally replaced allmost all
instances of it in my development env with CodebaseAccessClassLoader,
including all the qa tests).
By annotating some other information about the originating classes, we
could provide more information to CodebaseAccessClassLoader, there are
two ways of doing so, the first would be to create a new URL scheme and
handler that included it in URL strings, but this is too much like a
hack. (you can include version information as part of the codebase name
now, although this won't suit all environments).
The other option would be the increment the stream version, add the
possiblility of annotating additional information into the stream, the
stream version in use in an environment could be set as an envirionment
variable.
The difficulity is of course working out the minimum information
required in a modular environment, which could include OSGi or Java 9
for instance.
We could allow an additional string, that has a list of comma separated
key value pairs, allowing implementors to pass additional information
through the stream.
The netbeans model is interesting, it basically means you can walk the
call stack, just like ObjectInputStream, although at present we choose
to use the Thread context ClassLoader, which causes breakage, but
walking the call stack won't.
Check out net.jini.security.Security if you need an example of stack
inspection.
I need a little more time figure it out properly, then I'll come back
with some further thoughts / suggestions.
Cheers,
Peter.
> The modular programming systems like OSGi and Netbeans etc., often have differing needs, depending on what you want to expose to "who", and those things might change over time, or depending on users.
>
> Netbeans modules, in particular have a hierarchical structure based on declared dependencies. This can mean that two modules might resolve a class differently, depending on what he developer declares. So, it can be necessary to do some interesting things. The Netbeans system will largely provide the right structure for you, so that you can use the "current class" in a particular module's class loader as the "parent" for proxy resolution.
>
> Gregg Wonderly
>
> On Aug 19, 2012, at 7:09 AM, Peter Firmstone<ji...@zeus.net.au> wrote:
>
>> As you know Gregg's contributed his CodebaseAccessClassLoader alternative to RMIClassLoader.
>>
>> I figure we need to review MarshalInputStream and MarshalOutputStream, a final interface for CodebaseAccessClassLoader and expected behaviour of Class loading.
>>
>> I'm also considering class loading for classes other than preferred.
>>
>> You might have other ideas and opinions, please speak up.
>>
>> How do we find a suitable parent class loader?
>>
>> 1. In a modular system, ClassLoaders don't necessarily follow a tree
>> hierarchy.
>> 2. If the class belongs to a smart proxy, the parent ClassLoader is
>> the ClassLoader that contains the Service API.
>> 1. The smart proxy may contain several classes.
>> 2. Smart proxy classes should generally be preferred.
>>
>> 3. If we know the name of the service API interface, we can find the
>> parent ClassLoader in any system.
>> 1. Annotate all smart proxy classes with the Service API
>> interface name (in addition to the codebase annotation).
>> 2. Only annotate proxy classes with the proxy Codebase annotation.
>> 3. During marshaling annotate the stream with the Service API
>> interface name and whether the class is preferred (this
>> removes the need to download the preferred list from
>> codebase URL to confirm if the class is preferred or not.
>> 4. During unmarshalling, walk the call stack, find the first
>> non system ClassLoader and call findClass(String name), keep
>> traversing the stack until the Service API class is found.
>> 5. The ClassLoader of the Service API is the parent
>> ClassLoader, this will work as expected in any modular system.
>> 4. Having found the parent ClassLoader we can now lookup our Map to
>> find if a PreferredClassLoader has already been assigned to the
>> unmarshaling object class. This eliminates the need to use the
>> problematic Thread context ClassLoader, which may not be set or
>> may be set incorrectly.
>>
>> Lacking codebase annotations, ObjectInputStream usually just tries to resolve classes using the first non null ClassLoader in the call stack (little more to it than that see the spec for details). Standard serialisation doesn't play well with OSGi, in fact exceptions are commonplace during deserialisation with OSGi, there is an easy way to solve this, also annotate the package version, with this information, we can find the correct classloader in a system where more than one ClassLoader may contain different versions of the same classes. In a system where only one version of a package can be found and this package uses a different version, we'll have to accept it, or provide another way for the client to resolve the package, this needs to be a standard interface that can be implemented at the client (or server for objects traveling the other way). Then maven, osgi or some other means can be used to resolve the missing dependency.
>>
>> Thoughts?
>>
>> Cheers,
>>
>> Peter.
>>
>>
Re: Develop new spec for RMIClassLoader replacement
Posted by GREGG WONDERLY <gr...@gmail.com>.
The point of my changes was to make it possible for arbitrary behaviors. What you describe below, could be done as a "version" of the SPI. The modular programming systems like OSGi and Netbeans etc., often have differing needs, depending on what you want to expose to "who", and those things might change over time, or depending on users.
Netbeans modules, in particular have a hierarchical structure based on declared dependencies. This can mean that two modules might resolve a class differently, depending on what he developer declares. So, it can be necessary to do some interesting things. The Netbeans system will largely provide the right structure for you, so that you can use the "current class" in a particular module's class loader as the "parent" for proxy resolution.
Gregg Wonderly
On Aug 19, 2012, at 7:09 AM, Peter Firmstone <ji...@zeus.net.au> wrote:
> As you know Gregg's contributed his CodebaseAccessClassLoader alternative to RMIClassLoader.
>
> I figure we need to review MarshalInputStream and MarshalOutputStream, a final interface for CodebaseAccessClassLoader and expected behaviour of Class loading.
>
> I'm also considering class loading for classes other than preferred.
>
> You might have other ideas and opinions, please speak up.
>
> How do we find a suitable parent class loader?
>
> 1. In a modular system, ClassLoaders don't necessarily follow a tree
> hierarchy.
> 2. If the class belongs to a smart proxy, the parent ClassLoader is
> the ClassLoader that contains the Service API.
> 1. The smart proxy may contain several classes.
> 2. Smart proxy classes should generally be preferred.
>
> 3. If we know the name of the service API interface, we can find the
> parent ClassLoader in any system.
> 1. Annotate all smart proxy classes with the Service API
> interface name (in addition to the codebase annotation).
> 2. Only annotate proxy classes with the proxy Codebase annotation.
> 3. During marshaling annotate the stream with the Service API
> interface name and whether the class is preferred (this
> removes the need to download the preferred list from
> codebase URL to confirm if the class is preferred or not.
> 4. During unmarshalling, walk the call stack, find the first
> non system ClassLoader and call findClass(String name), keep
> traversing the stack until the Service API class is found.
> 5. The ClassLoader of the Service API is the parent
> ClassLoader, this will work as expected in any modular system.
> 4. Having found the parent ClassLoader we can now lookup our Map to
> find if a PreferredClassLoader has already been assigned to the
> unmarshaling object class. This eliminates the need to use the
> problematic Thread context ClassLoader, which may not be set or
> may be set incorrectly.
>
> Lacking codebase annotations, ObjectInputStream usually just tries to resolve classes using the first non null ClassLoader in the call stack (little more to it than that see the spec for details). Standard serialisation doesn't play well with OSGi, in fact exceptions are commonplace during deserialisation with OSGi, there is an easy way to solve this, also annotate the package version, with this information, we can find the correct classloader in a system where more than one ClassLoader may contain different versions of the same classes. In a system where only one version of a package can be found and this package uses a different version, we'll have to accept it, or provide another way for the client to resolve the package, this needs to be a standard interface that can be implemented at the client (or server for objects traveling the other way). Then maven, osgi or some other means can be used to resolve the missing dependency.
>
> Thoughts?
>
> Cheers,
>
> Peter.
>
>
Re: Develop new spec for RMIClassLoader replacement
Posted by GREGG WONDERLY <gr...@gmail.com>.
On Aug 23, 2012, at 5:57 AM, Simon IJskes - QCG <si...@qcg.nl> wrote:
> On 19-08-12 14:09, Peter Firmstone wrote:
>
>> Thoughts?
>
> What is the main driver for the RMIClassLoader replacement?
The technical details, are/were that the default behavior of PreferredClassProvider and RMIClassLoaderSPI, together said that the parent class loader of the unmarshalled objects' URLClassLoader, which was created using the annotation returned by RMIClassLoaderSPI, was null, or otherwise known as the system class loader.
In netbeans, the correct parent class loader, is the class loader of the invoking class, because the system class loader is empty of anything except the basic netbeans platform classes which are enough to start module loading from the other class loaders which are created/associated with the "modules" that make up the netbeans application (whether or not it's the actual IDE application itself).
So, I needed to augment the behavior of PreferredClassProvider and replace the RMIClassLoaderSPI implementation with a new one. The problem is, that you can't change the netbeans "classpath", at startup, with a "module". You have to rewrite all kinds of "configuration" and such and thus you can't plug in a different RMIClassLoaderSPI, at all.
Thus, I decided to do what developers do best to solve software problems. I added another layer of indirection between Jini/River and the RMIClassLoaderSPI so that we could replace the behavior of RMIClassLoaderSPI programmatically, at runtime. I augmented PreferredClassProvider to call out to this new layer as well, to access the details about which parent class loader should be used.
Additionally...
When I was doing this, I was also imagining changes to how unmarshalling would be done, and so I wanted to be able to replace the entire behavior of RMIClassLoaderSPI in the river source, to potentially provide the start of a layer which would allow this to be done.
In the old days, we had ObjectSpace's Voyager in the RPC community, and they provided a seamless behavior of sending objects between VMs. It just worked. The codebase annotation available through the RMIClassLoaderSPI getAnnotation() mechanisms, was passed on to the recipient VM so that it could load the same vintage class definition as was used by the sender.
This, of course doesn't always "just work", but for many more applications, then not, it provides a more dependable mechanism because versioning all but disappears.
Gregg Wonderly
Re: Develop new spec for RMIClassLoader replacement
Posted by Peter Firmstone <ji...@zeus.net.au>.
Just in case people mistake this as another discussion about OSGi, read
this first:
/Project Jigsaw/: Module Class Loading and *...* - OpenJDK -
/Java/
<http://www.google.com.au/url?sa=t&rct=j&q=java%20project%20jigsaw%20classloader%20serialisation&source=web&cd=2&cad=rja&ved=0CC0QFjAB&url=http%3A%2F%2Fopenjdk.java.net%2Fprojects%2Fjigsaw%2Fdoc%2Fmodule-class-loading.pdf&ei=-Eo3UKnpLNC0iQe9hoGIBQ&usg=AFQjCNGDi5lO-yOpKMDmK14pMnQfzbEelg>
Gregg's on the right track, I support the work he's doing. There's
plenty of time to get this sorted, for now we just need to make
decisions that will allow flexibility later, when modularity becomes the
norm and not the exception.
Cheers,
Peter.
Greg Trasuk wrote:
> I can't speak for Gregg, but as I understood it, the
> CodebaseAccessClassLoader was an attempt to host clients to Jini
> services and/or Jini services themselves inside Netbeans. That's
> unrelated to developing services in an IDE. Certainly the IDE doesn't
> care what you do with the '.class' files it produces.
>
> Speaking for myself (and I've thought about this a fair bit), the real
> issue from the developer's perspective is ease of running the code that
> you write. Is it actually necessary to load a Jini service proxy inside
> the very complicated, modularized classloader environment that Netbeans
> implements? It would be convenient if it were easy to do that,
> certainly, but it isn't necessary.
>
> Back in the dark ages, we used Ant to create a '.war' file and copy that
> file into Tomcat's deployment directory, where a standalone instance of
> Tomcat (unaware of either Ant or an IDE) would note the '.war' file's
> presence and deploy it in the container. Many servlet-style IDE's do
> this today.
>
> In other words, there's no need for a Jini client inside Netbeans or
> Eclipse if you just want to deploy stuff.
>
> One level up is to have a remote control interface between the IDE and
> the deployment environment. Tomcat's admin application allows for
> this. IBM's Rational Application Developer makes web service calls to
> the Websphere instance. The IDE could then monitor and control parts of
> the container. This is useful for things like deploying applications
> "in-place" in the IDE's build directory rather than packaging and
> deploying. I think this is how Netbeans now interacts with Tomcat and
> Glassfish. I can't think offhand of an example where the IDE interface
> does nomitoring, though - it seems to be more common to have an
> administration application on the server, although often the IDE can
> open a browser to the admin page. This might be different on a Jini
> server because there isn't a natural servlet environment to run an
> application in.
>
> If the goal is for the IDE to "control" the server, I don't have a
> problem with them using a traditional non-Jini protocol (i.e. no
> downloaded proxy - they need to agree on the protocol) to do it, since
> they're going to be pretty tightly coupled anyway.
>
> Second level-up would be to have visibility into the Jini workgroup from
> inside Netbeans or eclipse. In other words, could we have the service
> browser functionality inside Netbeans? Again, it would be convenient to
> be able to host a Jini client inside Netbeans if it were easy. However,
> I think what Gregg W proved with CodebaseAccessClassLoader is that it
> isn't that easy. Again, I think a separate executable that Netbeans
> talks to using a plain socket protocol would prove easier than trying to
> navigate Netbeans' modular classloader hierarchy. This would also be a
> good example of where the surrogate spec would come in handy. Netbeans
> could ask a surrogate host to launch a surrogate that talks to the Djinn
> on its behalf.
>
> You could make an argument similar to the one in "A Note on Distributed
> Computing" (Waldo et al) that the idea of serializing an arbitrary
> object for transmission over the network is a bit of a fool's errand. I
> think it seems perfectly reasonable that the set of classes used in a
> service interface could be distributed as a jar file and subjected to
> external governance.
>
> To summarize, there's no reason to talk about serialization protocols,
> further extensions to codebase annotations, or replacing RMIClassLoader
> if the goal is to develop services using an IDE.
>
> Does a service container need a classloader where you can do mystical
> things with the codebase and classpath? You betcha! But that's the
> container's problem, and it probably has complexities that are more
> about the container design than modular classpaths. (If you're curious,
> have a look in Surrogate container under
> 'org.apache.river.container.classloading')
>
> A better question, perhaps, is "what environments should we consider as
> clients to business services implemented using River?" For that, I'd
> argue the answers "Plain Java", "Servlet environments", and "Other Jini
> Services" are the clear winners. I don't see a lot of evidence of
> OSGI-based environments for applications in the real world (by which I
> mean regular 'Joe Corporate' developers deploying OSGI bundles. Yes, I
> accept that servers and IDE's may use an OSGI container as their
> implementation environment).
>
> Hosting Jini services in anything besides a Jini-centric environment
> (whether it's the 'ServiceStarter' utility or a Jini-centric container)
> is guaranteed to be more trouble than it's worth, IMO.
>
> Cheers,
>
> Greg.
>
>
>
> On Thu, 2012-08-23 at 08:54, Peter Firmstone wrote:
>
>> Enabling developers to develop services in IDE's like Netbeans was the
>> original motivator (Gregg Wonderly), see
>>
>> https://issues.apache.org/jira/browse/RIVER-336
>>
>> Regards,
>>
>> Peter.
>>
>> On 23/08/2012 8:57 PM, Simon IJskes - QCG wrote:
>>
>>> On 19-08-12 14:09, Peter Firmstone wrote:
>>>
>>>
>>>> Thoughts?
>>>>
>>> What is the main driver for the RMIClassLoader replacement?
>>>
>>> Gr. Sim
>>>
>>>
>
>
>
Re: Develop new spec for RMIClassLoader replacement
Posted by Gregg Wonderly <ge...@cox.net>.
On Aug 24, 2012, at 5:53 AM, Simon IJskes - QCG <si...@qcg.nl> wrote:
> On 24-08-12 03:38, GREGG WONDERLY wrote:
>> I want to use the netbeans modular application framework as my
>> service UI host.
>
>> As I said, what I needed was to set the parent class loader for the
>> downloaded codes context class loader, specifically for netbeans.
>
> To make things simple for me: What you want, is another classloader that is currently used in jini, and currently it is not 'pluggable'/configurable?
The specific instance of ClassLoader that the URLClassLoader is created with, should be controllable. What PreferredClassProvider did as part of its activities related to RMIClassLoaderSPI, was create a new URLClassLoader for the "annotation" that the serialized data contained, and specify null as the parent class loader, so that in effect, the system class loader was to be used as the parent.
In netbeans, the most likely choice would be the ClassLoader of the module which the calling class (the one that is doing service lookup and thus causing the unmarshalling of the service object to occur.
>
> Does the material in jira solve the problem?
It does provide the plug-in point for implementations of the new "interface" to be used. From my perspective, it works for what I want to do. The default is to providing the previously existing behavior.
With the proper policy specified by the SecurityManager, code running in the VM, can replace that default implementation with a new one. Because there is a certain bit of "exploitation" that is always possible, anytime that the code can "change" something that other classes are using, we need to vet that code out.
> Can we make it pluggable? If so, shall we change the specs to include this plugin 'interface'.
Once it's in place, there should be specs that detail how it is designed to be used, so that we can help people
understand the power and limits of the features provided. It functions at a very basic, low level part of the system, but its features are essential to moving service lookup and unmarshalling into a modular JVM environment such as that which Netbeans and other such environments provide.
Gregg
Re: Develop new spec for RMIClassLoader replacement
Posted by Simon IJskes - QCG <si...@qcg.nl>.
On 24-08-12 03:38, GREGG WONDERLY wrote:
> I want to use the netbeans modular application framework as my
> service UI host.
> As I said, what I needed was to set the parent class loader for the
> downloaded codes context class loader, specifically for netbeans.
To make things simple for me: What you want, is another classloader that
is currently used in jini, and currently it is not 'pluggable'/configurable?
Does the material in jira solve the problem?
Can we make it pluggable? If so, shall we change the specs to include
this plugin 'interface'.
> What I've learned over the years, is that there are lot of subtle yet
> detrimental assumptions about the environment embedded in Jini/RMI.
> Many of the things I've run up against, don't add value or otherwise
> simplify the developers life. Rather, they complicate things, and
> leave little flexibility in deployment scenarios.
>
> This, I feel, is the number one reason why Jini has never gained much
> traction.
I completely agree.
Gr. Sim
Re: Develop new spec for RMIClassLoader replacement
Posted by Peter Firmstone <ji...@zeus.net.au>.
On 27/08/2012 10:43 AM, Dennis Reedy wrote:
> On Aug 26, 2012, at 723PM, Gregg Wonderly wrote:
>
>> On Aug 26, 2012, at 5:12 PM, Dennis Reedy<de...@gmail.com> wrote:
>>
>>> Gregg,
>>>
>>> If you want to use Netbeans RCP, then why not consider making everything OSGi-able? We are using a Netbeans RCP front end on a project I'm working on now (with a Rio-backend that uses a custom RMIClassLoaderSpi that does artifact resolution for an artifact URL, goodbye http codebases and good riddance:) ), and from what I see, you either need to wrap everything up and turn it into an OSGi module, or go whole-hog and make everything OSGi (its all or nothing).
>> Hi Dennis!
> Hi Gregg :)
>
>> What I'd like to not have to do, is all kinds of packaging and versioning control work. I'd like to be able to just "run watcha brung". Versioning happens, and I can appreciate dealing with it. However, I think it's also a good point of abuse which can lead to stuff just not working well, because versioning leads to breakage when testing escalates by orders of magnitude and some tests stop getting run. That's what I don't like about the world of OSGi. It seems too ridged and too tool intensive for the small applications.
> I agree with your issues wrt OSGi, the only reason I brought it up was that NB RCP module system is OSGi based. I'm working with one of the NB dream team guys, I think he's on this list, if not maybe I can get him to comment and advise.
>
>> I would like to have something that enables all the dynamic code flexibility, but which has a much better depends-on graph resolution strategy so that one could build all the various "code source" bits and fully believe that there wasn't missing classes.
> I'm not sure if this helps (or is of interest to) you, but what I've been doing wrt to codebase support is to use the dependency resolution that you find with maven based artifacts (note you dont have to have a maven project, you just deploy your jars to a maven repository). We've been finding it much easier to configure services in a versioned and easy-to-deploy way.
>
> So what you end up with is the runtime dependencies of a particular artifact resolved (direct and transitive dependencies) as the codebase for a service, or a service-ui. So your 'depends on graph' is complete, in as much as your dependency graph is correctly constructed for your artifact. This comes naturally for maven/gradle projects, you can't produce your artifact unless the dependencies have been declared correctly.
>
> Note that this becomes important especially for a client that uses a service. With the artifact URL scheme, instead of annotating a service's codebase with http:// based jars, the service's codebase contains the artifact URL, which (when resolved) resolves the dependencies for the service's codebase at the client. This not only presents a performance boost (why load classes over http if they can be loaded locally), but also addresses lost codebase issues. Add to that secure repository connections that require uid/password, and you can confirm that the artifact a service requires you download in order for you to use that service, comes from a site you trust.
>
> Of course this is all versioned, and once loaded over the wire never needs to be loaded again. If the service's version changes, it's artifact changes, and any jars that have not been resolved get resolved.
>
> So the dynamic proxy capabilities remain, no change there.
>
> Just one more thing. One of my primary motivations for even looking into this was to address permanent heap OOME. I found a main contributor here was the RMIClassLoader that would keep class loader references around because of an HTTP keep alive thread. Addressing that issue lead me to this idea of using the artifact URL scheme, and avoiding the http URL altogether.
>
> HTH
>
> Regards
>
> Dennis
>
Dennis,
Thank you for speaking up, this is the only comprehensive solution to so
many issues we have all struggled with, issues that plagued Jini,
decimated in one fell swoop, I'm gobsmacked. Do you realise how many
Jira issues this probably solves?
I'm ashamed to say that you've mentioned it previously on this list.
Is the artifact URL scheme provider available publicly? If so, we need
to include it with River, to ensure it's available at the platform
level. Do you know if the URL scheme is compatible with RFC 3986 URI
normalisation?
Did you write the RMIClassLoaderSPI implementation? Again I'd very much
appreciate it if you wanted to contribute it back to River.
Whether people like Maven or not is beyond the point, this is the only
comprehensive solution available, it needs to be the preferred solution
for anyone new to River,this lowers the bar significantly, developers no
longer even need to know what a ClassLoader is! Anyone that doesn't
like it needn't use it, but I suspect given time, it'll grow on even the
most staunch disbeliever.
Cheers,
Peter.
Re: Develop new spec for RMIClassLoader replacement
Posted by Dennis Reedy <de...@gmail.com>.
Hi Peter,
On Aug 29, 2012, at 828AM, Peter Firmstone wrote:
> Dennis,
>
> Where can I find the artifact url handler, I only seem to be able to find the mvn url handler?
Check on github. The artifact URL support is here
https://github.com/dreedyman/Rio/tree/master/rio-platform/src/main/java/org/rioproject/url/artifact
Dennis
Re: Develop new spec for RMIClassLoader replacement
Posted by Peter Firmstone <ji...@zeus.net.au>.
Dennis,
Where can I find the artifact url handler, I only seem to be able to
find the mvn url handler?
Regards,
Peter.
On 27/08/2012 10:50 PM, Dennis Reedy wrote:
>
>>> I'm not sure if this helps (or is of interest to) you, but what I've
>>> been doing wrt to codebase support is to use the dependency resolution
>>> that you find with maven based artifacts (note you dont have to have a
>>> maven project, you just deploy your jars to a maven repository). We've
>>> been finding it much easier to configure services in a versioned and
>>> easy-to-deploy way.
>>>
>>> So what you end up with is the runtime dependencies of a particular
>>> artifact resolved (direct and transitive dependencies) as the codebase
>>> for a service, or a service-ui. So your 'depends on graph' is
>>> complete, in as much as your dependency graph is correctly constructed
>>> for your artifact. This comes naturally for maven/gradle projects, you
>>> can't produce your artifact unless the dependencies have been declared
>>> correctly.
>>>
>>> Note that this becomes important especially for a client that uses a
>>> service. With the artifact URL scheme, instead of annotating a
>>> service's codebase with http:// based jars, the service's codebase
>>> contains the artifact URL, which (when resolved) resolves the
>>> dependencies for the service's codebase at the client. This not only
>>> presents a performance boost (why load classes over http if they can
>>> be loaded locally), but also addresses lost codebase issues.
>> So, the artifact url sticks with the classloader? Or is it that the
>> service's artifact url implies a set of jars that will be resolved for
>> anyone else as well?
> The artifact URL gets resolved to locally available jars. These then are passed to the default RMI provider instance as codebase to load.
>
>>
>>> Add to that secure repository connections that require uid/password,
>>> and you can confirm that the artifact a service requires you download
>>> in order for you to use that service, comes from a site you trust.
>>>
>> Does everyone in the Djinn need to agree on where the Maven
>> repository(s) is(are)?
> The artifact URL scheme decomposes as follows:
>
> artifact:groupId/artifactId/version[/type[/classifier]][;[repository[@repositoryId]]
>
> Here are some examples:
>
> An artifact URL for groupId, artifactId and version:
>
> artifact:org.rioproject.examples.calculator/calculator-service/2.0.1
>
> An artifact URL for groupId, artifactId, version and repository with an id
> artifact:org.rioproject.examples.calculator/calculator-proxy/2.0.1;http://www.rio-project.org@rio
>
> The receiving client does not need to know the repository in advance. If the repository is secure, the client must have locally available credentials to apply when making the connection though. So using this approach you could easily verify the source before you download. Basically give trust to the code you download before you use it.
>
>> The thing I like about the codebase annotation
>> is that the service provider can go ahead and setup a web server and the
>> client finds out everything it needs to know from the annotation. Can
>> the clients find the repository configurations dynamically? Plus, I
>> suppose you actually have to have a repository setup somewhere, or can a
>> service container host its own repository? These may be stupid
>> questions - I haven't used Maven very much.
> One of the things I have found is that having well known places to get things from (like service jars) in a deployment environment is very helpful. It allows organizations (both development and commercial) to control, organize and secure their assets. In my experience the dynamic codebase issues that come along with River offer unnecessary complexity.
>
> Setting up a repository is needed. Thats easy though. Its really just a web server that manages a structure of maven-deployed artifacts. Or, it can be a Maven Repository Manager (Archiva, Artifactory, Nexus). From the projects I've been working with this becomes a natural extension of the development process, its hooked into the continuous integration that gets done. Then when we deploy services we deploy them using an artifact that represents the whole system. We maintain multiple versions side-by-side.
>
> I'm curious as to how others develop and deploy their systems. I know many on this list are anti-maven, what do you use?
>
> Dennis
>
>
>
Re: Develop new spec for RMIClassLoader replacement
Posted by Dennis Reedy <de...@gmail.com>.
On Aug 26, 2012, at 1147PM, Greg Trasuk wrote:
>
>
> Hi Dennis:
>
> Possibly dumb questions inlined...
>
> Cheers,
>
> Greg.
>
> On Sun, 2012-08-26 at 20:43, Dennis Reedy wrote:
>>
>>
>> I'm not sure if this helps (or is of interest to) you, but what I've
>> been doing wrt to codebase support is to use the dependency resolution
>> that you find with maven based artifacts (note you dont have to have a
>> maven project, you just deploy your jars to a maven repository). We've
>> been finding it much easier to configure services in a versioned and
>> easy-to-deploy way.
>>
>> So what you end up with is the runtime dependencies of a particular
>> artifact resolved (direct and transitive dependencies) as the codebase
>> for a service, or a service-ui. So your 'depends on graph' is
>> complete, in as much as your dependency graph is correctly constructed
>> for your artifact. This comes naturally for maven/gradle projects, you
>> can't produce your artifact unless the dependencies have been declared
>> correctly.
>>
>> Note that this becomes important especially for a client that uses a
>> service. With the artifact URL scheme, instead of annotating a
>> service's codebase with http:// based jars, the service's codebase
>> contains the artifact URL, which (when resolved) resolves the
>> dependencies for the service's codebase at the client. This not only
>> presents a performance boost (why load classes over http if they can
>> be loaded locally), but also addresses lost codebase issues.
>
> So, the artifact url sticks with the classloader? Or is it that the
> service's artifact url implies a set of jars that will be resolved for
> anyone else as well?
The artifact URL gets resolved to locally available jars. These then are passed to the default RMI provider instance as codebase to load.
>
>
>> Add to that secure repository connections that require uid/password,
>> and you can confirm that the artifact a service requires you download
>> in order for you to use that service, comes from a site you trust.
>>
>
> Does everyone in the Djinn need to agree on where the Maven
> repository(s) is(are)?
The artifact URL scheme decomposes as follows:
artifact:groupId/artifactId/version[/type[/classifier]][;[repository[@repositoryId]]
Here are some examples:
An artifact URL for groupId, artifactId and version:
artifact:org.rioproject.examples.calculator/calculator-service/2.0.1
An artifact URL for groupId, artifactId, version and repository with an id
artifact:org.rioproject.examples.calculator/calculator-proxy/2.0.1;http://www.rio-project.org@rio
The receiving client does not need to know the repository in advance. If the repository is secure, the client must have locally available credentials to apply when making the connection though. So using this approach you could easily verify the source before you download. Basically give trust to the code you download before you use it.
> The thing I like about the codebase annotation
> is that the service provider can go ahead and setup a web server and the
> client finds out everything it needs to know from the annotation. Can
> the clients find the repository configurations dynamically? Plus, I
> suppose you actually have to have a repository setup somewhere, or can a
> service container host its own repository? These may be stupid
> questions - I haven't used Maven very much.
One of the things I have found is that having well known places to get things from (like service jars) in a deployment environment is very helpful. It allows organizations (both development and commercial) to control, organize and secure their assets. In my experience the dynamic codebase issues that come along with River offer unnecessary complexity.
Setting up a repository is needed. Thats easy though. Its really just a web server that manages a structure of maven-deployed artifacts. Or, it can be a Maven Repository Manager (Archiva, Artifactory, Nexus). From the projects I've been working with this becomes a natural extension of the development process, its hooked into the continuous integration that gets done. Then when we deploy services we deploy them using an artifact that represents the whole system. We maintain multiple versions side-by-side.
I'm curious as to how others develop and deploy their systems. I know many on this list are anti-maven, what do you use?
Dennis
Re: Develop new spec for RMIClassLoader replacement
Posted by Greg Trasuk <tr...@stratuscom.com>.
Hi Dennis:
Possibly dumb questions inlined...
Cheers,
Greg.
On Sun, 2012-08-26 at 20:43, Dennis Reedy wrote:
> On Aug 26, 2012, at 723PM, Gregg Wonderly wrote:
>
> >
> > On Aug 26, 2012, at 5:12 PM, Dennis Reedy <de...@gmail.com>
> wrote:
> >
> >> Gregg,
> >>
> >> If you want to use Netbeans RCP, then why not consider making
> everything OSGi-able? We are using a Netbeans RCP front end on a
> project I'm working on now (with a Rio-backend that uses a custom
> RMIClassLoaderSpi that does artifact resolution for an artifact URL,
> goodbye http codebases and good riddance:) ), and from what I see, you
> either need to wrap everything up and turn it into an OSGi module, or
> go whole-hog and make everything OSGi (its all or nothing).
> >
> > Hi Dennis!
>
> Hi Gregg :)
>
> > What I'd like to not have to do, is all kinds of packaging and
> versioning control work. I'd like to be able to just "run watcha
> brung". Versioning happens, and I can appreciate dealing with it.
> However, I think it's also a good point of abuse which can lead to
> stuff just not working well, because versioning leads to breakage when
> testing escalates by orders of magnitude and some tests stop getting
> run. That's what I don't like about the world of OSGi. It seems too
> ridged and too tool intensive for the small applications.
>
> I agree with your issues wrt OSGi, the only reason I brought it up was
> that NB RCP module system is OSGi based. I'm working with one of the
> NB dream team guys, I think he's on this list, if not maybe I can get
> him to comment and advise.
>
> >
> > I would like to have something that enables all the dynamic code
> flexibility, but which has a much better depends-on graph resolution
> strategy so that one could build all the various "code source" bits
> and fully believe that there wasn't missing classes.
>
> I'm not sure if this helps (or is of interest to) you, but what I've
> been doing wrt to codebase support is to use the dependency resolution
> that you find with maven based artifacts (note you dont have to have a
> maven project, you just deploy your jars to a maven repository). We've
> been finding it much easier to configure services in a versioned and
> easy-to-deploy way.
>
> So what you end up with is the runtime dependencies of a particular
> artifact resolved (direct and transitive dependencies) as the codebase
> for a service, or a service-ui. So your 'depends on graph' is
> complete, in as much as your dependency graph is correctly constructed
> for your artifact. This comes naturally for maven/gradle projects, you
> can't produce your artifact unless the dependencies have been declared
> correctly.
>
> Note that this becomes important especially for a client that uses a
> service. With the artifact URL scheme, instead of annotating a
> service's codebase with http:// based jars, the service's codebase
> contains the artifact URL, which (when resolved) resolves the
> dependencies for the service's codebase at the client. This not only
> presents a performance boost (why load classes over http if they can
> be loaded locally), but also addresses lost codebase issues.
So, the artifact url sticks with the classloader? Or is it that the
service's artifact url implies a set of jars that will be resolved for
anyone else as well?
> Add to that secure repository connections that require uid/password,
> and you can confirm that the artifact a service requires you download
> in order for you to use that service, comes from a site you trust.
>
Does everyone in the Djinn need to agree on where the Maven
repository(s) is(are)? The thing I like about the codebase annotation
is that the service provider can go ahead and setup a web server and the
client finds out everything it needs to know from the annotation. Can
the clients find the repository configurations dynamically? Plus, I
suppose you actually have to have a repository setup somewhere, or can a
service container host its own repository? These may be stupid
questions - I haven't used Maven very much.
> Of course this is all versioned, and once loaded over the wire never
> needs to be loaded again. If the service's version changes, it's
> artifact changes, and any jars that have not been resolved get
> resolved.
>
> So the dynamic proxy capabilities remain, no change there.
>
> Just one more thing. One of my primary motivations for even looking
> into this was to address permanent heap OOME. I found a main
> contributor here was the RMIClassLoader that would keep class loader
> references around because of an HTTP keep alive thread. Addressing
> that issue lead me to this idea of using the artifact URL scheme, and
> avoiding the http URL altogether.
>
> HTH
>
> Regards
>
> Dennis
>
Re: Develop new spec for RMIClassLoader replacement
Posted by Dennis Reedy <de...@gmail.com>.
On Aug 26, 2012, at 723PM, Gregg Wonderly wrote:
>
> On Aug 26, 2012, at 5:12 PM, Dennis Reedy <de...@gmail.com> wrote:
>
>> Gregg,
>>
>> If you want to use Netbeans RCP, then why not consider making everything OSGi-able? We are using a Netbeans RCP front end on a project I'm working on now (with a Rio-backend that uses a custom RMIClassLoaderSpi that does artifact resolution for an artifact URL, goodbye http codebases and good riddance:) ), and from what I see, you either need to wrap everything up and turn it into an OSGi module, or go whole-hog and make everything OSGi (its all or nothing).
>
> Hi Dennis!
Hi Gregg :)
> What I'd like to not have to do, is all kinds of packaging and versioning control work. I'd like to be able to just "run watcha brung". Versioning happens, and I can appreciate dealing with it. However, I think it's also a good point of abuse which can lead to stuff just not working well, because versioning leads to breakage when testing escalates by orders of magnitude and some tests stop getting run. That's what I don't like about the world of OSGi. It seems too ridged and too tool intensive for the small applications.
I agree with your issues wrt OSGi, the only reason I brought it up was that NB RCP module system is OSGi based. I'm working with one of the NB dream team guys, I think he's on this list, if not maybe I can get him to comment and advise.
>
> I would like to have something that enables all the dynamic code flexibility, but which has a much better depends-on graph resolution strategy so that one could build all the various "code source" bits and fully believe that there wasn't missing classes.
I'm not sure if this helps (or is of interest to) you, but what I've been doing wrt to codebase support is to use the dependency resolution that you find with maven based artifacts (note you dont have to have a maven project, you just deploy your jars to a maven repository). We've been finding it much easier to configure services in a versioned and easy-to-deploy way.
So what you end up with is the runtime dependencies of a particular artifact resolved (direct and transitive dependencies) as the codebase for a service, or a service-ui. So your 'depends on graph' is complete, in as much as your dependency graph is correctly constructed for your artifact. This comes naturally for maven/gradle projects, you can't produce your artifact unless the dependencies have been declared correctly.
Note that this becomes important especially for a client that uses a service. With the artifact URL scheme, instead of annotating a service's codebase with http:// based jars, the service's codebase contains the artifact URL, which (when resolved) resolves the dependencies for the service's codebase at the client. This not only presents a performance boost (why load classes over http if they can be loaded locally), but also addresses lost codebase issues. Add to that secure repository connections that require uid/password, and you can confirm that the artifact a service requires you download in order for you to use that service, comes from a site you trust.
Of course this is all versioned, and once loaded over the wire never needs to be loaded again. If the service's version changes, it's artifact changes, and any jars that have not been resolved get resolved.
So the dynamic proxy capabilities remain, no change there.
Just one more thing. One of my primary motivations for even looking into this was to address permanent heap OOME. I found a main contributor here was the RMIClassLoader that would keep class loader references around because of an HTTP keep alive thread. Addressing that issue lead me to this idea of using the artifact URL scheme, and avoiding the http URL altogether.
HTH
Regards
Dennis
Re: Develop new spec for RMIClassLoader replacement
Posted by Gregg Wonderly <gr...@wonderly.org>.
On Aug 26, 2012, at 5:12 PM, Dennis Reedy <de...@gmail.com> wrote:
> Gregg,
>
> If you want to use Netbeans RCP, then why not consider making everything OSGi-able? We are using a Netbeans RCP front end on a project I'm working on now (with a Rio-backend that uses a custom RMIClassLoaderSpi that does artifact resolution for an artifact URL, goodbye http codebases and good riddance:) ), and from what I see, you either need to wrap everything up and turn it into an OSGi module, or go whole-hog and make everything OSGi (its all or nothing).
Hi Dennis! What I'd like to not have to do, is all kinds of packaging and versioning control work. I'd like to be able to just "run watcha brung". Versioning happens, and I can appreciate dealing with it. However, I think it's also a good point of abuse which can lead to stuff just not working well, because versioning leads to breakage when testing escalates by orders of magnitude and some tests stop getting run. That's what I don't like about the world of OSGi. It seems too ridged and too tool intensive for the small applications.
I would like to have something that enables all the dynamic code flexibility, but which has a much better depends-on graph resolution strategy so that one could build all the various "code source" bits and fully believe that there wasn't missing classes.
Gregg
> Dennis
>
> On Aug 26, 2012, at 211PM, Gregg Wonderly wrote:
>
>>
>> On Aug 26, 2012, at 9:17 AM, Greg Trasuk <tr...@stratuscom.com> wrote:
>>
>>> See comments interspersed...
>>>
>>>
>>> On Sun, 2012-08-26 at 04:53, Peter Firmstone wrote:
>>>> There are some notable conditions that may exist in modular environments.
>>>>
>>>> 1. Lets get this out of the way first: Preferred proxy classes are
>>>> implementation classes, the modular environment has no place
>>>> managing proxy private implementation classes.
>>>> 2. In addition to the marshaled smart proxy implementation,
>>>> Serialized objects will be transferred between remote hosts and
>>>> these may not belong to the proxy preferred class list, instead
>>>> they will belong to the Java or Jini platforms, or the Service
>>>> API, a local modular environment requires these objects be
>>>> instantiated by the correct class instance, which may reside in
>>>> different ClassLoaders.
>>>> 3. Different versions of classes may be present locally in separate
>>>> ClassLoaders.
>>>> 4. Resolving classes for objects a smart proxy refers to may not be
>>>> visible via traditionally expected ClassLoader hierarchy trees.
>>>> These classes may need to be found by other means. These classes
>>>> needn't be visible to the smart proxy, fields may refer to a
>>>> common super class or interface.
>>>> 5. Service API must reside in a parent ClassLoader, so we need to
>>>> know super class names, to be able to nominate a suitable parent
>>>> ClassLoader for a smart proxy.
>>>>
>>> Small point - #5 isn't quite true, although that's the way most
>>> containers, Jini and otherwise, seem to operate. For instance, in most
>>> servlet containers, the servlet API is in a parent classloader to the
>>> servlet itself. Similarly, in the Jini Starter utility, the River API
>>> classes are in a parent classloader. This is usually done for the
>>> convenience of the container - it allows the container to access the
>>> same classes that the application does, without needing to use
>>> reflection.
>>>
>>> The correct, and slightly looser, constraint is that classes and
>>> interfaces used in the API must be visible to application classes. They
>>> could be in the same classloader as the applicaton classes. In the
>>> River container I'm working on in /skunk, the service API classes are
>>> actually in the same classloader as the application (much like the
>>> application would include the interface classes for a user-written
>>> service). The container communicates with the application using the
>>> Reflection API, and doesn't share any classloaders with the contained
>>> applications (except the system CL, of course).
>>>
>>> I guess what I'm saying is that it's hazardous to assume any particular
>>> classloader hierarchy. The classloader used to house the proxy should
>>> probably just have the application's context class loader as its parent.
>>
>> This is the flexibility that I think we need to focus on. We need to think about class origins, and any time/place that River is interacting with class loading, we should try and evaluate how that can be the least troublesome. In some cases, no-flexibility can be less trouble. 100 interfaces and abstract classes don't always make the best programming environment. One way to do things, which works well, can be a good thing.
>>
>>>
>>>> This won't be so hard to implement if we have a ClassLoader registry and
>>>> we know super class names and version information, this information is
>>>> currently not annotated with the Object stream.
>>>>
>>> I don't think versions are required. Simply load any preferred classes
>>> from the codebase url that the remote object provides.
>>
>> I'd like to keep "Versioning" completely out of what River considers as part of the "platform". The plugability of class loading should not hinder the use of versioning. It may be a good idea to put down some sentences which represent the features which we all agree versioning is represented by. Then we can keep River from prohibiting those things as a choice.
>>
>> Gregg
>>
>
Re: Develop new spec for RMIClassLoader replacement
Posted by Dennis Reedy <de...@gmail.com>.
Gregg,
If you want to use Netbeans RCP, then why not consider making everything OSGi-able? We are using a Netbeans RCP front end on a project I'm working on now (with a Rio-backend that uses a custom RMIClassLoaderSpi that does artifact resolution for an artifact URL, goodbye http codebases and good riddance:) ), and from what I see, you either need to wrap everything up and turn it into an OSGi module, or go whole-hog and make everything OSGi (its all or nothing).
Dennis
On Aug 26, 2012, at 211PM, Gregg Wonderly wrote:
>
> On Aug 26, 2012, at 9:17 AM, Greg Trasuk <tr...@stratuscom.com> wrote:
>
>> See comments interspersed...
>>
>>
>> On Sun, 2012-08-26 at 04:53, Peter Firmstone wrote:
>>> There are some notable conditions that may exist in modular environments.
>>>
>>> 1. Lets get this out of the way first: Preferred proxy classes are
>>> implementation classes, the modular environment has no place
>>> managing proxy private implementation classes.
>>> 2. In addition to the marshaled smart proxy implementation,
>>> Serialized objects will be transferred between remote hosts and
>>> these may not belong to the proxy preferred class list, instead
>>> they will belong to the Java or Jini platforms, or the Service
>>> API, a local modular environment requires these objects be
>>> instantiated by the correct class instance, which may reside in
>>> different ClassLoaders.
>>> 3. Different versions of classes may be present locally in separate
>>> ClassLoaders.
>>> 4. Resolving classes for objects a smart proxy refers to may not be
>>> visible via traditionally expected ClassLoader hierarchy trees.
>>> These classes may need to be found by other means. These classes
>>> needn't be visible to the smart proxy, fields may refer to a
>>> common super class or interface.
>>> 5. Service API must reside in a parent ClassLoader, so we need to
>>> know super class names, to be able to nominate a suitable parent
>>> ClassLoader for a smart proxy.
>>>
>> Small point - #5 isn't quite true, although that's the way most
>> containers, Jini and otherwise, seem to operate. For instance, in most
>> servlet containers, the servlet API is in a parent classloader to the
>> servlet itself. Similarly, in the Jini Starter utility, the River API
>> classes are in a parent classloader. This is usually done for the
>> convenience of the container - it allows the container to access the
>> same classes that the application does, without needing to use
>> reflection.
>>
>> The correct, and slightly looser, constraint is that classes and
>> interfaces used in the API must be visible to application classes. They
>> could be in the same classloader as the applicaton classes. In the
>> River container I'm working on in /skunk, the service API classes are
>> actually in the same classloader as the application (much like the
>> application would include the interface classes for a user-written
>> service). The container communicates with the application using the
>> Reflection API, and doesn't share any classloaders with the contained
>> applications (except the system CL, of course).
>>
>> I guess what I'm saying is that it's hazardous to assume any particular
>> classloader hierarchy. The classloader used to house the proxy should
>> probably just have the application's context class loader as its parent.
>
> This is the flexibility that I think we need to focus on. We need to think about class origins, and any time/place that River is interacting with class loading, we should try and evaluate how that can be the least troublesome. In some cases, no-flexibility can be less trouble. 100 interfaces and abstract classes don't always make the best programming environment. One way to do things, which works well, can be a good thing.
>
>>
>>> This won't be so hard to implement if we have a ClassLoader registry and
>>> we know super class names and version information, this information is
>>> currently not annotated with the Object stream.
>>>
>> I don't think versions are required. Simply load any preferred classes
>> from the codebase url that the remote object provides.
>
> I'd like to keep "Versioning" completely out of what River considers as part of the "platform". The plugability of class loading should not hinder the use of versioning. It may be a good idea to put down some sentences which represent the features which we all agree versioning is represented by. Then we can keep River from prohibiting those things as a choice.
>
> Gregg
>
Re: Develop new spec for RMIClassLoader replacement
Posted by Gregg Wonderly <gr...@gmail.com>.
On Aug 26, 2012, at 9:17 AM, Greg Trasuk <tr...@stratuscom.com> wrote:
> See comments interspersed...
>
>
> On Sun, 2012-08-26 at 04:53, Peter Firmstone wrote:
>> There are some notable conditions that may exist in modular environments.
>>
>> 1. Lets get this out of the way first: Preferred proxy classes are
>> implementation classes, the modular environment has no place
>> managing proxy private implementation classes.
>> 2. In addition to the marshaled smart proxy implementation,
>> Serialized objects will be transferred between remote hosts and
>> these may not belong to the proxy preferred class list, instead
>> they will belong to the Java or Jini platforms, or the Service
>> API, a local modular environment requires these objects be
>> instantiated by the correct class instance, which may reside in
>> different ClassLoaders.
>> 3. Different versions of classes may be present locally in separate
>> ClassLoaders.
>> 4. Resolving classes for objects a smart proxy refers to may not be
>> visible via traditionally expected ClassLoader hierarchy trees.
>> These classes may need to be found by other means. These classes
>> needn't be visible to the smart proxy, fields may refer to a
>> common super class or interface.
>> 5. Service API must reside in a parent ClassLoader, so we need to
>> know super class names, to be able to nominate a suitable parent
>> ClassLoader for a smart proxy.
>>
> Small point - #5 isn't quite true, although that's the way most
> containers, Jini and otherwise, seem to operate. For instance, in most
> servlet containers, the servlet API is in a parent classloader to the
> servlet itself. Similarly, in the Jini Starter utility, the River API
> classes are in a parent classloader. This is usually done for the
> convenience of the container - it allows the container to access the
> same classes that the application does, without needing to use
> reflection.
>
> The correct, and slightly looser, constraint is that classes and
> interfaces used in the API must be visible to application classes. They
> could be in the same classloader as the applicaton classes. In the
> River container I'm working on in /skunk, the service API classes are
> actually in the same classloader as the application (much like the
> application would include the interface classes for a user-written
> service). The container communicates with the application using the
> Reflection API, and doesn't share any classloaders with the contained
> applications (except the system CL, of course).
>
> I guess what I'm saying is that it's hazardous to assume any particular
> classloader hierarchy. The classloader used to house the proxy should
> probably just have the application's context class loader as its parent.
This is the flexibility that I think we need to focus on. We need to think about class origins, and any time/place that River is interacting with class loading, we should try and evaluate how that can be the least troublesome. In some cases, no-flexibility can be less trouble. 100 interfaces and abstract classes don't always make the best programming environment. One way to do things, which works well, can be a good thing.
>
>> This won't be so hard to implement if we have a ClassLoader registry and
>> we know super class names and version information, this information is
>> currently not annotated with the Object stream.
>>
> I don't think versions are required. Simply load any preferred classes
> from the codebase url that the remote object provides.
I'd like to keep "Versioning" completely out of what River considers as part of the "platform". The plugability of class loading should not hinder the use of versioning. It may be a good idea to put down some sentences which represent the features which we all agree versioning is represented by. Then we can keep River from prohibiting those things as a choice.
Gregg
Re: Develop new spec for RMIClassLoader replacement
Posted by Greg Trasuk <tr...@stratuscom.com>.
See comments interspersed...
On Sun, 2012-08-26 at 04:53, Peter Firmstone wrote:
> There are some notable conditions that may exist in modular environments.
>
> 1. Lets get this out of the way first: Preferred proxy classes are
> implementation classes, the modular environment has no place
> managing proxy private implementation classes.
> 2. In addition to the marshaled smart proxy implementation,
> Serialized objects will be transferred between remote hosts and
> these may not belong to the proxy preferred class list, instead
> they will belong to the Java or Jini platforms, or the Service
> API, a local modular environment requires these objects be
> instantiated by the correct class instance, which may reside in
> different ClassLoaders.
> 3. Different versions of classes may be present locally in separate
> ClassLoaders.
> 4. Resolving classes for objects a smart proxy refers to may not be
> visible via traditionally expected ClassLoader hierarchy trees.
> These classes may need to be found by other means. These classes
> needn't be visible to the smart proxy, fields may refer to a
> common super class or interface.
> 5. Service API must reside in a parent ClassLoader, so we need to
> know super class names, to be able to nominate a suitable parent
> ClassLoader for a smart proxy.
>
Small point - #5 isn't quite true, although that's the way most
containers, Jini and otherwise, seem to operate. For instance, in most
servlet containers, the servlet API is in a parent classloader to the
servlet itself. Similarly, in the Jini Starter utility, the River API
classes are in a parent classloader. This is usually done for the
convenience of the container - it allows the container to access the
same classes that the application does, without needing to use
reflection.
The correct, and slightly looser, constraint is that classes and
interfaces used in the API must be visible to application classes. They
could be in the same classloader as the applicaton classes. In the
River container I'm working on in /skunk, the service API classes are
actually in the same classloader as the application (much like the
application would include the interface classes for a user-written
service). The container communicates with the application using the
Reflection API, and doesn't share any classloaders with the contained
applications (except the system CL, of course).
I guess what I'm saying is that it's hazardous to assume any particular
classloader hierarchy. The classloader used to house the proxy should
probably just have the application's context class loader as its parent.
> This won't be so hard to implement if we have a ClassLoader registry and
> we know super class names and version information, this information is
> currently not annotated with the Object stream.
>
I don't think versions are required. Simply load any preferred classes
from the codebase url that the remote object provides.
> A modular JRE won't be released until Java 9, so there's no hurry, but
> to ensure we don't overlook any important details, lets create a related
> Jira issue to develop a spec.
>
> Cheers,
>
> Peter.
Cheers,
Greg.
Re: Develop new spec for RMIClassLoader replacement
Posted by Peter Firmstone <ji...@zeus.net.au>.
There are some notable conditions that may exist in modular environments.
1. Lets get this out of the way first: Preferred proxy classes are
implementation classes, the modular environment has no place
managing proxy private implementation classes.
2. In addition to the marshaled smart proxy implementation,
Serialized objects will be transferred between remote hosts and
these may not belong to the proxy preferred class list, instead
they will belong to the Java or Jini platforms, or the Service
API, a local modular environment requires these objects be
instantiated by the correct class instance, which may reside in
different ClassLoaders.
3. Different versions of classes may be present locally in separate
ClassLoaders.
4. Resolving classes for objects a smart proxy refers to may not be
visible via traditionally expected ClassLoader hierarchy trees.
These classes may need to be found by other means. These classes
needn't be visible to the smart proxy, fields may refer to a
common super class or interface.
5. Service API must reside in a parent ClassLoader, so we need to
know super class names, to be able to nominate a suitable parent
ClassLoader for a smart proxy.
This won't be so hard to implement if we have a ClassLoader registry and
we know super class names and version information, this information is
currently not annotated with the Object stream.
A modular JRE won't be released until Java 9, so there's no hurry, but
to ensure we don't overlook any important details, lets create a related
Jira issue to develop a spec.
Cheers,
Peter.
GREGG WONDERLY wrote:
> I want to use the netbeans modular application framework as my service UI host.
>
> That is, I want to be able to use all the features of the RCP to create a complex service UI based client environment. Thus, I want to be able to download and run serviceUIs inside of netbeans, and that's why I need to be able to manage how classes are resolved relative to the modularization of netbeans.
>
> I could then facilitate, as well, interaction with service management UIs so that we could have a River admin module that could be used in the IDE to talk to service container management endpoints.
>
> No matter whether the proxy is in the client or in another process, for the IDE to talk to it, you have to have the same "level" of object compatibility and communication, to make use of the services exported endpoint and/or serviceUI.
>
> The changes I made were not hard, nor complex. There are relatively few places that the interactions with RMIClassLoaderSPI occur, and with JINI 2.0 and PreferredClassLoader, RMIClassLoaderSPIs only useful remaining activity is to convey the class annotation across the wire.
>
> As I said, what I needed was to set the parent class loader for the downloaded codes context class loader, specifically for netbeans.
>
> What I've learned over the years, is that there are lot of subtle yet detrimental assumptions about the environment embedded in Jini/RMI. Many of the things I've run up against, don't add value or otherwise simplify the developers life. Rather, they complicate things, and leave little flexibility in deployment scenarios.
>
> This, I feel, is the number one reason why Jini has never gained much traction. It's hard to "just use it"! We have had lots of examples of this over the years on the mailing lists and in conversations with people at JavaOne and Jini community meetings.
>
> At this point, changing "everything" to remove the issues, is what I would find stimulating enough to fully engage.
>
> At this point in my life, I've spent more than a decade using Jini for everything I've done in service based applications. It's been a long hard road to provide my customers with something that worked well enough in their fragile computing environments, that they could be productive. Many just want a web page, because they think that their browser is the only "desktop" they need.
>
> Sending code and marshaling and unmarshaling objects is a huge, huge barrier to some. Many of us have done this enough, that its just old hat. People love to "buy" XML marshaling and use http for transport. It seems easy because there are tools that have been written on both ends and in the middle to simplify the developers life.
>
> Because we've always just used ServiceStarter (well not everyone, but that's all the community seems to want to support as a part of the "platform"), we have an extremely primitive and highly configuration dependent "platform".
>
> For many, that seems to be a giant turn off. People want finite choices and "GUIs" that enumerate all the choices and the ability to just click and pick a solution. That's where tooling is at in this day and age.
>
> So, that's what I still would like to achieve with River. There are far more choices then necessary, visible to new developers. The majority of that flexibility is not needed to get started, and it's possible to put tooling together that would make it much easier.
>
> Dennis's work on Rio has gone largely ignored, even though it has so much really useful automation and management in it.
>
> We lost out with Seven as well. Mark is not around any longer, really. His life situation did not allow him to do all the work that was needed to put something together for public use, and so we ended up without that simplification of the service deployment scenario either.
>
> There are several different JavaEE deployment tool sets that have been crafted as well.
>
> I think its very clear that there are a lot of different ideas that have been layed out, implemented and deployed, which did in fact demonstrate improvements.
>
> With all of that behind us now, I'd really like to see something really powerful happen with a change in how we view River and how we want to present it to the world, anew.
>
> Our community is full of lots of ideas and experiences, and I don't want to diminish any of that by the statements I made above. I'm just trying to relate what I've seen and where I'd still like to see something happen.
>
> Gregg
> -----------------------------
>
> As an aside, the application I want to create would be used as a "come as you are" desktop in a first-responder or disaster support/recovery environment. There would be services available for accessing various bits of information, and a complex "this is the state of the event" display would need to be presentable on all applicable "clients". Different people would have access to different types of services based on their responsibilities.
>
> I'd like to use netbeans to create this desktop so that the serviceUIs could say "put me in a tab" or "I'm an editor pane" or "I'm an output pane" etc. This would make it much easier for the serviceUI assist the user in constructing the right display layout. This would also allow roving managers or others actively moving through an event to know where to find information on various clients/displays they would walk past.
>
>
> On Aug 23, 2012, at 1:25 PM, Greg Trasuk <tr...@stratuscom.com> wrote:
>
>
>> I can't speak for Gregg, but as I understood it, the
>> CodebaseAccessClassLoader was an attempt to host clients to Jini
>> services and/or Jini services themselves inside Netbeans. That's
>> unrelated to developing services in an IDE. Certainly the IDE doesn't
>> care what you do with the '.class' files it produces.
>>
>> Speaking for myself (and I've thought about this a fair bit), the real
>> issue from the developer's perspective is ease of running the code that
>> you write. Is it actually necessary to load a Jini service proxy inside
>> the very complicated, modularized classloader environment that Netbeans
>> implements? It would be convenient if it were easy to do that,
>> certainly, but it isn't necessary.
>>
>> Back in the dark ages, we used Ant to create a '.war' file and copy that
>> file into Tomcat's deployment directory, where a standalone instance of
>> Tomcat (unaware of either Ant or an IDE) would note the '.war' file's
>> presence and deploy it in the container. Many servlet-style IDE's do
>> this today.
>>
>> In other words, there's no need for a Jini client inside Netbeans or
>> Eclipse if you just want to deploy stuff.
>>
>> One level up is to have a remote control interface between the IDE and
>> the deployment environment. Tomcat's admin application allows for
>> this. IBM's Rational Application Developer makes web service calls to
>> the Websphere instance. The IDE could then monitor and control parts of
>> the container. This is useful for things like deploying applications
>> "in-place" in the IDE's build directory rather than packaging and
>> deploying. I think this is how Netbeans now interacts with Tomcat and
>> Glassfish. I can't think offhand of an example where the IDE interface
>> does nomitoring, though - it seems to be more common to have an
>> administration application on the server, although often the IDE can
>> open a browser to the admin page. This might be different on a Jini
>> server because there isn't a natural servlet environment to run an
>> application in.
>>
>> If the goal is for the IDE to "control" the server, I don't have a
>> problem with them using a traditional non-Jini protocol (i.e. no
>> downloaded proxy - they need to agree on the protocol) to do it, since
>> they're going to be pretty tightly coupled anyway.
>>
>> Second level-up would be to have visibility into the Jini workgroup from
>> inside Netbeans or eclipse. In other words, could we have the service
>> browser functionality inside Netbeans? Again, it would be convenient to
>> be able to host a Jini client inside Netbeans if it were easy. However,
>> I think what Gregg W proved with CodebaseAccessClassLoader is that it
>> isn't that easy. Again, I think a separate executable that Netbeans
>> talks to using a plain socket protocol would prove easier than trying to
>> navigate Netbeans' modular classloader hierarchy. This would also be a
>> good example of where the surrogate spec would come in handy. Netbeans
>> could ask a surrogate host to launch a surrogate that talks to the Djinn
>> on its behalf.
>>
>> You could make an argument similar to the one in "A Note on Distributed
>> Computing" (Waldo et al) that the idea of serializing an arbitrary
>> object for transmission over the network is a bit of a fool's errand. I
>> think it seems perfectly reasonable that the set of classes used in a
>> service interface could be distributed as a jar file and subjected to
>> external governance.
>>
>> To summarize, there's no reason to talk about serialization protocols,
>> further extensions to codebase annotations, or replacing RMIClassLoader
>> if the goal is to develop services using an IDE.
>>
>> Does a service container need a classloader where you can do mystical
>> things with the codebase and classpath? You betcha! But that's the
>> container's problem, and it probably has complexities that are more
>> about the container design than modular classpaths. (If you're curious,
>> have a look in Surrogate container under
>> 'org.apache.river.container.classloading')
>>
>> A better question, perhaps, is "what environments should we consider as
>> clients to business services implemented using River?" For that, I'd
>> argue the answers "Plain Java", "Servlet environments", and "Other Jini
>> Services" are the clear winners. I don't see a lot of evidence of
>> OSGI-based environments for applications in the real world (by which I
>> mean regular 'Joe Corporate' developers deploying OSGI bundles. Yes, I
>> accept that servers and IDE's may use an OSGI container as their
>> implementation environment).
>>
>> Hosting Jini services in anything besides a Jini-centric environment
>> (whether it's the 'ServiceStarter' utility or a Jini-centric container)
>> is guaranteed to be more trouble than it's worth, IMO.
>>
>> Cheers,
>>
>> Greg.
>>
>>
>>
>> On Thu, 2012-08-23 at 08:54, Peter Firmstone wrote:
>>
>>> Enabling developers to develop services in IDE's like Netbeans was the
>>> original motivator (Gregg Wonderly), see
>>>
>>> https://issues.apache.org/jira/browse/RIVER-336
>>>
>>> Regards,
>>>
>>> Peter.
>>>
>>> On 23/08/2012 8:57 PM, Simon IJskes - QCG wrote:
>>>
>>>> On 19-08-12 14:09, Peter Firmstone wrote:
>>>>
>>>>
>>>>> Thoughts?
>>>>>
>>>> What is the main driver for the RMIClassLoader replacement?
>>>>
>>>> Gr. Sim
>>>>
>>>>
>
>
>
Re: Develop new spec for RMIClassLoader replacement
Posted by GREGG WONDERLY <gr...@gmail.com>.
I want to use the netbeans modular application framework as my service UI host.
That is, I want to be able to use all the features of the RCP to create a complex service UI based client environment. Thus, I want to be able to download and run serviceUIs inside of netbeans, and that's why I need to be able to manage how classes are resolved relative to the modularization of netbeans.
I could then facilitate, as well, interaction with service management UIs so that we could have a River admin module that could be used in the IDE to talk to service container management endpoints.
No matter whether the proxy is in the client or in another process, for the IDE to talk to it, you have to have the same "level" of object compatibility and communication, to make use of the services exported endpoint and/or serviceUI.
The changes I made were not hard, nor complex. There are relatively few places that the interactions with RMIClassLoaderSPI occur, and with JINI 2.0 and PreferredClassLoader, RMIClassLoaderSPIs only useful remaining activity is to convey the class annotation across the wire.
As I said, what I needed was to set the parent class loader for the downloaded codes context class loader, specifically for netbeans.
What I've learned over the years, is that there are lot of subtle yet detrimental assumptions about the environment embedded in Jini/RMI. Many of the things I've run up against, don't add value or otherwise simplify the developers life. Rather, they complicate things, and leave little flexibility in deployment scenarios.
This, I feel, is the number one reason why Jini has never gained much traction. It's hard to "just use it"! We have had lots of examples of this over the years on the mailing lists and in conversations with people at JavaOne and Jini community meetings.
At this point, changing "everything" to remove the issues, is what I would find stimulating enough to fully engage.
At this point in my life, I've spent more than a decade using Jini for everything I've done in service based applications. It's been a long hard road to provide my customers with something that worked well enough in their fragile computing environments, that they could be productive. Many just want a web page, because they think that their browser is the only "desktop" they need.
Sending code and marshaling and unmarshaling objects is a huge, huge barrier to some. Many of us have done this enough, that its just old hat. People love to "buy" XML marshaling and use http for transport. It seems easy because there are tools that have been written on both ends and in the middle to simplify the developers life.
Because we've always just used ServiceStarter (well not everyone, but that's all the community seems to want to support as a part of the "platform"), we have an extremely primitive and highly configuration dependent "platform".
For many, that seems to be a giant turn off. People want finite choices and "GUIs" that enumerate all the choices and the ability to just click and pick a solution. That's where tooling is at in this day and age.
So, that's what I still would like to achieve with River. There are far more choices then necessary, visible to new developers. The majority of that flexibility is not needed to get started, and it's possible to put tooling together that would make it much easier.
Dennis's work on Rio has gone largely ignored, even though it has so much really useful automation and management in it.
We lost out with Seven as well. Mark is not around any longer, really. His life situation did not allow him to do all the work that was needed to put something together for public use, and so we ended up without that simplification of the service deployment scenario either.
There are several different JavaEE deployment tool sets that have been crafted as well.
I think its very clear that there are a lot of different ideas that have been layed out, implemented and deployed, which did in fact demonstrate improvements.
With all of that behind us now, I'd really like to see something really powerful happen with a change in how we view River and how we want to present it to the world, anew.
Our community is full of lots of ideas and experiences, and I don't want to diminish any of that by the statements I made above. I'm just trying to relate what I've seen and where I'd still like to see something happen.
Gregg
-----------------------------
As an aside, the application I want to create would be used as a "come as you are" desktop in a first-responder or disaster support/recovery environment. There would be services available for accessing various bits of information, and a complex "this is the state of the event" display would need to be presentable on all applicable "clients". Different people would have access to different types of services based on their responsibilities.
I'd like to use netbeans to create this desktop so that the serviceUIs could say "put me in a tab" or "I'm an editor pane" or "I'm an output pane" etc. This would make it much easier for the serviceUI assist the user in constructing the right display layout. This would also allow roving managers or others actively moving through an event to know where to find information on various clients/displays they would walk past.
On Aug 23, 2012, at 1:25 PM, Greg Trasuk <tr...@stratuscom.com> wrote:
>
> I can't speak for Gregg, but as I understood it, the
> CodebaseAccessClassLoader was an attempt to host clients to Jini
> services and/or Jini services themselves inside Netbeans. That's
> unrelated to developing services in an IDE. Certainly the IDE doesn't
> care what you do with the '.class' files it produces.
>
> Speaking for myself (and I've thought about this a fair bit), the real
> issue from the developer's perspective is ease of running the code that
> you write. Is it actually necessary to load a Jini service proxy inside
> the very complicated, modularized classloader environment that Netbeans
> implements? It would be convenient if it were easy to do that,
> certainly, but it isn't necessary.
>
> Back in the dark ages, we used Ant to create a '.war' file and copy that
> file into Tomcat's deployment directory, where a standalone instance of
> Tomcat (unaware of either Ant or an IDE) would note the '.war' file's
> presence and deploy it in the container. Many servlet-style IDE's do
> this today.
>
> In other words, there's no need for a Jini client inside Netbeans or
> Eclipse if you just want to deploy stuff.
>
> One level up is to have a remote control interface between the IDE and
> the deployment environment. Tomcat's admin application allows for
> this. IBM's Rational Application Developer makes web service calls to
> the Websphere instance. The IDE could then monitor and control parts of
> the container. This is useful for things like deploying applications
> "in-place" in the IDE's build directory rather than packaging and
> deploying. I think this is how Netbeans now interacts with Tomcat and
> Glassfish. I can't think offhand of an example where the IDE interface
> does nomitoring, though - it seems to be more common to have an
> administration application on the server, although often the IDE can
> open a browser to the admin page. This might be different on a Jini
> server because there isn't a natural servlet environment to run an
> application in.
>
> If the goal is for the IDE to "control" the server, I don't have a
> problem with them using a traditional non-Jini protocol (i.e. no
> downloaded proxy - they need to agree on the protocol) to do it, since
> they're going to be pretty tightly coupled anyway.
>
> Second level-up would be to have visibility into the Jini workgroup from
> inside Netbeans or eclipse. In other words, could we have the service
> browser functionality inside Netbeans? Again, it would be convenient to
> be able to host a Jini client inside Netbeans if it were easy. However,
> I think what Gregg W proved with CodebaseAccessClassLoader is that it
> isn't that easy. Again, I think a separate executable that Netbeans
> talks to using a plain socket protocol would prove easier than trying to
> navigate Netbeans' modular classloader hierarchy. This would also be a
> good example of where the surrogate spec would come in handy. Netbeans
> could ask a surrogate host to launch a surrogate that talks to the Djinn
> on its behalf.
>
> You could make an argument similar to the one in "A Note on Distributed
> Computing" (Waldo et al) that the idea of serializing an arbitrary
> object for transmission over the network is a bit of a fool's errand. I
> think it seems perfectly reasonable that the set of classes used in a
> service interface could be distributed as a jar file and subjected to
> external governance.
>
> To summarize, there's no reason to talk about serialization protocols,
> further extensions to codebase annotations, or replacing RMIClassLoader
> if the goal is to develop services using an IDE.
>
> Does a service container need a classloader where you can do mystical
> things with the codebase and classpath? You betcha! But that's the
> container's problem, and it probably has complexities that are more
> about the container design than modular classpaths. (If you're curious,
> have a look in Surrogate container under
> 'org.apache.river.container.classloading')
>
> A better question, perhaps, is "what environments should we consider as
> clients to business services implemented using River?" For that, I'd
> argue the answers "Plain Java", "Servlet environments", and "Other Jini
> Services" are the clear winners. I don't see a lot of evidence of
> OSGI-based environments for applications in the real world (by which I
> mean regular 'Joe Corporate' developers deploying OSGI bundles. Yes, I
> accept that servers and IDE's may use an OSGI container as their
> implementation environment).
>
> Hosting Jini services in anything besides a Jini-centric environment
> (whether it's the 'ServiceStarter' utility or a Jini-centric container)
> is guaranteed to be more trouble than it's worth, IMO.
>
> Cheers,
>
> Greg.
>
>
>
> On Thu, 2012-08-23 at 08:54, Peter Firmstone wrote:
>> Enabling developers to develop services in IDE's like Netbeans was the
>> original motivator (Gregg Wonderly), see
>>
>> https://issues.apache.org/jira/browse/RIVER-336
>>
>> Regards,
>>
>> Peter.
>>
>> On 23/08/2012 8:57 PM, Simon IJskes - QCG wrote:
>>> On 19-08-12 14:09, Peter Firmstone wrote:
>>>
>>>> Thoughts?
>>>
>>> What is the main driver for the RMIClassLoader replacement?
>>>
>>> Gr. Sim
>>>
>
Re: Develop new spec for RMIClassLoader replacement
Posted by Greg Trasuk <tr...@stratuscom.com>.
I can't speak for Gregg, but as I understood it, the
CodebaseAccessClassLoader was an attempt to host clients to Jini
services and/or Jini services themselves inside Netbeans. That's
unrelated to developing services in an IDE. Certainly the IDE doesn't
care what you do with the '.class' files it produces.
Speaking for myself (and I've thought about this a fair bit), the real
issue from the developer's perspective is ease of running the code that
you write. Is it actually necessary to load a Jini service proxy inside
the very complicated, modularized classloader environment that Netbeans
implements? It would be convenient if it were easy to do that,
certainly, but it isn't necessary.
Back in the dark ages, we used Ant to create a '.war' file and copy that
file into Tomcat's deployment directory, where a standalone instance of
Tomcat (unaware of either Ant or an IDE) would note the '.war' file's
presence and deploy it in the container. Many servlet-style IDE's do
this today.
In other words, there's no need for a Jini client inside Netbeans or
Eclipse if you just want to deploy stuff.
One level up is to have a remote control interface between the IDE and
the deployment environment. Tomcat's admin application allows for
this. IBM's Rational Application Developer makes web service calls to
the Websphere instance. The IDE could then monitor and control parts of
the container. This is useful for things like deploying applications
"in-place" in the IDE's build directory rather than packaging and
deploying. I think this is how Netbeans now interacts with Tomcat and
Glassfish. I can't think offhand of an example where the IDE interface
does nomitoring, though - it seems to be more common to have an
administration application on the server, although often the IDE can
open a browser to the admin page. This might be different on a Jini
server because there isn't a natural servlet environment to run an
application in.
If the goal is for the IDE to "control" the server, I don't have a
problem with them using a traditional non-Jini protocol (i.e. no
downloaded proxy - they need to agree on the protocol) to do it, since
they're going to be pretty tightly coupled anyway.
Second level-up would be to have visibility into the Jini workgroup from
inside Netbeans or eclipse. In other words, could we have the service
browser functionality inside Netbeans? Again, it would be convenient to
be able to host a Jini client inside Netbeans if it were easy. However,
I think what Gregg W proved with CodebaseAccessClassLoader is that it
isn't that easy. Again, I think a separate executable that Netbeans
talks to using a plain socket protocol would prove easier than trying to
navigate Netbeans' modular classloader hierarchy. This would also be a
good example of where the surrogate spec would come in handy. Netbeans
could ask a surrogate host to launch a surrogate that talks to the Djinn
on its behalf.
You could make an argument similar to the one in "A Note on Distributed
Computing" (Waldo et al) that the idea of serializing an arbitrary
object for transmission over the network is a bit of a fool's errand. I
think it seems perfectly reasonable that the set of classes used in a
service interface could be distributed as a jar file and subjected to
external governance.
To summarize, there's no reason to talk about serialization protocols,
further extensions to codebase annotations, or replacing RMIClassLoader
if the goal is to develop services using an IDE.
Does a service container need a classloader where you can do mystical
things with the codebase and classpath? You betcha! But that's the
container's problem, and it probably has complexities that are more
about the container design than modular classpaths. (If you're curious,
have a look in Surrogate container under
'org.apache.river.container.classloading')
A better question, perhaps, is "what environments should we consider as
clients to business services implemented using River?" For that, I'd
argue the answers "Plain Java", "Servlet environments", and "Other Jini
Services" are the clear winners. I don't see a lot of evidence of
OSGI-based environments for applications in the real world (by which I
mean regular 'Joe Corporate' developers deploying OSGI bundles. Yes, I
accept that servers and IDE's may use an OSGI container as their
implementation environment).
Hosting Jini services in anything besides a Jini-centric environment
(whether it's the 'ServiceStarter' utility or a Jini-centric container)
is guaranteed to be more trouble than it's worth, IMO.
Cheers,
Greg.
On Thu, 2012-08-23 at 08:54, Peter Firmstone wrote:
> Enabling developers to develop services in IDE's like Netbeans was the
> original motivator (Gregg Wonderly), see
>
> https://issues.apache.org/jira/browse/RIVER-336
>
> Regards,
>
> Peter.
>
> On 23/08/2012 8:57 PM, Simon IJskes - QCG wrote:
> > On 19-08-12 14:09, Peter Firmstone wrote:
> >
> >> Thoughts?
> >
> > What is the main driver for the RMIClassLoader replacement?
> >
> > Gr. Sim
> >
Re: Develop new spec for RMIClassLoader replacement
Posted by Peter Firmstone <ji...@zeus.net.au>.
Enabling developers to develop services in IDE's like Netbeans was the
original motivator (Gregg Wonderly), see
https://issues.apache.org/jira/browse/RIVER-336
Regards,
Peter.
On 23/08/2012 8:57 PM, Simon IJskes - QCG wrote:
> On 19-08-12 14:09, Peter Firmstone wrote:
>
>> Thoughts?
>
> What is the main driver for the RMIClassLoader replacement?
>
> Gr. Sim
>
Re: Develop new spec for RMIClassLoader replacement
Posted by Simon IJskes - QCG <si...@qcg.nl>.
On 19-08-12 14:09, Peter Firmstone wrote:
> Thoughts?
What is the main driver for the RMIClassLoader replacement?
Gr. Sim