You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@river.apache.org by Peter <ji...@zeus.net.au> on 2017/02/11 11:21:28 UTC
OSGi NP Complete Was: OSGi - deserialization remote invocation strategy
In a word, ServiceDiscoveryManager
ServiceDiscoveryManager is the solution.
ServiceDiscoveryManager performs discovery and looks up services from
registrars based on filters. ServiceDiscoveryManager then performs
local filtering. This allows time for proxy bundles to be installed,
resolve, started and confirmed type compatible, prior to them being made
available (via OSGi service registry if you so desire) for client use.
The new interfaces that are part of JGDMS that I'd like to see their way
into River, found here:
https://github.com/pfirmstone/JGDMS/blob/Maven_build/modularize/JGDMS/jgdms-lib-dl/src/main/java/net/jini/lookup/SafeServiceRegistrar.java
https://github.com/pfirmstone/JGDMS/blob/Maven_build/modularize/JGDMS/jgdms-lib-dl/src/main/java/net/jini/lookup/ServiceAttributesAccessor.java
https://github.com/pfirmstone/JGDMS/blob/Maven_build/modularize/JGDMS/jgdms-lib-dl/src/main/java/net/jini/lookup/ServiceCodebaseAccessor.java
https://github.com/pfirmstone/JGDMS/blob/Maven_build/modularize/JGDMS/jgdms-lib-dl/src/main/java/net/jini/lookup/ServiceIDAccessor.java
https://github.com/pfirmstone/JGDMS/blob/Maven_build/modularize/JGDMS/jgdms-lib-dl/src/main/java/net/jini/lookup/ServiceProxyAccessor.java
ServiceCodebaseAccessor is also used as part of secure discovery, but
the codebase string and certs are transferred as primitives over the
network.
In this case codebase annotations don't need to be included in the
stream, the JERI endpoints don't need them at all.
How so?
We can use ServiceItemFilter and ProxyPreparer to install, resolve and
start out proxy codebase, before downloading the proxy. The interfaces
listed above allow an array bootstrap proxy's (java.lang.reflect.Proxy)
to be obtained from SafeServiceRegistrar.
Firstly the bootstrap proxy's JERI endpoint will be loaded in the
ServiceDiscoveryManager's ClassLoader, so after we've retrieved the
codebase annotation and signers, created a bundle for the proxy,
resolved it's dependencies (via OSGi resolution and repository
services), we need to remarshall the bootstrap proxy into a
MarshalledInstance, then unmarshall it using the ClassLoader of the
recently started proxy bundle. Then when we cast the bootstrap proxy to
ServiceProxyAccessor and retrieve the smart proxy, it will be loaded
into the same ClassLoader that the bootstrap proxy uses, our newly
provisioned and loaded bundle (the correct ClassLoader) without need to
serialize any annotations, then the smart proxy can have constraints
applied etc and be registered as an OSGi service with the OSGi
registrar, where client code can interact with the remote proxy.
Now if public Serializable classes that are imported by the proxy's
bundle (service api) or private classes in the proxy's bundle can be
deserialized and the JERI endpoint has a reference to the ClassLoader of
the proxy.
This should be good enough so we don't require the "bundle stack"
proposed earlier, which also saves the need to explain it and simplifies
the solution (the intent of the bundle stack was to allow
deserialization of private classes within other bundles whose packages
have been imported by the proxy).
The client won't be able to pass a smart proxy to the service (like a
Listener), but it can still pass a non smart proxy and it will still
function. So clients can still export their own remote service (albiet
without a codebase, excluding smart proxy's), but it'll be good enough
for a listener etc.
Cheers,
Peter.
On 8/02/2017 1:09 PM, Niclas Hedhman wrote:
> Maybe there are some misunderstanding somewhere... see below;
>
> On Wed, Feb 8, 2017 at 3:35 AM, Peter<ji...@zeus.net.au> wrote:
>> I'm currently only considering OSGi server -> OSGi client. Mick's
> investigating all four options.
>
> Ok, makes it a lot easier for me to follow.
>
>> Not expecting the client calling bundle to resolve everything, hence the
>> stack, so we have the full visibility of the bundle of the class that was
>> last resolved, so we can resolve its fields from it's bundle. Eg it
>> might import packages the client does not.
> What is "client calling bundle"? I assume that it is the bundle that
> contains the service lookup...
>
> First of all, is the OSGi integration intending to be at "OSGi service
> level" ? If so, the Jini support bundle(s) in OSGi, will monitor Jini
> service registrations and register those in the local OSGi registry and all
> clients would end up being rather ignorant about the remote lookup. It is
> also possible to "be involved" in the service lookup (and registration)
> through the Service Hook API (I might got the name of it wrong), so that
> only when services are requested or service listeners are registered, does
> the Jini support kick in. And this can therefore work both ways...
>
> So, that said; I agree that the "client calling bundle" does not resolve
> anything. And I would go further and say; IF the client calling bundle
> looks up Jini services directly, all bets are off. This introduces some
> workable constraints of what could happen.
>
>
>> The "exact version" thing (only applies to the proxy bundle as we
>> expect the framework to load its deps) can be relaxed to compatible
>> versions to increase class sharing if you think it helps. The proxy
> bundle
>> doesn't export anything at the client, only the server, it just seems to
>> make sense to keep the latest proxy communicating in case that last
>> bug fix release addresses a security issue. All proxy classes are
>> implementation only classes.
> Yes, correct. The 'temporary' proxy bundle should not export any packages.
>
>> Because the proxy bundle manifest declares version import ranges,
>> I'm expecting the framework to favour already loaded bundles to
>> satisfy package import deps.
> Uhhhh... The OSGi framework doesn't "auto load" bundles. It is an explicit
> step. There are many "bundle loaders" around, such as Apache Karaf
> "Features" and "deploy/" directory. Most frameworks can also be instructed
> to load bundles at boot. So there is no "favor already loaded bundles". If
> there is no bundle satisfying the Import-Package, the bundle being resolved
> will not go to RESOLVED state. If there are many bundles (quite typical)
> satisfying an Import-Package, with all its additional contraints (versions,
> attributes, uses, ...), then you enter the NP-complete problem that Michal
> mentioned, finding a combination of wiring that satisfy as many bundles as
> possible. This is a problem mainly due to "uses" (since we seldom use
> attributes), where graphs of types must end up matching in the class space.
>
> See (especially) section 3.7 in OSGi 6.0 Core Specification. If anyone can
> get their head around those details in the first pass, it is you Peter. Not
> easy reading...
>
>
>> If the client is matching service api with the correct import package
>> version ranges (requirements defined by entry's), the proxy bundle
>> should find the service api and other imported packages are already
>> loaded. Eg the client may use the requirements to use the resource
>> service or whatever the new bundle repository standard service is
>> called now to preload the requirements. The client may also perform
>> upgrades before downloading a service.
> I think this is a misunderstanding as well. By doing what I wrote in the
> beginning (listen on Jini Service Registrations and register "something"
> (Remote proxy or a local proxy) in the OSGi registry) then the client
> bundle doesn't need to know anything. Also, the Jini support bundle gets
> plenty of information both from the OSGi Registry as well as the Jini
> Registry. So, when something disappears from Reggie, remove it from the
> local Registry and vice versa.
>
> Now, the deserialization of the Reggie proxy should detect version changes
> and update a cache. And I think that Paremus idea of "Bundle Garbage
> Collection" is sound, but something for later discussion. Point being; No
> need to figure out what can and can not be unloaded. ALSO, since OSGi
> mandates intermittent service availability, most OSGi applications are
> reasonably capable to handle that the Jini service will "disappear" and is
> required to release any held references to the object(s) so that regular GC
> can toss out the classes and classloader when bundle is unloaded.
>
>> In the majority of cases I don't think there's going to be much state
>> in the smart proxy that can't be loaded via the smart proxy bundle
>> and it's package imports, except for the odd handback, which the
>> client bundle should have the opportunity to resolve before
>> resorting to using an annotation.
> Sorry, I don't understand this statement.
>
>> I'm not quite ready to agree it's too complex and it's unsolveable, I
>> think we should at least explore it and understand it before we junk
>> the idea of supporting OSGi.
> If we are talking osgi<->osgi, I think there is reasonable chance to
> succeed.
>
>> Rather than utilise the Java2 class loading I was planning to cast
>> ClassLoaders to BundleReference where appropriate and utilise
>> the Bundle.
> My gut says that this is not needed, if you go with my initial proposal. A
> Jini support bundle ends up having access to the BundleContext, from where
> everything else can be reached.
>
>> I did notice you're interpretation of what I've written is different than
>> mine, so I think I need to put some effort into communicating more
>> effectively. I think you're interpretation of codebase annotation
>> "version is fixed" ignores that the annotation is only consulted after
>> determining that the current class is not available in our Bundles
>> currently participating in deserialization. It doesn't apply to resolved
>> imported packages as annotations aren't used for them at all.
> OR, I have been burned in class resolution in OSGi enough times to have a
> feeling that it is more difficult than it seems. Any simple example I can
> think of would work...
>
>> For example, the first class we attempt to resolve during unmarshalling
>> belongs to a smart proxy, the client Bundle can't find the class. Ask the
>> framework to load the proxy bundle from the codebase annotation, it
>> does so and resolves all necessary package imports declared in its
>> manifest. We now continue deserializing the smart proxy class fields
>> with the visibility of the smart proxy's bundle. The smart proxy may
>> contain fields referencing objects resolved from its imports, we ensure
>> those classes deserialize their fields with the visibility of their own
> bundles.
>
> So, the smart proxy's "bundle" is a bundle on the server side as well? The
> smart proxy may also contain objects that are of a class that is not
> visible... And it may be N levels deep from the "field" in the smart proxy.
>
>> Every time we can't resolve a class we first check if it's a handback
>> or parameter from a preceeding object in the graph, thus we walk
>> our graphs bundle stack.
> This is probably the bit I don't understand at all. On one hand you want to
> depend on OSGi framework to do the resolution, but OTOH you have something
> called a "bundle stack"? What is that?
>
>> If we still haven't resolved a class only then do we load a bundle from
>> it's codebase annotation url and check it can be cast to the field
>> before assigning it. If it can't be cast to that field, we throw an
> exception.
>
> "assigning it"? Doesn't the code in reality looks something like;
>
> private Map map;
>
> private void readObject( ObjectInputStream in ) {
> map = (Map) in.readObject();
> }
>
> where the read object may be of a class not visible to neither the smart
> proxy's classloader nor any helper? Maybe you meant that it will implicitly
> thrown just by the above code.
>
>
>> In the case of a non smart proxy, there is no codebase, deserialization
>> will be loaded by and rely completely on the visibility of the client
> bundle.
>> I think OSGi will be a lot less dependant on annotations than say a std
> env.
>
> Possibly... I think that largely depend on the usecase.
>
>> Still I guess wiring may be an an option, so as Michael suggests,
> annotate objects with their wiring graphs.
>
> Ok, here is what I see being the issue at hand; you think that it is
> possible to delay the bundle resolution of packages until deserialization
> itself. I think that is not possible. There is a need to bring all
> "non-available classes" into the client (and I think as a bundle is the
> correct solution). So, pick up the bundle reference of the smart proxy, and
> the bundle wiring graph of it. When the smart proxy arrives on the client,
> before doing anything else, load the bundles on the client. After that, it
> should be a "local JVM" problem, both to deserialize the smart proxy as
> well as every object communicated over the network.
>
>> What would we be considering if we hadn't been pre exposed to codebase
> annotations?
>> Standard deserialization uses one classpath, each bundle has its own
> unique classpath.
>
> It is not only about classpath. It is about class space and visibility as
> well. And serialization needs to bypass visibility, just like it bypassed
> other constraints before (such as bypassing constructors and initializing
> final fields), and that is a separate issue than the bundle loading
> mechanism.
>
>
> Cheers
> --
> Niclas Hedhman, Software Developer
> http://polygene.apache.org - New Energy for Java
>
Re: OSGi NP Complete Was: OSGi - deserialization remote invocation
strategy
Posted by Peter <ji...@zeus.net.au>.
Thanks Michal,
See inline below.
On 12/02/2017 7:23 AM, Micha\u0142 K\u0142eczek wrote:
> I am sorry but I think that to solve various issues we need to make
> sure fundamentals are right:
>
> 1. There is NO such a thing as "reflective non-smart" proxy - EVERY
> proxy is "smart" (even if it is "reflective") - there is an
> InvocationHandler down there, isn't there?
Easy fixed, we change the terminology to a proxy without a codebase
download, "dynamic proxy" was a term that Sun used for it previously.
Sun used the terminology "Smart Proxy" to describe a proxy that had a
codebase. Dynamic proxy's still have an InvocationHandler, it's part of
one of the existing jeri implementations and already installed at both ends.
> 2. Solving this on service discovery level is trying to do it on the
> WRONG level of abstraction. Services DEPEND on class loading - not the
> other way around.
Can you explain a little more please?
> 3. What you propose is a partial "solution". Not being able to
> register "smart" event listeners means no custom endpoints for example
> (UDPEndpoint anyone?)
Yes, you can still have custom endpoints, but the endpoints need to be
installed at both ends, prior. Initially, smart proxy's cannot be used
for remote objects, only exported services.
> 4. Trying to squeeze partial solutions into the framework is IMHO a
> BIG no no.
> This is simply creating more code, more maintenance burden and more
> headache for users trying to workaround "edge, unsupported cases".
Not really, it provides working solutions for the majority of cases,
with minimal changes, low hanging fruit so to speak and allows adoption,
the other cases can be worked on later, when there are more users and
more developers to assist.
The proxy implementation (provider) is abstracted behind the service api
completely, typical of OSGi practise, ServiceDiscoveryManager allows
time for service proxy's to be wired up, prior to deserialization using
delayed unmarshalling.
The consumer must use the service api to interact with the service and
must honour the service api documentation, which may state which classes
may be overridden, that providers must interact using certain classes in
the local jvm and not transfer them remotely, a good way to do that is
ensure they're not serializable. A listener is intended to be
implemented, however the consumer would only be able to export a remote
object that doesn't have a codebase and still be able expect the
provider to deserialize it.
Unless of course we create a new service that allows a client to request
a bundle be installed, resolved and started in a remote end, allowing
consumers to request remote provisioning. Still I'd like to allow the
solution to grow, we shouldn't expect to be able to do it all at once.
We can still agree that at some point addressing the remaining
functionality would be nice to have.
>
> Please - lets try to come up with the RIGHT solution that is going to
> REALLY fix class loading issues.
Can you explain a little more please? I'm only trying to provide
support for OSGi users at this stage, so might not be considering the
problem you want solved. At least for the OSGi user, there shouldn't be
any class loader issues, each endpoint has been designated a ClassLoader
and it only needs the visibility it's been given in order to provide the
service, this also has security benefits as an attacker is no longer
given access to every class in the jvm.
Regards,
Peter.
>
> Thanks,
> Michal
>
> Peter wrote:
>> In a word, ServiceDiscoveryManager
>>
>> ServiceDiscoveryManager is the solution.
>>
>> ServiceDiscoveryManager performs discovery and looks up services from
>> registrars based on filters. ServiceDiscoveryManager then performs
>> local filtering. This allows time for proxy bundles to be installed,
>> resolve, started and confirmed type compatible, prior to them being
>> made available (via OSGi service registry if you so desire) for
>> client use.
>>
>> The new interfaces that are part of JGDMS that I'd like to see their
>> way into River, found here:
>>
>> https://github.com/pfirmstone/JGDMS/blob/Maven_build/modularize/JGDMS/jgdms-lib-dl/src/main/java/net/jini/lookup/SafeServiceRegistrar.java
>>
>>
>> https://github.com/pfirmstone/JGDMS/blob/Maven_build/modularize/JGDMS/jgdms-lib-dl/src/main/java/net/jini/lookup/ServiceAttributesAccessor.java
>>
>> https://github.com/pfirmstone/JGDMS/blob/Maven_build/modularize/JGDMS/jgdms-lib-dl/src/main/java/net/jini/lookup/ServiceCodebaseAccessor.java
>>
>> https://github.com/pfirmstone/JGDMS/blob/Maven_build/modularize/JGDMS/jgdms-lib-dl/src/main/java/net/jini/lookup/ServiceIDAccessor.java
>>
>> https://github.com/pfirmstone/JGDMS/blob/Maven_build/modularize/JGDMS/jgdms-lib-dl/src/main/java/net/jini/lookup/ServiceProxyAccessor.java
>>
>>
>> ServiceCodebaseAccessor is also used as part of secure discovery, but
>> the codebase string and certs are transferred as primitives over the
>> network.
>>
>> In this case codebase annotations don't need to be included in the
>> stream, the JERI endpoints don't need them at all.
>>
>> How so?
>>
>> We can use ServiceItemFilter and ProxyPreparer to install, resolve
>> and start out proxy codebase, before downloading the proxy. The
>> interfaces listed above allow an array bootstrap proxy's
>> (java.lang.reflect.Proxy) to be obtained from SafeServiceRegistrar.
>>
>> Firstly the bootstrap proxy's JERI endpoint will be loaded in the
>> ServiceDiscoveryManager's ClassLoader, so after we've retrieved the
>> codebase annotation and signers, created a bundle for the proxy,
>> resolved it's dependencies (via OSGi resolution and repository
>> services), we need to remarshall the bootstrap proxy into a
>> MarshalledInstance, then unmarshall it using the ClassLoader of the
>> recently started proxy bundle. Then when we cast the bootstrap proxy
>> to ServiceProxyAccessor and retrieve the smart proxy, it will be
>> loaded into the same ClassLoader that the bootstrap proxy uses, our
>> newly provisioned and loaded bundle (the correct ClassLoader) without
>> need to serialize any annotations, then the smart proxy can have
>> constraints applied etc and be registered as an OSGi service with the
>> OSGi registrar, where client code can interact with the remote proxy.
>>
>> Now if public Serializable classes that are imported by the proxy's
>> bundle (service api) or private classes in the proxy's bundle can be
>> deserialized and the JERI endpoint has a reference to the ClassLoader
>> of the proxy.
>>
>> This should be good enough so we don't require the "bundle stack"
>> proposed earlier, which also saves the need to explain it and
>> simplifies the solution (the intent of the bundle stack was to allow
>> deserialization of private classes within other bundles whose
>> packages have been imported by the proxy).
>>
>> The client won't be able to pass a smart proxy to the service (like a
>> Listener), but it can still pass a non smart proxy and it will still
>> function. So clients can still export their own remote service
>> (albiet without a codebase, excluding smart proxy's), but it'll be
>> good enough for a listener etc.
>>
>> Cheers,
>>
>> Peter.
>>
>>
>> On 8/02/2017 1:09 PM, Niclas Hedhman wrote:
>>> Maybe there are some misunderstanding somewhere... see below;
>>>
>>> On Wed, Feb 8, 2017 at 3:35 AM, Peter<ji...@zeus.net.au> wrote:
>>>> I'm currently only considering OSGi server -> OSGi client. Mick's
>>> investigating all four options.
>>>
>>> Ok, makes it a lot easier for me to follow.
>>>
>>>> Not expecting the client calling bundle to resolve everything,
>>>> hence the
>>>> stack, so we have the full visibility of the bundle of the class
>>>> that was
>>>> last resolved, so we can resolve its fields from it's bundle. Eg it
>>>> might import packages the client does not.
>>> What is "client calling bundle"? I assume that it is the bundle that
>>> contains the service lookup...
>>>
>>> First of all, is the OSGi integration intending to be at "OSGi service
>>> level" ? If so, the Jini support bundle(s) in OSGi, will monitor Jini
>>> service registrations and register those in the local OSGi registry
>>> and all
>>> clients would end up being rather ignorant about the remote lookup.
>>> It is
>>> also possible to "be involved" in the service lookup (and registration)
>>> through the Service Hook API (I might got the name of it wrong), so
>>> that
>>> only when services are requested or service listeners are
>>> registered, does
>>> the Jini support kick in. And this can therefore work both ways...
>>>
>>> So, that said; I agree that the "client calling bundle" does not
>>> resolve
>>> anything. And I would go further and say; IF the client calling bundle
>>> looks up Jini services directly, all bets are off. This introduces some
>>> workable constraints of what could happen.
>>>
>>>
>>>> The "exact version" thing (only applies to the proxy bundle as we
>>>> expect the framework to load its deps) can be relaxed to compatible
>>>> versions to increase class sharing if you think it helps. The proxy
>>> bundle
>>>> doesn't export anything at the client, only the server, it just
>>>> seems to
>>>> make sense to keep the latest proxy communicating in case that last
>>>> bug fix release addresses a security issue. All proxy classes are
>>>> implementation only classes.
>>> Yes, correct. The 'temporary' proxy bundle should not export any
>>> packages.
>>>
>>>> Because the proxy bundle manifest declares version import ranges,
>>>> I'm expecting the framework to favour already loaded bundles to
>>>> satisfy package import deps.
>>> Uhhhh... The OSGi framework doesn't "auto load" bundles. It is an
>>> explicit
>>> step. There are many "bundle loaders" around, such as Apache Karaf
>>> "Features" and "deploy/" directory. Most frameworks can also be
>>> instructed
>>> to load bundles at boot. So there is no "favor already loaded
>>> bundles". If
>>> there is no bundle satisfying the Import-Package, the bundle being
>>> resolved
>>> will not go to RESOLVED state. If there are many bundles (quite
>>> typical)
>>> satisfying an Import-Package, with all its additional contraints
>>> (versions,
>>> attributes, uses, ...), then you enter the NP-complete problem that
>>> Michal
>>> mentioned, finding a combination of wiring that satisfy as many
>>> bundles as
>>> possible. This is a problem mainly due to "uses" (since we seldom use
>>> attributes), where graphs of types must end up matching in the class
>>> space.
>>>
>>> See (especially) section 3.7 in OSGi 6.0 Core Specification. If
>>> anyone can
>>> get their head around those details in the first pass, it is you
>>> Peter. Not
>>> easy reading...
>>>
>>>
>>>> If the client is matching service api with the correct import package
>>>> version ranges (requirements defined by entry's), the proxy bundle
>>>> should find the service api and other imported packages are already
>>>> loaded. Eg the client may use the requirements to use the resource
>>>> service or whatever the new bundle repository standard service is
>>>> called now to preload the requirements. The client may also perform
>>>> upgrades before downloading a service.
>>> I think this is a misunderstanding as well. By doing what I wrote in
>>> the
>>> beginning (listen on Jini Service Registrations and register
>>> "something"
>>> (Remote proxy or a local proxy) in the OSGi registry) then the client
>>> bundle doesn't need to know anything. Also, the Jini support bundle
>>> gets
>>> plenty of information both from the OSGi Registry as well as the Jini
>>> Registry. So, when something disappears from Reggie, remove it from the
>>> local Registry and vice versa.
>>>
>>> Now, the deserialization of the Reggie proxy should detect version
>>> changes
>>> and update a cache. And I think that Paremus idea of "Bundle Garbage
>>> Collection" is sound, but something for later discussion. Point
>>> being; No
>>> need to figure out what can and can not be unloaded. ALSO, since OSGi
>>> mandates intermittent service availability, most OSGi applications are
>>> reasonably capable to handle that the Jini service will "disappear"
>>> and is
>>> required to release any held references to the object(s) so that
>>> regular GC
>>> can toss out the classes and classloader when bundle is unloaded.
>>>
>>>> In the majority of cases I don't think there's going to be much state
>>>> in the smart proxy that can't be loaded via the smart proxy bundle
>>>> and it's package imports, except for the odd handback, which the
>>>> client bundle should have the opportunity to resolve before
>>>> resorting to using an annotation.
>>> Sorry, I don't understand this statement.
>>>
>>>> I'm not quite ready to agree it's too complex and it's unsolveable, I
>>>> think we should at least explore it and understand it before we junk
>>>> the idea of supporting OSGi.
>>> If we are talking osgi<->osgi, I think there is reasonable chance to
>>> succeed.
>>>
>>>> Rather than utilise the Java2 class loading I was planning to cast
>>>> ClassLoaders to BundleReference where appropriate and utilise
>>>> the Bundle.
>>> My gut says that this is not needed, if you go with my initial
>>> proposal. A
>>> Jini support bundle ends up having access to the BundleContext, from
>>> where
>>> everything else can be reached.
>>>
>>>> I did notice you're interpretation of what I've written is
>>>> different than
>>>> mine, so I think I need to put some effort into communicating more
>>>> effectively. I think you're interpretation of codebase annotation
>>>> "version is fixed" ignores that the annotation is only consulted after
>>>> determining that the current class is not available in our Bundles
>>>> currently participating in deserialization. It doesn't apply to
>>>> resolved
>>>> imported packages as annotations aren't used for them at all.
>>> OR, I have been burned in class resolution in OSGi enough times to
>>> have a
>>> feeling that it is more difficult than it seems. Any simple example
>>> I can
>>> think of would work...
>>>
>>>> For example, the first class we attempt to resolve during
>>>> unmarshalling
>>>> belongs to a smart proxy, the client Bundle can't find the class.
>>>> Ask the
>>>> framework to load the proxy bundle from the codebase annotation, it
>>>> does so and resolves all necessary package imports declared in its
>>>> manifest. We now continue deserializing the smart proxy class fields
>>>> with the visibility of the smart proxy's bundle. The smart proxy may
>>>> contain fields referencing objects resolved from its imports, we
>>>> ensure
>>>> those classes deserialize their fields with the visibility of their
>>>> own
>>> bundles.
>>>
>>> So, the smart proxy's "bundle" is a bundle on the server side as
>>> well? The
>>> smart proxy may also contain objects that are of a class that is not
>>> visible... And it may be N levels deep from the "field" in the smart
>>> proxy.
>>>
>>>> Every time we can't resolve a class we first check if it's a handback
>>>> or parameter from a preceeding object in the graph, thus we walk
>>>> our graphs bundle stack.
>>> This is probably the bit I don't understand at all. On one hand you
>>> want to
>>> depend on OSGi framework to do the resolution, but OTOH you have
>>> something
>>> called a "bundle stack"? What is that?
>>>
>>>> If we still haven't resolved a class only then do we load a bundle
>>>> from
>>>> it's codebase annotation url and check it can be cast to the field
>>>> before assigning it. If it can't be cast to that field, we throw an
>>> exception.
>>>
>>> "assigning it"? Doesn't the code in reality looks something like;
>>>
>>> private Map map;
>>>
>>> private void readObject( ObjectInputStream in ) {
>>> map = (Map) in.readObject();
>>> }
>>>
>>> where the read object may be of a class not visible to neither the
>>> smart
>>> proxy's classloader nor any helper? Maybe you meant that it will
>>> implicitly
>>> thrown just by the above code.
>>>
>>>
>>>> In the case of a non smart proxy, there is no codebase,
>>>> deserialization
>>>> will be loaded by and rely completely on the visibility of the client
>>> bundle.
>>>> I think OSGi will be a lot less dependant on annotations than say a
>>>> std
>>> env.
>>>
>>> Possibly... I think that largely depend on the usecase.
>>>
>>>> Still I guess wiring may be an an option, so as Michael suggests,
>>> annotate objects with their wiring graphs.
>>>
>>> Ok, here is what I see being the issue at hand; you think that it is
>>> possible to delay the bundle resolution of packages until
>>> deserialization
>>> itself. I think that is not possible. There is a need to bring all
>>> "non-available classes" into the client (and I think as a bundle is the
>>> correct solution). So, pick up the bundle reference of the smart
>>> proxy, and
>>> the bundle wiring graph of it. When the smart proxy arrives on the
>>> client,
>>> before doing anything else, load the bundles on the client. After
>>> that, it
>>> should be a "local JVM" problem, both to deserialize the smart proxy as
>>> well as every object communicated over the network.
>>>
>>>> What would we be considering if we hadn't been pre exposed to codebase
>>> annotations?
>>>> Standard deserialization uses one classpath, each bundle has its own
>>> unique classpath.
>>>
>>> It is not only about classpath. It is about class space and
>>> visibility as
>>> well. And serialization needs to bypass visibility, just like it
>>> bypassed
>>> other constraints before (such as bypassing constructors and
>>> initializing
>>> final fields), and that is a separate issue than the bundle loading
>>> mechanism.
>>>
>>>
>>> Cheers
>>> --
>>> Niclas Hedhman, Software Developer
>>> http://polygene.apache.org - New Energy for Java
>>>
>>
>
>
Re: OSGi NP Complete Was: OSGi - deserialization remote invocation
strategy
Posted by Patricia Shanahan <pa...@acm.org>.
Sorry, I'm trying to find out the meaning of the current subject line.
I'm not sure when it changed to "OSGi MP Complete".
On 2/12/2017 10:50 PM, Micha\u0142 K\u0142eczek wrote:
> Sorry, NP Completness of what?
> I have been the first to mention NP hardness of constraint satisfaction
> problem
> but I am not sure if this is what you are asking about.
>
> Thanks,
> Michal
>
> Patricia Shanahan wrote:
>> Are you literally claiming NP Completeness, or just using that as an
>> analogy for really, really difficult?
>>
>
Re: OSGi NP Complete Was: OSGi - deserialization remote invocation
strategy
Posted by Michał Kłeczek <mi...@kleczek.org>.
Comments inline.
Peter wrote:
> Mic,
>
> I'm attempting to get my head around your proposal:
>
> In the case of JERI, the InvocationHandler is part of the smart
> proxy's serialized state. A number of smart proxy classes will need
> to be unmarshalled before the UnmarshallingInvocationHandler is
> deserialized.
>
> The smart proxy contains a reference to a dynamic proxy (which sun
> called the bootstrap proxy) and the dynamic proxy contains a reference
> to your UnmarshallingInvocationHandler. This means the smart proxy
> must be unmarshalled first.
>
> How do you get access to UnmarshallingInvocationHandler without
> unmarshalling the smart proxy first?
No no - I am saying about wrapping the smart proxy inside another
object. It can be either a dynamic proxy, or simply an object that
implements "readResolve" returning the unmarshalled smart proxy.
>
> More comments inline below.
>
> On 13/02/2017 6:11 PM, Micha\u0142 K\u0142eczek wrote:
>> We are talking about the same thing.
>>
>> We are turning circles, Peter - all of this has been already discussed.
>>
>> 1. Yes - you need to resolve bundles in advance (in OSGi it is not
>> possible to do otherwise anyway)
> Agree.
>> 2. You cannot decide upon the bundle chosen by the container to load
>> the proxy class (the container does the resolution)
> Disagree, nothing in the client depends on the proxy bundle, there's
> no reason to provision a different version.
>> 3. The runtime graph of object places additional constraints on the
>> bundle resolution process (to what is specified in bundles' manifests).
>> Since you do not have any way to pass these additional constraints to
>> the container - the case is lost.
> Disagree. The proxy bundle contains a manifest with requirements.
> The stream has no knowledge of versioning, nor does it need to, there
> are no additional constraints. If the service proxy dependencies
> cannot be resolved, or it doesn't unmarshall, then it will not be
> registered with the OSGi registry in the client, client code will not
> discover it and the client will have no knowledge of it's existance
> except for some logging.
>
This is totally backwards.
That way no client is able to find any service because there is a
chicken and egg problem - we do not know the proxy interfaces until the
proxy's bundle is resolved.
Understand that when you place a bundle identifier in the stream - it is
equivalent to specifying a Require-Bundle constraint - nothing more
nothing less.
>>
>> Additionally - to explain what I've said before about wrong level of
>> abstraction:
>>
>> Your general idea is very similar to mine: have a special object
>> (let's call it installer) that will install software prior to proxy
>> unmarshalling.
>>
>> 1. For some reason unclear to me you want to constrain the way how
>> this "installer object" is passed only via the route of
>> ServiceRegistrar (as attributes)
>
> Disagree, I'm not proposing the service have any control over
> installation at the client, other than the manifest in the proxy
> bundle, nor am I proposing using service attributes, or the use of any
> existing ServiceRegistar methods (see SafeServiceRegistrar link posted
> earlier).
If you think about it from the higher architectural view - there is no
difference. It does not really matter what steps are made - important
thing is that:
a) you have a special object used to download code - this object is
supposed to be of a class installed locally in advance
b) the above object is used to create a ClassLoader that you will use it
load the actual deserialized object's class
It does not matter where the first object is taken from - be it
"SafeServiceRegistrar", the stream itself, a JavaSpace or the Moon.
Thanks,
Michal
Re: OSGi NP Complete Was: OSGi - deserialization remote invocation
strategy
Posted by Peter <ji...@zeus.net.au>.
To be fair, my position changed somewhat after Nic's email and some
further research, it may of course develop further with understanding
and experimentation.
Cheers,
Peter.
On 13/02/2017 7:52 PM, Peter wrote:
> Mic,
>
> I'm attempting to get my head around your proposal:
>
> In the case of JERI, the InvocationHandler is part of the smart
> proxy's serialized state. A number of smart proxy classes will need
> to be unmarshalled before the UnmarshallingInvocationHandler is
> deserialized.
>
> The smart proxy contains a reference to a dynamic proxy (which sun
> called the bootstrap proxy) and the dynamic proxy contains a reference
> to your UnmarshallingInvocationHandler. This means the smart proxy
> must be unmarshalled first.
>
> How do you get access to UnmarshallingInvocationHandler without
> unmarshalling the smart proxy first?
>
> More comments inline below.
>
> On 13/02/2017 6:11 PM, Micha\u0142 K\u0142eczek wrote:
>> We are talking about the same thing.
>>
>> We are turning circles, Peter - all of this has been already discussed.
>>
>> 1. Yes - you need to resolve bundles in advance (in OSGi it is not
>> possible to do otherwise anyway)
> Agree.
>> 2. You cannot decide upon the bundle chosen by the container to load
>> the proxy class (the container does the resolution)
> Disagree, nothing in the client depends on the proxy bundle, there's
> no reason to provision a different version.
>> 3. The runtime graph of object places additional constraints on the
>> bundle resolution process (to what is specified in bundles' manifests).
>> Since you do not have any way to pass these additional constraints to
>> the container - the case is lost.
> Disagree. The proxy bundle contains a manifest with requirements.
> The stream has no knowledge of versioning, nor does it need to, there
> are no additional constraints. If the service proxy dependencies
> cannot be resolved, or it doesn't unmarshall, then it will not be
> registered with the OSGi registry in the client, client code will not
> discover it and the client will have no knowledge of it's existance
> except for some logging.
>
>
>>
>> Additionally - to explain what I've said before about wrong level of
>> abstraction:
>>
>> Your general idea is very similar to mine: have a special object
>> (let's call it installer) that will install software prior to proxy
>> unmarshalling.
>>
>> 1. For some reason unclear to me you want to constrain the way how
>> this "installer object" is passed only via the route of
>> ServiceRegistrar (as attributes)
>
> Disagree, I'm not proposing the service have any control over
> installation at the client, other than the manifest in the proxy
> bundle, nor am I proposing using service attributes, or the use of any
> existing ServiceRegistar methods (see SafeServiceRegistrar link posted
> earlier).
>
>> But why not:
>>
>> public interface RemoteEventProducer {
>> void registerListener(SmartProxyInstaller installer, byte[]
>> remoteEventListenerBytes);
>> }
>
> I guess you could have a service that installs the proxy bundle that
> way, but how will you know which ClassLoader the deserialize into at
> the client?
>
>>
>> I cannot see a difference at all and this is why I say that mixing
>> ServiceRegistrar and ServiceDiscoveryManager
>> into it is really mixing levels of abstraction (and does not add
>> anything to the solution).
>
> I'm proposing to use ServiceDiscoveryManager to discover
> SafeServiceRegistrar's, then look up matching services, receive
> dynamic proxy's from each of the matching services (using input
> validating deserialization, a security measure) that allow the
> ProxyPreparer to authenticate each service, get attributes, perform
> local logical attribute comparisons, get the codebase URL string,
> certificate signers, grant permissions, then finally provision the
> codebase and finally retrieve the smart proxy, directly from the
> dynamic proxy.
>
> Now we could put your UnmarshallingInvocationHandler into the dynamic
> proxy, but then we might be installing a bundle we decide we don't
> want after logical comparisons of attributes. The InvocationHandler's
> intent is to marshall object arguments, when methods are invocated on
> the dynamic proxy. The InvocationDispatcher unmarshalls the arguments
> at the remote end and invokes the methods on the Remote object.
>
> So I'm proposing to write some code that performs service discovery
> using the above and registers the successfully matched services with
> the local OSGi service registry. So the client needn't be concerned
> with the time taken to remotely discover and provision a service, all
> that happens in the background until the services can be made
> available in the OSGi service registry for the client to utilise.
>
>>
>> 2. If you allow to pass "installer" to unmarshall proxies - then the
>> next question is - why do you require doing it explicitly in the
>> application code???
>
> That's not what I'm proposing, see above.
>
> Cheers,
>
> Peter.
>
>> THIS is really mixing levels of abstraction. A programmer expects a
>> natural programming model (and will not buy into Jini if it does not
>> offer such):
>>
>> registerListener(RemoteEventListener listener);
>>
>> without any additional complexities that should be solved by the
>> infrastructure code.
>>
>> 3. But the above is easy to solve:
>>
>> class UnmarshallingInvocationHandler implements InvocationHandler {
>>
>> private Object unmarshalledProxy;
>> private SmartProxyInstaller installer;
>>
>> writeObject(...) { write out installer and proxy bytes }
>> readObject(...) {read installer then bytes and then unmarshall the
>> proxy}
>>
>> }
>>
>> 4. But then the next question is why not put it in the object stream
>> implementation itself???
>>
>> Thanks,
>> Michal
>>
>>
>> Peter wrote:
>>> Also see the OSGi Enterprise specification, v6, Chapter 136, page
>>> 691, there's some discussion about the NP-complete nature of
>>> dependency resolution there as well.
>>>
>>> https://www.osgi.org/developer/downloads/release-6/release-6-download/
>>>
>>> On 13/02/2017 5:19 PM, Peter wrote:
>>>> OSGi Dependency resolution is.
>>>>
>>>> http://underlap.blogspot.com.au/2010/02/osgi-resolution-is-np-complete-so-what.html
>>>>
>>>>
>>>> Which means if we want to support an OSGi environment properly, we
>>>> may need some time to resolve the dependencies for a smart proxy,
>>>> before deserializing the proxy, rather than downloading the proxy
>>>> dynamically during unmarshalling, it's better to delay
>>>> unmarshalling until the dependencies are resolved, so the client
>>>> isn't impacted by delays.
>>>>
>>>> Cheers,
>>>>
>>>> Peter.
>>>>
>>>> On 13/02/2017 4:50 PM, Micha\u0142 K\u0142eczek wrote:
>>>>> Sorry, NP Completness of what?
>>>>> I have been the first to mention NP hardness of constraint
>>>>> satisfaction problem
>>>>> but I am not sure if this is what you are asking about.
>>>>>
>>>>> Thanks,
>>>>> Michal
>>>>>
>>>>> Patricia Shanahan wrote:
>>>>>> Are you literally claiming NP Completeness, or just using that as
>>>>>> an analogy for really, really difficult?
>>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>
>>
>>
>
>
Re: OSGi NP Complete Was: OSGi - deserialization remote invocation
strategy
Posted by Peter <ji...@zeus.net.au>.
Mic,
I'm attempting to get my head around your proposal:
In the case of JERI, the InvocationHandler is part of the smart proxy's
serialized state. A number of smart proxy classes will need to be
unmarshalled before the UnmarshallingInvocationHandler is deserialized.
The smart proxy contains a reference to a dynamic proxy (which sun
called the bootstrap proxy) and the dynamic proxy contains a reference
to your UnmarshallingInvocationHandler. This means the smart proxy
must be unmarshalled first.
How do you get access to UnmarshallingInvocationHandler without
unmarshalling the smart proxy first?
More comments inline below.
On 13/02/2017 6:11 PM, Micha\u0142 K\u0142eczek wrote:
> We are talking about the same thing.
>
> We are turning circles, Peter - all of this has been already discussed.
>
> 1. Yes - you need to resolve bundles in advance (in OSGi it is not
> possible to do otherwise anyway)
Agree.
> 2. You cannot decide upon the bundle chosen by the container to load
> the proxy class (the container does the resolution)
Disagree, nothing in the client depends on the proxy bundle, there's no
reason to provision a different version.
> 3. The runtime graph of object places additional constraints on the
> bundle resolution process (to what is specified in bundles' manifests).
> Since you do not have any way to pass these additional constraints to
> the container - the case is lost.
Disagree. The proxy bundle contains a manifest with requirements. The
stream has no knowledge of versioning, nor does it need to, there are no
additional constraints. If the service proxy dependencies cannot be
resolved, or it doesn't unmarshall, then it will not be registered with
the OSGi registry in the client, client code will not discover it and
the client will have no knowledge of it's existance except for some logging.
>
> Additionally - to explain what I've said before about wrong level of
> abstraction:
>
> Your general idea is very similar to mine: have a special object
> (let's call it installer) that will install software prior to proxy
> unmarshalling.
>
> 1. For some reason unclear to me you want to constrain the way how
> this "installer object" is passed only via the route of
> ServiceRegistrar (as attributes)
Disagree, I'm not proposing the service have any control over
installation at the client, other than the manifest in the proxy bundle,
nor am I proposing using service attributes, or the use of any existing
ServiceRegistar methods (see SafeServiceRegistrar link posted earlier).
> But why not:
>
> public interface RemoteEventProducer {
> void registerListener(SmartProxyInstaller installer, byte[]
> remoteEventListenerBytes);
> }
I guess you could have a service that installs the proxy bundle that
way, but how will you know which ClassLoader the deserialize into at the
client?
>
> I cannot see a difference at all and this is why I say that mixing
> ServiceRegistrar and ServiceDiscoveryManager
> into it is really mixing levels of abstraction (and does not add
> anything to the solution).
I'm proposing to use ServiceDiscoveryManager to discover
SafeServiceRegistrar's, then look up matching services, receive dynamic
proxy's from each of the matching services (using input validating
deserialization, a security measure) that allow the ProxyPreparer to
authenticate each service, get attributes, perform local logical
attribute comparisons, get the codebase URL string, certificate signers,
grant permissions, then finally provision the codebase and finally
retrieve the smart proxy, directly from the dynamic proxy.
Now we could put your UnmarshallingInvocationHandler into the dynamic
proxy, but then we might be installing a bundle we decide we don't want
after logical comparisons of attributes. The InvocationHandler's intent
is to marshall object arguments, when methods are invocated on the
dynamic proxy. The InvocationDispatcher unmarshalls the arguments at
the remote end and invokes the methods on the Remote object.
So I'm proposing to write some code that performs service discovery
using the above and registers the successfully matched services with the
local OSGi service registry. So the client needn't be concerned with
the time taken to remotely discover and provision a service, all that
happens in the background until the services can be made available in
the OSGi service registry for the client to utilise.
>
> 2. If you allow to pass "installer" to unmarshall proxies - then the
> next question is - why do you require doing it explicitly in the
> application code???
That's not what I'm proposing, see above.
Cheers,
Peter.
> THIS is really mixing levels of abstraction. A programmer expects a
> natural programming model (and will not buy into Jini if it does not
> offer such):
>
> registerListener(RemoteEventListener listener);
>
> without any additional complexities that should be solved by the
> infrastructure code.
>
> 3. But the above is easy to solve:
>
> class UnmarshallingInvocationHandler implements InvocationHandler {
>
> private Object unmarshalledProxy;
> private SmartProxyInstaller installer;
>
> writeObject(...) { write out installer and proxy bytes }
> readObject(...) {read installer then bytes and then unmarshall the
> proxy}
>
> }
>
> 4. But then the next question is why not put it in the object stream
> implementation itself???
>
> Thanks,
> Michal
>
>
> Peter wrote:
>> Also see the OSGi Enterprise specification, v6, Chapter 136, page
>> 691, there's some discussion about the NP-complete nature of
>> dependency resolution there as well.
>>
>> https://www.osgi.org/developer/downloads/release-6/release-6-download/
>>
>> On 13/02/2017 5:19 PM, Peter wrote:
>>> OSGi Dependency resolution is.
>>>
>>> http://underlap.blogspot.com.au/2010/02/osgi-resolution-is-np-complete-so-what.html
>>>
>>>
>>> Which means if we want to support an OSGi environment properly, we
>>> may need some time to resolve the dependencies for a smart proxy,
>>> before deserializing the proxy, rather than downloading the proxy
>>> dynamically during unmarshalling, it's better to delay unmarshalling
>>> until the dependencies are resolved, so the client isn't impacted by
>>> delays.
>>>
>>> Cheers,
>>>
>>> Peter.
>>>
>>> On 13/02/2017 4:50 PM, Micha\u0142 K\u0142eczek wrote:
>>>> Sorry, NP Completness of what?
>>>> I have been the first to mention NP hardness of constraint
>>>> satisfaction problem
>>>> but I am not sure if this is what you are asking about.
>>>>
>>>> Thanks,
>>>> Michal
>>>>
>>>> Patricia Shanahan wrote:
>>>>> Are you literally claiming NP Completeness, or just using that as
>>>>> an analogy for really, really difficult?
>>>>>
>>>>
>>>>
>>>
>>>
>>
>
>
Re: OSGi NP Complete Was: OSGi - deserialization remote invocation
strategy
Posted by Michał Kłeczek <mi...@kleczek.org>.
We are talking about the same thing.
We are turning circles, Peter - all of this has been already discussed.
1. Yes - you need to resolve bundles in advance (in OSGi it is not
possible to do otherwise anyway)
2. You cannot decide upon the bundle chosen by the container to load the
proxy class (the container does the resolution)
3. The runtime graph of object places additional constraints on the
bundle resolution process (to what is specified in bundles' manifests).
Since you do not have any way to pass these additional constraints to
the container - the case is lost.
Additionally - to explain what I've said before about wrong level of
abstraction:
Your general idea is very similar to mine: have a special object (let's
call it installer) that will install software prior to proxy unmarshalling.
1. For some reason unclear to me you want to constrain the way how this
"installer object" is passed only via the route of ServiceRegistrar (as
attributes)
But why not:
public interface RemoteEventProducer {
void registerListener(SmartProxyInstaller installer, byte[]
remoteEventListenerBytes);
}
I cannot see a difference at all and this is why I say that mixing
ServiceRegistrar and ServiceDiscoveryManager
into it is really mixing levels of abstraction (and does not add
anything to the solution).
2. If you allow to pass "installer" to unmarshall proxies - then the
next question is - why do you require doing it explicitly in the
application code???
THIS is really mixing levels of abstraction. A programmer expects a
natural programming model (and will not buy into Jini if it does not
offer such):
registerListener(RemoteEventListener listener);
without any additional complexities that should be solved by the
infrastructure code.
3. But the above is easy to solve:
class UnmarshallingInvocationHandler implements InvocationHandler {
private Object unmarshalledProxy;
private SmartProxyInstaller installer;
writeObject(...) { write out installer and proxy bytes }
readObject(...) {read installer then bytes and then unmarshall the proxy}
}
4. But then the next question is why not put it in the object stream
implementation itself???
Thanks,
Michal
Peter wrote:
> Also see the OSGi Enterprise specification, v6, Chapter 136, page 691,
> there's some discussion about the NP-complete nature of dependency
> resolution there as well.
>
> https://www.osgi.org/developer/downloads/release-6/release-6-download/
>
> On 13/02/2017 5:19 PM, Peter wrote:
>> OSGi Dependency resolution is.
>>
>> http://underlap.blogspot.com.au/2010/02/osgi-resolution-is-np-complete-so-what.html
>>
>>
>> Which means if we want to support an OSGi environment properly, we
>> may need some time to resolve the dependencies for a smart proxy,
>> before deserializing the proxy, rather than downloading the proxy
>> dynamically during unmarshalling, it's better to delay unmarshalling
>> until the dependencies are resolved, so the client isn't impacted by
>> delays.
>>
>> Cheers,
>>
>> Peter.
>>
>> On 13/02/2017 4:50 PM, Micha\u0142 K\u0142eczek wrote:
>>> Sorry, NP Completness of what?
>>> I have been the first to mention NP hardness of constraint
>>> satisfaction problem
>>> but I am not sure if this is what you are asking about.
>>>
>>> Thanks,
>>> Michal
>>>
>>> Patricia Shanahan wrote:
>>>> Are you literally claiming NP Completeness, or just using that as
>>>> an analogy for really, really difficult?
>>>>
>>>
>>>
>>
>>
>
Re: OSGi NP Complete Was: OSGi - deserialization remote invocation
strategy
Posted by Peter <ji...@zeus.net.au>.
Also see the OSGi Enterprise specification, v6, Chapter 136, page 691,
there's some discussion about the NP-complete nature of dependency
resolution there as well.
https://www.osgi.org/developer/downloads/release-6/release-6-download/
On 13/02/2017 5:19 PM, Peter wrote:
> OSGi Dependency resolution is.
>
> http://underlap.blogspot.com.au/2010/02/osgi-resolution-is-np-complete-so-what.html
>
>
> Which means if we want to support an OSGi environment properly, we may
> need some time to resolve the dependencies for a smart proxy, before
> deserializing the proxy, rather than downloading the proxy dynamically
> during unmarshalling, it's better to delay unmarshalling until the
> dependencies are resolved, so the client isn't impacted by delays.
>
> Cheers,
>
> Peter.
>
> On 13/02/2017 4:50 PM, Micha\u0142 K\u0142eczek wrote:
>> Sorry, NP Completness of what?
>> I have been the first to mention NP hardness of constraint
>> satisfaction problem
>> but I am not sure if this is what you are asking about.
>>
>> Thanks,
>> Michal
>>
>> Patricia Shanahan wrote:
>>> Are you literally claiming NP Completeness, or just using that as an
>>> analogy for really, really difficult?
>>>
>>
>>
>
>
Re: OSGi NP Complete Was: OSGi - deserialization remote invocation
strategy
Posted by Peter <ji...@zeus.net.au>.
OSGi Dependency resolution is.
http://underlap.blogspot.com.au/2010/02/osgi-resolution-is-np-complete-so-what.html
Which means if we want to support an OSGi environment properly, we may
need some time to resolve the dependencies for a smart proxy, before
deserializing the proxy, rather than downloading the proxy dynamically
during unmarshalling, it's better to delay unmarshalling until the
dependencies are resolved, so the client isn't impacted by delays.
Cheers,
Peter.
On 13/02/2017 4:50 PM, Micha\u0142 K\u0142eczek wrote:
> Sorry, NP Completness of what?
> I have been the first to mention NP hardness of constraint
> satisfaction problem
> but I am not sure if this is what you are asking about.
>
> Thanks,
> Michal
>
> Patricia Shanahan wrote:
>> Are you literally claiming NP Completeness, or just using that as an
>> analogy for really, really difficult?
>>
>
>
Re: OSGi NP Complete Was: OSGi - deserialization remote invocation
strategy
Posted by Michał Kłeczek <mi...@kleczek.org>.
Sorry, NP Completness of what?
I have been the first to mention NP hardness of constraint satisfaction
problem
but I am not sure if this is what you are asking about.
Thanks,
Michal
Patricia Shanahan wrote:
> Are you literally claiming NP Completeness, or just using that as an
> analogy for really, really difficult?
>
Re: OSGi NP Complete Was: OSGi - deserialization remote invocation
strategy
Posted by Patricia Shanahan <pa...@acm.org>.
Are you literally claiming NP Completeness, or just using that as an
analogy for really, really difficult?
On 2/11/2017 1:23 PM, Micha\u0142 K\u0142eczek wrote:
> I am sorry but I think that to solve various issues we need to make sure
> fundamentals are right:
>
> 1. There is NO such a thing as "reflective non-smart" proxy - EVERY
> proxy is "smart" (even if it is "reflective") - there is an
> InvocationHandler down there, isn't there?
> 2. Solving this on service discovery level is trying to do it on the
> WRONG level of abstraction. Services DEPEND on class loading - not the
> other way around.
> 3. What you propose is a partial "solution". Not being able to register
> "smart" event listeners means no custom endpoints for example
> (UDPEndpoint anyone?)
> 4. Trying to squeeze partial solutions into the framework is IMHO a BIG
> no no.
> This is simply creating more code, more maintenance burden and more
> headache for users trying to workaround "edge, unsupported cases".
>
> Please - lets try to come up with the RIGHT solution that is going to
> REALLY fix class loading issues.
>
> Thanks,
> Michal
>
> Peter wrote:
>> In a word, ServiceDiscoveryManager
>>
>> ServiceDiscoveryManager is the solution.
>>
>> ServiceDiscoveryManager performs discovery and looks up services from
>> registrars based on filters. ServiceDiscoveryManager then performs
>> local filtering. This allows time for proxy bundles to be installed,
>> resolve, started and confirmed type compatible, prior to them being
>> made available (via OSGi service registry if you so desire) for client
>> use.
>>
>> The new interfaces that are part of JGDMS that I'd like to see their
>> way into River, found here:
>>
>> https://github.com/pfirmstone/JGDMS/blob/Maven_build/modularize/JGDMS/jgdms-lib-dl/src/main/java/net/jini/lookup/SafeServiceRegistrar.java
>>
>>
>> https://github.com/pfirmstone/JGDMS/blob/Maven_build/modularize/JGDMS/jgdms-lib-dl/src/main/java/net/jini/lookup/ServiceAttributesAccessor.java
>>
>> https://github.com/pfirmstone/JGDMS/blob/Maven_build/modularize/JGDMS/jgdms-lib-dl/src/main/java/net/jini/lookup/ServiceCodebaseAccessor.java
>>
>> https://github.com/pfirmstone/JGDMS/blob/Maven_build/modularize/JGDMS/jgdms-lib-dl/src/main/java/net/jini/lookup/ServiceIDAccessor.java
>>
>> https://github.com/pfirmstone/JGDMS/blob/Maven_build/modularize/JGDMS/jgdms-lib-dl/src/main/java/net/jini/lookup/ServiceProxyAccessor.java
>>
>>
>> ServiceCodebaseAccessor is also used as part of secure discovery, but
>> the codebase string and certs are transferred as primitives over the
>> network.
>>
>> In this case codebase annotations don't need to be included in the
>> stream, the JERI endpoints don't need them at all.
>>
>> How so?
>>
>> We can use ServiceItemFilter and ProxyPreparer to install, resolve and
>> start out proxy codebase, before downloading the proxy. The
>> interfaces listed above allow an array bootstrap proxy's
>> (java.lang.reflect.Proxy) to be obtained from SafeServiceRegistrar.
>>
>> Firstly the bootstrap proxy's JERI endpoint will be loaded in the
>> ServiceDiscoveryManager's ClassLoader, so after we've retrieved the
>> codebase annotation and signers, created a bundle for the proxy,
>> resolved it's dependencies (via OSGi resolution and repository
>> services), we need to remarshall the bootstrap proxy into a
>> MarshalledInstance, then unmarshall it using the ClassLoader of the
>> recently started proxy bundle. Then when we cast the bootstrap proxy
>> to ServiceProxyAccessor and retrieve the smart proxy, it will be
>> loaded into the same ClassLoader that the bootstrap proxy uses, our
>> newly provisioned and loaded bundle (the correct ClassLoader) without
>> need to serialize any annotations, then the smart proxy can have
>> constraints applied etc and be registered as an OSGi service with the
>> OSGi registrar, where client code can interact with the remote proxy.
>>
>> Now if public Serializable classes that are imported by the proxy's
>> bundle (service api) or private classes in the proxy's bundle can be
>> deserialized and the JERI endpoint has a reference to the ClassLoader
>> of the proxy.
>>
>> This should be good enough so we don't require the "bundle stack"
>> proposed earlier, which also saves the need to explain it and
>> simplifies the solution (the intent of the bundle stack was to allow
>> deserialization of private classes within other bundles whose packages
>> have been imported by the proxy).
>>
>> The client won't be able to pass a smart proxy to the service (like a
>> Listener), but it can still pass a non smart proxy and it will still
>> function. So clients can still export their own remote service
>> (albiet without a codebase, excluding smart proxy's), but it'll be
>> good enough for a listener etc.
>>
>> Cheers,
>>
>> Peter.
>>
>>
>> On 8/02/2017 1:09 PM, Niclas Hedhman wrote:
>>> Maybe there are some misunderstanding somewhere... see below;
>>>
>>> On Wed, Feb 8, 2017 at 3:35 AM, Peter<ji...@zeus.net.au> wrote:
>>>> I'm currently only considering OSGi server -> OSGi client. Mick's
>>> investigating all four options.
>>>
>>> Ok, makes it a lot easier for me to follow.
>>>
>>>> Not expecting the client calling bundle to resolve everything, hence
>>>> the
>>>> stack, so we have the full visibility of the bundle of the class
>>>> that was
>>>> last resolved, so we can resolve its fields from it's bundle. Eg it
>>>> might import packages the client does not.
>>> What is "client calling bundle"? I assume that it is the bundle that
>>> contains the service lookup...
>>>
>>> First of all, is the OSGi integration intending to be at "OSGi service
>>> level" ? If so, the Jini support bundle(s) in OSGi, will monitor Jini
>>> service registrations and register those in the local OSGi registry
>>> and all
>>> clients would end up being rather ignorant about the remote lookup.
>>> It is
>>> also possible to "be involved" in the service lookup (and registration)
>>> through the Service Hook API (I might got the name of it wrong), so that
>>> only when services are requested or service listeners are registered,
>>> does
>>> the Jini support kick in. And this can therefore work both ways...
>>>
>>> So, that said; I agree that the "client calling bundle" does not resolve
>>> anything. And I would go further and say; IF the client calling bundle
>>> looks up Jini services directly, all bets are off. This introduces some
>>> workable constraints of what could happen.
>>>
>>>
>>>> The "exact version" thing (only applies to the proxy bundle as we
>>>> expect the framework to load its deps) can be relaxed to compatible
>>>> versions to increase class sharing if you think it helps. The proxy
>>> bundle
>>>> doesn't export anything at the client, only the server, it just
>>>> seems to
>>>> make sense to keep the latest proxy communicating in case that last
>>>> bug fix release addresses a security issue. All proxy classes are
>>>> implementation only classes.
>>> Yes, correct. The 'temporary' proxy bundle should not export any
>>> packages.
>>>
>>>> Because the proxy bundle manifest declares version import ranges,
>>>> I'm expecting the framework to favour already loaded bundles to
>>>> satisfy package import deps.
>>> Uhhhh... The OSGi framework doesn't "auto load" bundles. It is an
>>> explicit
>>> step. There are many "bundle loaders" around, such as Apache Karaf
>>> "Features" and "deploy/" directory. Most frameworks can also be
>>> instructed
>>> to load bundles at boot. So there is no "favor already loaded
>>> bundles". If
>>> there is no bundle satisfying the Import-Package, the bundle being
>>> resolved
>>> will not go to RESOLVED state. If there are many bundles (quite typical)
>>> satisfying an Import-Package, with all its additional contraints
>>> (versions,
>>> attributes, uses, ...), then you enter the NP-complete problem that
>>> Michal
>>> mentioned, finding a combination of wiring that satisfy as many
>>> bundles as
>>> possible. This is a problem mainly due to "uses" (since we seldom use
>>> attributes), where graphs of types must end up matching in the class
>>> space.
>>>
>>> See (especially) section 3.7 in OSGi 6.0 Core Specification. If
>>> anyone can
>>> get their head around those details in the first pass, it is you
>>> Peter. Not
>>> easy reading...
>>>
>>>
>>>> If the client is matching service api with the correct import package
>>>> version ranges (requirements defined by entry's), the proxy bundle
>>>> should find the service api and other imported packages are already
>>>> loaded. Eg the client may use the requirements to use the resource
>>>> service or whatever the new bundle repository standard service is
>>>> called now to preload the requirements. The client may also perform
>>>> upgrades before downloading a service.
>>> I think this is a misunderstanding as well. By doing what I wrote in the
>>> beginning (listen on Jini Service Registrations and register "something"
>>> (Remote proxy or a local proxy) in the OSGi registry) then the client
>>> bundle doesn't need to know anything. Also, the Jini support bundle gets
>>> plenty of information both from the OSGi Registry as well as the Jini
>>> Registry. So, when something disappears from Reggie, remove it from the
>>> local Registry and vice versa.
>>>
>>> Now, the deserialization of the Reggie proxy should detect version
>>> changes
>>> and update a cache. And I think that Paremus idea of "Bundle Garbage
>>> Collection" is sound, but something for later discussion. Point
>>> being; No
>>> need to figure out what can and can not be unloaded. ALSO, since OSGi
>>> mandates intermittent service availability, most OSGi applications are
>>> reasonably capable to handle that the Jini service will "disappear"
>>> and is
>>> required to release any held references to the object(s) so that
>>> regular GC
>>> can toss out the classes and classloader when bundle is unloaded.
>>>
>>>> In the majority of cases I don't think there's going to be much state
>>>> in the smart proxy that can't be loaded via the smart proxy bundle
>>>> and it's package imports, except for the odd handback, which the
>>>> client bundle should have the opportunity to resolve before
>>>> resorting to using an annotation.
>>> Sorry, I don't understand this statement.
>>>
>>>> I'm not quite ready to agree it's too complex and it's unsolveable, I
>>>> think we should at least explore it and understand it before we junk
>>>> the idea of supporting OSGi.
>>> If we are talking osgi<->osgi, I think there is reasonable chance to
>>> succeed.
>>>
>>>> Rather than utilise the Java2 class loading I was planning to cast
>>>> ClassLoaders to BundleReference where appropriate and utilise
>>>> the Bundle.
>>> My gut says that this is not needed, if you go with my initial
>>> proposal. A
>>> Jini support bundle ends up having access to the BundleContext, from
>>> where
>>> everything else can be reached.
>>>
>>>> I did notice you're interpretation of what I've written is
>>>> different than
>>>> mine, so I think I need to put some effort into communicating more
>>>> effectively. I think you're interpretation of codebase annotation
>>>> "version is fixed" ignores that the annotation is only consulted after
>>>> determining that the current class is not available in our Bundles
>>>> currently participating in deserialization. It doesn't apply to
>>>> resolved
>>>> imported packages as annotations aren't used for them at all.
>>> OR, I have been burned in class resolution in OSGi enough times to
>>> have a
>>> feeling that it is more difficult than it seems. Any simple example I
>>> can
>>> think of would work...
>>>
>>>> For example, the first class we attempt to resolve during unmarshalling
>>>> belongs to a smart proxy, the client Bundle can't find the class.
>>>> Ask the
>>>> framework to load the proxy bundle from the codebase annotation, it
>>>> does so and resolves all necessary package imports declared in its
>>>> manifest. We now continue deserializing the smart proxy class fields
>>>> with the visibility of the smart proxy's bundle. The smart proxy may
>>>> contain fields referencing objects resolved from its imports, we ensure
>>>> those classes deserialize their fields with the visibility of their own
>>> bundles.
>>>
>>> So, the smart proxy's "bundle" is a bundle on the server side as
>>> well? The
>>> smart proxy may also contain objects that are of a class that is not
>>> visible... And it may be N levels deep from the "field" in the smart
>>> proxy.
>>>
>>>> Every time we can't resolve a class we first check if it's a handback
>>>> or parameter from a preceeding object in the graph, thus we walk
>>>> our graphs bundle stack.
>>> This is probably the bit I don't understand at all. On one hand you
>>> want to
>>> depend on OSGi framework to do the resolution, but OTOH you have
>>> something
>>> called a "bundle stack"? What is that?
>>>
>>>> If we still haven't resolved a class only then do we load a bundle from
>>>> it's codebase annotation url and check it can be cast to the field
>>>> before assigning it. If it can't be cast to that field, we throw an
>>> exception.
>>>
>>> "assigning it"? Doesn't the code in reality looks something like;
>>>
>>> private Map map;
>>>
>>> private void readObject( ObjectInputStream in ) {
>>> map = (Map) in.readObject();
>>> }
>>>
>>> where the read object may be of a class not visible to neither the smart
>>> proxy's classloader nor any helper? Maybe you meant that it will
>>> implicitly
>>> thrown just by the above code.
>>>
>>>
>>>> In the case of a non smart proxy, there is no codebase, deserialization
>>>> will be loaded by and rely completely on the visibility of the client
>>> bundle.
>>>> I think OSGi will be a lot less dependant on annotations than say a std
>>> env.
>>>
>>> Possibly... I think that largely depend on the usecase.
>>>
>>>> Still I guess wiring may be an an option, so as Michael suggests,
>>> annotate objects with their wiring graphs.
>>>
>>> Ok, here is what I see being the issue at hand; you think that it is
>>> possible to delay the bundle resolution of packages until
>>> deserialization
>>> itself. I think that is not possible. There is a need to bring all
>>> "non-available classes" into the client (and I think as a bundle is the
>>> correct solution). So, pick up the bundle reference of the smart
>>> proxy, and
>>> the bundle wiring graph of it. When the smart proxy arrives on the
>>> client,
>>> before doing anything else, load the bundles on the client. After
>>> that, it
>>> should be a "local JVM" problem, both to deserialize the smart proxy as
>>> well as every object communicated over the network.
>>>
>>>> What would we be considering if we hadn't been pre exposed to codebase
>>> annotations?
>>>> Standard deserialization uses one classpath, each bundle has its own
>>> unique classpath.
>>>
>>> It is not only about classpath. It is about class space and
>>> visibility as
>>> well. And serialization needs to bypass visibility, just like it
>>> bypassed
>>> other constraints before (such as bypassing constructors and
>>> initializing
>>> final fields), and that is a separate issue than the bundle loading
>>> mechanism.
>>>
>>>
>>> Cheers
>>> --
>>> Niclas Hedhman, Software Developer
>>> http://polygene.apache.org - New Energy for Java
>>>
>>
>
Re: OSGi NP Complete Was: OSGi - deserialization remote invocation
strategy
Posted by Michał Kłeczek <mi...@kleczek.org>.
I am sorry but I think that to solve various issues we need to make sure
fundamentals are right:
1. There is NO such a thing as "reflective non-smart" proxy - EVERY
proxy is "smart" (even if it is "reflective") - there is an
InvocationHandler down there, isn't there?
2. Solving this on service discovery level is trying to do it on the
WRONG level of abstraction. Services DEPEND on class loading - not the
other way around.
3. What you propose is a partial "solution". Not being able to register
"smart" event listeners means no custom endpoints for example
(UDPEndpoint anyone?)
4. Trying to squeeze partial solutions into the framework is IMHO a BIG
no no.
This is simply creating more code, more maintenance burden and more
headache for users trying to workaround "edge, unsupported cases".
Please - lets try to come up with the RIGHT solution that is going to
REALLY fix class loading issues.
Thanks,
Michal
Peter wrote:
> In a word, ServiceDiscoveryManager
>
> ServiceDiscoveryManager is the solution.
>
> ServiceDiscoveryManager performs discovery and looks up services from
> registrars based on filters. ServiceDiscoveryManager then performs
> local filtering. This allows time for proxy bundles to be installed,
> resolve, started and confirmed type compatible, prior to them being
> made available (via OSGi service registry if you so desire) for client
> use.
>
> The new interfaces that are part of JGDMS that I'd like to see their
> way into River, found here:
>
> https://github.com/pfirmstone/JGDMS/blob/Maven_build/modularize/JGDMS/jgdms-lib-dl/src/main/java/net/jini/lookup/SafeServiceRegistrar.java
>
>
> https://github.com/pfirmstone/JGDMS/blob/Maven_build/modularize/JGDMS/jgdms-lib-dl/src/main/java/net/jini/lookup/ServiceAttributesAccessor.java
>
> https://github.com/pfirmstone/JGDMS/blob/Maven_build/modularize/JGDMS/jgdms-lib-dl/src/main/java/net/jini/lookup/ServiceCodebaseAccessor.java
>
> https://github.com/pfirmstone/JGDMS/blob/Maven_build/modularize/JGDMS/jgdms-lib-dl/src/main/java/net/jini/lookup/ServiceIDAccessor.java
>
> https://github.com/pfirmstone/JGDMS/blob/Maven_build/modularize/JGDMS/jgdms-lib-dl/src/main/java/net/jini/lookup/ServiceProxyAccessor.java
>
>
> ServiceCodebaseAccessor is also used as part of secure discovery, but
> the codebase string and certs are transferred as primitives over the
> network.
>
> In this case codebase annotations don't need to be included in the
> stream, the JERI endpoints don't need them at all.
>
> How so?
>
> We can use ServiceItemFilter and ProxyPreparer to install, resolve and
> start out proxy codebase, before downloading the proxy. The
> interfaces listed above allow an array bootstrap proxy's
> (java.lang.reflect.Proxy) to be obtained from SafeServiceRegistrar.
>
> Firstly the bootstrap proxy's JERI endpoint will be loaded in the
> ServiceDiscoveryManager's ClassLoader, so after we've retrieved the
> codebase annotation and signers, created a bundle for the proxy,
> resolved it's dependencies (via OSGi resolution and repository
> services), we need to remarshall the bootstrap proxy into a
> MarshalledInstance, then unmarshall it using the ClassLoader of the
> recently started proxy bundle. Then when we cast the bootstrap proxy
> to ServiceProxyAccessor and retrieve the smart proxy, it will be
> loaded into the same ClassLoader that the bootstrap proxy uses, our
> newly provisioned and loaded bundle (the correct ClassLoader) without
> need to serialize any annotations, then the smart proxy can have
> constraints applied etc and be registered as an OSGi service with the
> OSGi registrar, where client code can interact with the remote proxy.
>
> Now if public Serializable classes that are imported by the proxy's
> bundle (service api) or private classes in the proxy's bundle can be
> deserialized and the JERI endpoint has a reference to the ClassLoader
> of the proxy.
>
> This should be good enough so we don't require the "bundle stack"
> proposed earlier, which also saves the need to explain it and
> simplifies the solution (the intent of the bundle stack was to allow
> deserialization of private classes within other bundles whose packages
> have been imported by the proxy).
>
> The client won't be able to pass a smart proxy to the service (like a
> Listener), but it can still pass a non smart proxy and it will still
> function. So clients can still export their own remote service
> (albiet without a codebase, excluding smart proxy's), but it'll be
> good enough for a listener etc.
>
> Cheers,
>
> Peter.
>
>
> On 8/02/2017 1:09 PM, Niclas Hedhman wrote:
>> Maybe there are some misunderstanding somewhere... see below;
>>
>> On Wed, Feb 8, 2017 at 3:35 AM, Peter<ji...@zeus.net.au> wrote:
>>> I'm currently only considering OSGi server -> OSGi client. Mick's
>> investigating all four options.
>>
>> Ok, makes it a lot easier for me to follow.
>>
>>> Not expecting the client calling bundle to resolve everything, hence
>>> the
>>> stack, so we have the full visibility of the bundle of the class
>>> that was
>>> last resolved, so we can resolve its fields from it's bundle. Eg it
>>> might import packages the client does not.
>> What is "client calling bundle"? I assume that it is the bundle that
>> contains the service lookup...
>>
>> First of all, is the OSGi integration intending to be at "OSGi service
>> level" ? If so, the Jini support bundle(s) in OSGi, will monitor Jini
>> service registrations and register those in the local OSGi registry
>> and all
>> clients would end up being rather ignorant about the remote lookup.
>> It is
>> also possible to "be involved" in the service lookup (and registration)
>> through the Service Hook API (I might got the name of it wrong), so that
>> only when services are requested or service listeners are registered,
>> does
>> the Jini support kick in. And this can therefore work both ways...
>>
>> So, that said; I agree that the "client calling bundle" does not resolve
>> anything. And I would go further and say; IF the client calling bundle
>> looks up Jini services directly, all bets are off. This introduces some
>> workable constraints of what could happen.
>>
>>
>>> The "exact version" thing (only applies to the proxy bundle as we
>>> expect the framework to load its deps) can be relaxed to compatible
>>> versions to increase class sharing if you think it helps. The proxy
>> bundle
>>> doesn't export anything at the client, only the server, it just
>>> seems to
>>> make sense to keep the latest proxy communicating in case that last
>>> bug fix release addresses a security issue. All proxy classes are
>>> implementation only classes.
>> Yes, correct. The 'temporary' proxy bundle should not export any
>> packages.
>>
>>> Because the proxy bundle manifest declares version import ranges,
>>> I'm expecting the framework to favour already loaded bundles to
>>> satisfy package import deps.
>> Uhhhh... The OSGi framework doesn't "auto load" bundles. It is an
>> explicit
>> step. There are many "bundle loaders" around, such as Apache Karaf
>> "Features" and "deploy/" directory. Most frameworks can also be
>> instructed
>> to load bundles at boot. So there is no "favor already loaded
>> bundles". If
>> there is no bundle satisfying the Import-Package, the bundle being
>> resolved
>> will not go to RESOLVED state. If there are many bundles (quite typical)
>> satisfying an Import-Package, with all its additional contraints
>> (versions,
>> attributes, uses, ...), then you enter the NP-complete problem that
>> Michal
>> mentioned, finding a combination of wiring that satisfy as many
>> bundles as
>> possible. This is a problem mainly due to "uses" (since we seldom use
>> attributes), where graphs of types must end up matching in the class
>> space.
>>
>> See (especially) section 3.7 in OSGi 6.0 Core Specification. If
>> anyone can
>> get their head around those details in the first pass, it is you
>> Peter. Not
>> easy reading...
>>
>>
>>> If the client is matching service api with the correct import package
>>> version ranges (requirements defined by entry's), the proxy bundle
>>> should find the service api and other imported packages are already
>>> loaded. Eg the client may use the requirements to use the resource
>>> service or whatever the new bundle repository standard service is
>>> called now to preload the requirements. The client may also perform
>>> upgrades before downloading a service.
>> I think this is a misunderstanding as well. By doing what I wrote in the
>> beginning (listen on Jini Service Registrations and register "something"
>> (Remote proxy or a local proxy) in the OSGi registry) then the client
>> bundle doesn't need to know anything. Also, the Jini support bundle gets
>> plenty of information both from the OSGi Registry as well as the Jini
>> Registry. So, when something disappears from Reggie, remove it from the
>> local Registry and vice versa.
>>
>> Now, the deserialization of the Reggie proxy should detect version
>> changes
>> and update a cache. And I think that Paremus idea of "Bundle Garbage
>> Collection" is sound, but something for later discussion. Point
>> being; No
>> need to figure out what can and can not be unloaded. ALSO, since OSGi
>> mandates intermittent service availability, most OSGi applications are
>> reasonably capable to handle that the Jini service will "disappear"
>> and is
>> required to release any held references to the object(s) so that
>> regular GC
>> can toss out the classes and classloader when bundle is unloaded.
>>
>>> In the majority of cases I don't think there's going to be much state
>>> in the smart proxy that can't be loaded via the smart proxy bundle
>>> and it's package imports, except for the odd handback, which the
>>> client bundle should have the opportunity to resolve before
>>> resorting to using an annotation.
>> Sorry, I don't understand this statement.
>>
>>> I'm not quite ready to agree it's too complex and it's unsolveable, I
>>> think we should at least explore it and understand it before we junk
>>> the idea of supporting OSGi.
>> If we are talking osgi<->osgi, I think there is reasonable chance to
>> succeed.
>>
>>> Rather than utilise the Java2 class loading I was planning to cast
>>> ClassLoaders to BundleReference where appropriate and utilise
>>> the Bundle.
>> My gut says that this is not needed, if you go with my initial
>> proposal. A
>> Jini support bundle ends up having access to the BundleContext, from
>> where
>> everything else can be reached.
>>
>>> I did notice you're interpretation of what I've written is
>>> different than
>>> mine, so I think I need to put some effort into communicating more
>>> effectively. I think you're interpretation of codebase annotation
>>> "version is fixed" ignores that the annotation is only consulted after
>>> determining that the current class is not available in our Bundles
>>> currently participating in deserialization. It doesn't apply to
>>> resolved
>>> imported packages as annotations aren't used for them at all.
>> OR, I have been burned in class resolution in OSGi enough times to
>> have a
>> feeling that it is more difficult than it seems. Any simple example I
>> can
>> think of would work...
>>
>>> For example, the first class we attempt to resolve during unmarshalling
>>> belongs to a smart proxy, the client Bundle can't find the class.
>>> Ask the
>>> framework to load the proxy bundle from the codebase annotation, it
>>> does so and resolves all necessary package imports declared in its
>>> manifest. We now continue deserializing the smart proxy class fields
>>> with the visibility of the smart proxy's bundle. The smart proxy may
>>> contain fields referencing objects resolved from its imports, we ensure
>>> those classes deserialize their fields with the visibility of their own
>> bundles.
>>
>> So, the smart proxy's "bundle" is a bundle on the server side as
>> well? The
>> smart proxy may also contain objects that are of a class that is not
>> visible... And it may be N levels deep from the "field" in the smart
>> proxy.
>>
>>> Every time we can't resolve a class we first check if it's a handback
>>> or parameter from a preceeding object in the graph, thus we walk
>>> our graphs bundle stack.
>> This is probably the bit I don't understand at all. On one hand you
>> want to
>> depend on OSGi framework to do the resolution, but OTOH you have
>> something
>> called a "bundle stack"? What is that?
>>
>>> If we still haven't resolved a class only then do we load a bundle from
>>> it's codebase annotation url and check it can be cast to the field
>>> before assigning it. If it can't be cast to that field, we throw an
>> exception.
>>
>> "assigning it"? Doesn't the code in reality looks something like;
>>
>> private Map map;
>>
>> private void readObject( ObjectInputStream in ) {
>> map = (Map) in.readObject();
>> }
>>
>> where the read object may be of a class not visible to neither the smart
>> proxy's classloader nor any helper? Maybe you meant that it will
>> implicitly
>> thrown just by the above code.
>>
>>
>>> In the case of a non smart proxy, there is no codebase, deserialization
>>> will be loaded by and rely completely on the visibility of the client
>> bundle.
>>> I think OSGi will be a lot less dependant on annotations than say a std
>> env.
>>
>> Possibly... I think that largely depend on the usecase.
>>
>>> Still I guess wiring may be an an option, so as Michael suggests,
>> annotate objects with their wiring graphs.
>>
>> Ok, here is what I see being the issue at hand; you think that it is
>> possible to delay the bundle resolution of packages until
>> deserialization
>> itself. I think that is not possible. There is a need to bring all
>> "non-available classes" into the client (and I think as a bundle is the
>> correct solution). So, pick up the bundle reference of the smart
>> proxy, and
>> the bundle wiring graph of it. When the smart proxy arrives on the
>> client,
>> before doing anything else, load the bundles on the client. After
>> that, it
>> should be a "local JVM" problem, both to deserialize the smart proxy as
>> well as every object communicated over the network.
>>
>>> What would we be considering if we hadn't been pre exposed to codebase
>> annotations?
>>> Standard deserialization uses one classpath, each bundle has its own
>> unique classpath.
>>
>> It is not only about classpath. It is about class space and
>> visibility as
>> well. And serialization needs to bypass visibility, just like it
>> bypassed
>> other constraints before (such as bypassing constructors and
>> initializing
>> final fields), and that is a separate issue than the bundle loading
>> mechanism.
>>
>>
>> Cheers
>> --
>> Niclas Hedhman, Software Developer
>> http://polygene.apache.org - New Energy for Java
>>
>