You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@servicemix.apache.org by Jeremias Maerki <de...@jeremias-maerki.ch> on 2009/05/26 10:32:47 UTC
[SMX4] Specs Locator observations
I'm currently looking into making certain libraries OSGi-capapable which
are to date using the META-INF/services mechanism for detecting plug-ins.
In this context I noticed the locator classes used by the spec bundles
SMX4 publishes. I've thought about doing something similar but was
surprised to find a copy of the two classes in each spec bundle. Each
activator is registering a synchronous bundle listener which scans for
all files in META-INF/services. I suspected this might be bad for
(startup) performance so I profiled the whole thing:
Profiling showed that the register() method of the Activator is called
966 times for (currently) 138 bundles. 966 / 138 = 7. Currently, seven
spec JARs in my application are all using that approach. The profiler
determined that 5% of the startup time is used just for this. This will
increase with every spec JAR or normal bundle that is added to the
application.
I wonder if it were not possible to keep the activator as a separate
locator bundle that all spec bundles could use together. That way, the
SPI files only need to be scanned for once per bundle. I haven't
investigated the technical implications to any depth, yet. I just wanted
to gather some thoughts.
I could actually then use that approach, too, for detecting plug-in
implementations in the various bundles. For example, Apache FOP & Batik
are using the SPI mechanism extensively and not just for looking up a
default factory implementation. Various plug-ins extend the two with
support for new output formats, image formats, etc. etc.
I wonder if it makes sense to make the locator into a more general
utility (maybe in Felix or Commons or Lab). I've started writing a very
small set of interfaces and classes that would help abstracting the use
of JAR SPI plug-ins inside and outside an OSGi environment. After seeing
the spec locator Activator I'm somewhat hesitant to create yet another
synchronous bundle listener that has to scan META-INF/services.
Another thought I had: Since scanning META-INF/services seems rather
expensive, a bundle header entry [1] that signals the presense of files
in META-INF/services could enable to early return from the listener. Of
course, that requires all affected bundles to be changed. Not ideal, I
know. Still, I think it is worth thinking about.
[1] Example: JAR-Service-Provider: true
I'm grateful for any thoughts on the matter.
Thanks,
Jeremias Maerki
Re: [SMX4] Specs Locator observations
Posted by Jeremias Maerki <de...@jeremias-maerki.ch>.
On 27.05.2009 14:34:33 Guillaume Nodet wrote:
> Why not reusing the classes as they are ?
> I mean we don't have to publish a service at all. If we use a
> Require-Bundle or even an Import-Package on the package containing the
> locator classes, it should still work.
> So if we remove the locator classes from each spec bundle and deploy
> those as a bundle on its own, they should all be able to access the
> information without any problem, and still only a single instance of
> the extender would run.
If that works, that's certainly already an improvement.
> As for the headers, it could work, but this require to repackage all
> the implementations, and given this is not a standard .... I would
> avoid doing that for the moment.
It was just an idea to further optimize, to reduce the number of bundles
to be scanned.
> On Wed, May 27, 2009 at 13:56, Jeremias Maerki <de...@jeremias-maerki.ch> wrote:
> > Thanks for your response, Gert.
> >
> > 47% of the time during scanning is spent in Bundle.findEntries()
> > (org.apache.felix.framework.BundleImpl.findEntries(String, String, boolean)).
> > Another 25% is spent in Enumeration.nextElement
> > (org.apache.felix.framework.FindEntriesEnumeration.nextElement())
> >
> > As for providing the discovered service implementations, I think it
> > can make sense to just publish a special OSGi service (per class) that
> > allows access to the Class and allows to create a new instance, for
> > example:
> >
> > public interface JarServiceFactory {
> >
> > Class getImplementationClass();
> > Object createInstance() throws InstantiationException, IllegalAccessException;
> >
> > }
> >
> > I've started writing something like that but I've put it aside in favor
> > of another task I'm currently working on. I intend to get back to it
> > soon.
> >
> > I think it would be possible and to go a step further and use a special
> > header to indicate certain implementation classes that should be
> > instantiated directly and published as OSGi service.
> >
> > One problem I see is that this probably only works if the name of the
> > file in META-INF/services matches the fully qualified name of a common
> > base class or interface (there's no requirement for that).
> >
> > I'm not sure how exactly the blueprint services will help here. I have
> > heard about the topic but haven't taken a closer look, yet.
> >
> > As you hinted at, I wouldn't want the solution to be tied to Karaf. I'd
> > prefer one based on standard OSGi infrastructure.
> >
> > On 27.05.2009 12:55:23 Gert Vanthienen wrote:
> >> Jeremias,
> >>
> >> Thanks for looking into the code and for doing this investigation
> >> work. If the Locator is indeed causing so much overhead, we do want
> >> to optimize that somehow. Do you an idea what part of the Locator
> >> code is causing the overhead?
> >>
> >> As for creating a single Locator service in the system, the code now
> >> works with a static container for the SPI class information so we
> >> can't really use that as it is. We could probably use the OSGi
> >> Service registry instead of a static container class to hold the
> >> information. In the spec bundle, we would then either have a
> >> ServiceTracker to dynamically keep track of those registry entries or
> >> we would use a new OsgiLocator implementation that does the lookup
> >> dynamically.
> >>
> >> I guess one of the main problems is the fact that the classes in the
> >> META-INF/services would have to be instantiated to put them into the
> >> registry or we would have to create some kind of lazy-activate proxy
> >> that only creates the class when it's needed (avoiding the need for
> >> the bundle to be started until it's needed). I think the Blueprint
> >> services spec that's currently being built could help us there, but
> >> Guillaume knows a lot more about this so he can probably clarify what
> >> this would do for us.
> >>
> >> In the long run, it would probably make sense if this information
> >> would be in the the OSGi manifest instead of in another file inside
> >> the bundle. Not sure if we should consider that, because that would
> >> be a change that would have to go into all these things. We could
> >> solve it for Apache Felix Karaf
> >> (the-open-source-project-formerly-known-as ServiceMix Kernel) by
> >> making the Deployer do the conversion on the fly, but that would only
> >> solve it for Karaf and I guess we want a more generic solution.
> >>
> >> Regards,
> >>
> >> Gert Vanthienen
> >> ------------------------
> >> Open Source SOA: http://fusesource.com
> >> Blog: http://gertvanthienen.blogspot.com/
> >>
> >>
> >>
> >> 2009/5/26 Jeremias Maerki <de...@jeremias-maerki.ch>:
> >> > I'm currently looking into making certain libraries OSGi-capapable which
> >> > are to date using the META-INF/services mechanism for detecting plug-ins.
> >> > In this context I noticed the locator classes used by the spec bundles
> >> > SMX4 publishes. I've thought about doing something similar but was
> >> > surprised to find a copy of the two classes in each spec bundle. Each
> >> > activator is registering a synchronous bundle listener which scans for
> >> > all files in META-INF/services. I suspected this might be bad for
> >> > (startup) performance so I profiled the whole thing:
> >> >
> >> > Profiling showed that the register() method of the Activator is called
> >> > 966 times for (currently) 138 bundles. 966 / 138 = 7. Currently, seven
> >> > spec JARs in my application are all using that approach. The profiler
> >> > determined that 5% of the startup time is used just for this. This will
> >> > increase with every spec JAR or normal bundle that is added to the
> >> > application.
> >> >
> >> > I wonder if it were not possible to keep the activator as a separate
> >> > locator bundle that all spec bundles could use together. That way, the
> >> > SPI files only need to be scanned for once per bundle. I haven't
> >> > investigated the technical implications to any depth, yet. I just wanted
> >> > to gather some thoughts.
> >> >
> >> > I could actually then use that approach, too, for detecting plug-in
> >> > implementations in the various bundles. For example, Apache FOP & Batik
> >> > are using the SPI mechanism extensively and not just for looking up a
> >> > default factory implementation. Various plug-ins extend the two with
> >> > support for new output formats, image formats, etc. etc.
> >> >
> >> > I wonder if it makes sense to make the locator into a more general
> >> > utility (maybe in Felix or Commons or Lab). I've started writing a very
> >> > small set of interfaces and classes that would help abstracting the use
> >> > of JAR SPI plug-ins inside and outside an OSGi environment. After seeing
> >> > the spec locator Activator I'm somewhat hesitant to create yet another
> >> > synchronous bundle listener that has to scan META-INF/services.
> >> >
> >> > Another thought I had: Since scanning META-INF/services seems rather
> >> > expensive, a bundle header entry [1] that signals the presense of files
> >> > in META-INF/services could enable to early return from the listener. Of
> >> > course, that requires all affected bundles to be changed. Not ideal, I
> >> > know. Still, I think it is worth thinking about.
> >> >
> >> > [1] Example: JAR-Service-Provider: true
> >> >
> >> > I'm grateful for any thoughts on the matter.
> >> >
> >> > Thanks,
> >> > Jeremias Maerki
> >> >
> >> >
> >
> >
> >
> >
> > Jeremias Maerki
> >
> >
>
>
>
> --
> Cheers,
> Guillaume Nodet
> ------------------------
> Blog: http://gnodet.blogspot.com/
> ------------------------
> Open Source SOA
> http://fusesource.com
Jeremias Maerki
Re: [SMX4] Specs Locator observations
Posted by Guillaume Nodet <gn...@gmail.com>.
Why not reusing the classes as they are ?
I mean we don't have to publish a service at all. If we use a
Require-Bundle or even an Import-Package on the package containing the
locator classes, it should still work.
So if we remove the locator classes from each spec bundle and deploy
those as a bundle on its own, they should all be able to access the
information without any problem, and still only a single instance of
the extender would run.
As for the headers, it could work, but this require to repackage all
the implementations, and given this is not a standard .... I would
avoid doing that for the moment.
On Wed, May 27, 2009 at 13:56, Jeremias Maerki <de...@jeremias-maerki.ch> wrote:
> Thanks for your response, Gert.
>
> 47% of the time during scanning is spent in Bundle.findEntries()
> (org.apache.felix.framework.BundleImpl.findEntries(String, String, boolean)).
> Another 25% is spent in Enumeration.nextElement
> (org.apache.felix.framework.FindEntriesEnumeration.nextElement())
>
> As for providing the discovered service implementations, I think it
> can make sense to just publish a special OSGi service (per class) that
> allows access to the Class and allows to create a new instance, for
> example:
>
> public interface JarServiceFactory {
>
> Class getImplementationClass();
> Object createInstance() throws InstantiationException, IllegalAccessException;
>
> }
>
> I've started writing something like that but I've put it aside in favor
> of another task I'm currently working on. I intend to get back to it
> soon.
>
> I think it would be possible and to go a step further and use a special
> header to indicate certain implementation classes that should be
> instantiated directly and published as OSGi service.
>
> One problem I see is that this probably only works if the name of the
> file in META-INF/services matches the fully qualified name of a common
> base class or interface (there's no requirement for that).
>
> I'm not sure how exactly the blueprint services will help here. I have
> heard about the topic but haven't taken a closer look, yet.
>
> As you hinted at, I wouldn't want the solution to be tied to Karaf. I'd
> prefer one based on standard OSGi infrastructure.
>
> On 27.05.2009 12:55:23 Gert Vanthienen wrote:
>> Jeremias,
>>
>> Thanks for looking into the code and for doing this investigation
>> work. If the Locator is indeed causing so much overhead, we do want
>> to optimize that somehow. Do you an idea what part of the Locator
>> code is causing the overhead?
>>
>> As for creating a single Locator service in the system, the code now
>> works with a static container for the SPI class information so we
>> can't really use that as it is. We could probably use the OSGi
>> Service registry instead of a static container class to hold the
>> information. In the spec bundle, we would then either have a
>> ServiceTracker to dynamically keep track of those registry entries or
>> we would use a new OsgiLocator implementation that does the lookup
>> dynamically.
>>
>> I guess one of the main problems is the fact that the classes in the
>> META-INF/services would have to be instantiated to put them into the
>> registry or we would have to create some kind of lazy-activate proxy
>> that only creates the class when it's needed (avoiding the need for
>> the bundle to be started until it's needed). I think the Blueprint
>> services spec that's currently being built could help us there, but
>> Guillaume knows a lot more about this so he can probably clarify what
>> this would do for us.
>>
>> In the long run, it would probably make sense if this information
>> would be in the the OSGi manifest instead of in another file inside
>> the bundle. Not sure if we should consider that, because that would
>> be a change that would have to go into all these things. We could
>> solve it for Apache Felix Karaf
>> (the-open-source-project-formerly-known-as ServiceMix Kernel) by
>> making the Deployer do the conversion on the fly, but that would only
>> solve it for Karaf and I guess we want a more generic solution.
>>
>> Regards,
>>
>> Gert Vanthienen
>> ------------------------
>> Open Source SOA: http://fusesource.com
>> Blog: http://gertvanthienen.blogspot.com/
>>
>>
>>
>> 2009/5/26 Jeremias Maerki <de...@jeremias-maerki.ch>:
>> > I'm currently looking into making certain libraries OSGi-capapable which
>> > are to date using the META-INF/services mechanism for detecting plug-ins.
>> > In this context I noticed the locator classes used by the spec bundles
>> > SMX4 publishes. I've thought about doing something similar but was
>> > surprised to find a copy of the two classes in each spec bundle. Each
>> > activator is registering a synchronous bundle listener which scans for
>> > all files in META-INF/services. I suspected this might be bad for
>> > (startup) performance so I profiled the whole thing:
>> >
>> > Profiling showed that the register() method of the Activator is called
>> > 966 times for (currently) 138 bundles. 966 / 138 = 7. Currently, seven
>> > spec JARs in my application are all using that approach. The profiler
>> > determined that 5% of the startup time is used just for this. This will
>> > increase with every spec JAR or normal bundle that is added to the
>> > application.
>> >
>> > I wonder if it were not possible to keep the activator as a separate
>> > locator bundle that all spec bundles could use together. That way, the
>> > SPI files only need to be scanned for once per bundle. I haven't
>> > investigated the technical implications to any depth, yet. I just wanted
>> > to gather some thoughts.
>> >
>> > I could actually then use that approach, too, for detecting plug-in
>> > implementations in the various bundles. For example, Apache FOP & Batik
>> > are using the SPI mechanism extensively and not just for looking up a
>> > default factory implementation. Various plug-ins extend the two with
>> > support for new output formats, image formats, etc. etc.
>> >
>> > I wonder if it makes sense to make the locator into a more general
>> > utility (maybe in Felix or Commons or Lab). I've started writing a very
>> > small set of interfaces and classes that would help abstracting the use
>> > of JAR SPI plug-ins inside and outside an OSGi environment. After seeing
>> > the spec locator Activator I'm somewhat hesitant to create yet another
>> > synchronous bundle listener that has to scan META-INF/services.
>> >
>> > Another thought I had: Since scanning META-INF/services seems rather
>> > expensive, a bundle header entry [1] that signals the presense of files
>> > in META-INF/services could enable to early return from the listener. Of
>> > course, that requires all affected bundles to be changed. Not ideal, I
>> > know. Still, I think it is worth thinking about.
>> >
>> > [1] Example: JAR-Service-Provider: true
>> >
>> > I'm grateful for any thoughts on the matter.
>> >
>> > Thanks,
>> > Jeremias Maerki
>> >
>> >
>
>
>
>
> Jeremias Maerki
>
>
--
Cheers,
Guillaume Nodet
------------------------
Blog: http://gnodet.blogspot.com/
------------------------
Open Source SOA
http://fusesource.com
Re: [SMX4] Specs Locator observations
Posted by Jeremias Maerki <de...@jeremias-maerki.ch>.
Thanks for your response, Gert.
47% of the time during scanning is spent in Bundle.findEntries()
(org.apache.felix.framework.BundleImpl.findEntries(String, String, boolean)).
Another 25% is spent in Enumeration.nextElement
(org.apache.felix.framework.FindEntriesEnumeration.nextElement())
As for providing the discovered service implementations, I think it
can make sense to just publish a special OSGi service (per class) that
allows access to the Class and allows to create a new instance, for
example:
public interface JarServiceFactory {
Class getImplementationClass();
Object createInstance() throws InstantiationException, IllegalAccessException;
}
I've started writing something like that but I've put it aside in favor
of another task I'm currently working on. I intend to get back to it
soon.
I think it would be possible and to go a step further and use a special
header to indicate certain implementation classes that should be
instantiated directly and published as OSGi service.
One problem I see is that this probably only works if the name of the
file in META-INF/services matches the fully qualified name of a common
base class or interface (there's no requirement for that).
I'm not sure how exactly the blueprint services will help here. I have
heard about the topic but haven't taken a closer look, yet.
As you hinted at, I wouldn't want the solution to be tied to Karaf. I'd
prefer one based on standard OSGi infrastructure.
On 27.05.2009 12:55:23 Gert Vanthienen wrote:
> Jeremias,
>
> Thanks for looking into the code and for doing this investigation
> work. If the Locator is indeed causing so much overhead, we do want
> to optimize that somehow. Do you an idea what part of the Locator
> code is causing the overhead?
>
> As for creating a single Locator service in the system, the code now
> works with a static container for the SPI class information so we
> can't really use that as it is. We could probably use the OSGi
> Service registry instead of a static container class to hold the
> information. In the spec bundle, we would then either have a
> ServiceTracker to dynamically keep track of those registry entries or
> we would use a new OsgiLocator implementation that does the lookup
> dynamically.
>
> I guess one of the main problems is the fact that the classes in the
> META-INF/services would have to be instantiated to put them into the
> registry or we would have to create some kind of lazy-activate proxy
> that only creates the class when it's needed (avoiding the need for
> the bundle to be started until it's needed). I think the Blueprint
> services spec that's currently being built could help us there, but
> Guillaume knows a lot more about this so he can probably clarify what
> this would do for us.
>
> In the long run, it would probably make sense if this information
> would be in the the OSGi manifest instead of in another file inside
> the bundle. Not sure if we should consider that, because that would
> be a change that would have to go into all these things. We could
> solve it for Apache Felix Karaf
> (the-open-source-project-formerly-known-as ServiceMix Kernel) by
> making the Deployer do the conversion on the fly, but that would only
> solve it for Karaf and I guess we want a more generic solution.
>
> Regards,
>
> Gert Vanthienen
> ------------------------
> Open Source SOA: http://fusesource.com
> Blog: http://gertvanthienen.blogspot.com/
>
>
>
> 2009/5/26 Jeremias Maerki <de...@jeremias-maerki.ch>:
> > I'm currently looking into making certain libraries OSGi-capapable which
> > are to date using the META-INF/services mechanism for detecting plug-ins.
> > In this context I noticed the locator classes used by the spec bundles
> > SMX4 publishes. I've thought about doing something similar but was
> > surprised to find a copy of the two classes in each spec bundle. Each
> > activator is registering a synchronous bundle listener which scans for
> > all files in META-INF/services. I suspected this might be bad for
> > (startup) performance so I profiled the whole thing:
> >
> > Profiling showed that the register() method of the Activator is called
> > 966 times for (currently) 138 bundles. 966 / 138 = 7. Currently, seven
> > spec JARs in my application are all using that approach. The profiler
> > determined that 5% of the startup time is used just for this. This will
> > increase with every spec JAR or normal bundle that is added to the
> > application.
> >
> > I wonder if it were not possible to keep the activator as a separate
> > locator bundle that all spec bundles could use together. That way, the
> > SPI files only need to be scanned for once per bundle. I haven't
> > investigated the technical implications to any depth, yet. I just wanted
> > to gather some thoughts.
> >
> > I could actually then use that approach, too, for detecting plug-in
> > implementations in the various bundles. For example, Apache FOP & Batik
> > are using the SPI mechanism extensively and not just for looking up a
> > default factory implementation. Various plug-ins extend the two with
> > support for new output formats, image formats, etc. etc.
> >
> > I wonder if it makes sense to make the locator into a more general
> > utility (maybe in Felix or Commons or Lab). I've started writing a very
> > small set of interfaces and classes that would help abstracting the use
> > of JAR SPI plug-ins inside and outside an OSGi environment. After seeing
> > the spec locator Activator I'm somewhat hesitant to create yet another
> > synchronous bundle listener that has to scan META-INF/services.
> >
> > Another thought I had: Since scanning META-INF/services seems rather
> > expensive, a bundle header entry [1] that signals the presense of files
> > in META-INF/services could enable to early return from the listener. Of
> > course, that requires all affected bundles to be changed. Not ideal, I
> > know. Still, I think it is worth thinking about.
> >
> > [1] Example: JAR-Service-Provider: true
> >
> > I'm grateful for any thoughts on the matter.
> >
> > Thanks,
> > Jeremias Maerki
> >
> >
Jeremias Maerki
Re: [SMX4] Specs Locator observations
Posted by Gert Vanthienen <ge...@gmail.com>.
Jeremias,
Thanks for looking into the code and for doing this investigation
work. If the Locator is indeed causing so much overhead, we do want
to optimize that somehow. Do you an idea what part of the Locator
code is causing the overhead?
As for creating a single Locator service in the system, the code now
works with a static container for the SPI class information so we
can't really use that as it is. We could probably use the OSGi
Service registry instead of a static container class to hold the
information. In the spec bundle, we would then either have a
ServiceTracker to dynamically keep track of those registry entries or
we would use a new OsgiLocator implementation that does the lookup
dynamically.
I guess one of the main problems is the fact that the classes in the
META-INF/services would have to be instantiated to put them into the
registry or we would have to create some kind of lazy-activate proxy
that only creates the class when it's needed (avoiding the need for
the bundle to be started until it's needed). I think the Blueprint
services spec that's currently being built could help us there, but
Guillaume knows a lot more about this so he can probably clarify what
this would do for us.
In the long run, it would probably make sense if this information
would be in the the OSGi manifest instead of in another file inside
the bundle. Not sure if we should consider that, because that would
be a change that would have to go into all these things. We could
solve it for Apache Felix Karaf
(the-open-source-project-formerly-known-as ServiceMix Kernel) by
making the Deployer do the conversion on the fly, but that would only
solve it for Karaf and I guess we want a more generic solution.
Regards,
Gert Vanthienen
------------------------
Open Source SOA: http://fusesource.com
Blog: http://gertvanthienen.blogspot.com/
2009/5/26 Jeremias Maerki <de...@jeremias-maerki.ch>:
> I'm currently looking into making certain libraries OSGi-capapable which
> are to date using the META-INF/services mechanism for detecting plug-ins.
> In this context I noticed the locator classes used by the spec bundles
> SMX4 publishes. I've thought about doing something similar but was
> surprised to find a copy of the two classes in each spec bundle. Each
> activator is registering a synchronous bundle listener which scans for
> all files in META-INF/services. I suspected this might be bad for
> (startup) performance so I profiled the whole thing:
>
> Profiling showed that the register() method of the Activator is called
> 966 times for (currently) 138 bundles. 966 / 138 = 7. Currently, seven
> spec JARs in my application are all using that approach. The profiler
> determined that 5% of the startup time is used just for this. This will
> increase with every spec JAR or normal bundle that is added to the
> application.
>
> I wonder if it were not possible to keep the activator as a separate
> locator bundle that all spec bundles could use together. That way, the
> SPI files only need to be scanned for once per bundle. I haven't
> investigated the technical implications to any depth, yet. I just wanted
> to gather some thoughts.
>
> I could actually then use that approach, too, for detecting plug-in
> implementations in the various bundles. For example, Apache FOP & Batik
> are using the SPI mechanism extensively and not just for looking up a
> default factory implementation. Various plug-ins extend the two with
> support for new output formats, image formats, etc. etc.
>
> I wonder if it makes sense to make the locator into a more general
> utility (maybe in Felix or Commons or Lab). I've started writing a very
> small set of interfaces and classes that would help abstracting the use
> of JAR SPI plug-ins inside and outside an OSGi environment. After seeing
> the spec locator Activator I'm somewhat hesitant to create yet another
> synchronous bundle listener that has to scan META-INF/services.
>
> Another thought I had: Since scanning META-INF/services seems rather
> expensive, a bundle header entry [1] that signals the presense of files
> in META-INF/services could enable to early return from the listener. Of
> course, that requires all affected bundles to be changed. Not ideal, I
> know. Still, I think it is worth thinking about.
>
> [1] Example: JAR-Service-Provider: true
>
> I'm grateful for any thoughts on the matter.
>
> Thanks,
> Jeremias Maerki
>
>