Re: Newbie question related to device communication

[Scott Lewis <sc...@gmail.com>]

Hi Folks,

Update:   ECF has recently completed initial documentation for 9 
distribution providers [1].  We've also recently had our 3.11.0 release 
[2], which includes a new API for registering custom distribution 
providers via the whiteboard pattern.

Scott

[1] 
http://eclipseecf.blogspot.com/2015/10/ecf-remote-services-distribution.html
[2] 
http://eclipseecf.blogspot.com/2015/09/ecf-3110-custom-remote-services.html


On 6/3/2015 10:03 AM, Scott Lewis wrote:
> On 6/3/2015 2:25 AM, Bruce Jackson wrote:
>> I’ve now built 2 different systems based on OSGi at scale (over 1M 
>> end devices) where the endpoints needed to communicate together. In 
>> both cases, it has required some ‘server’ infrastructure to help with 
>> message passing between endpoints, but that could be a side effect of 
>> the infrastructure we had to use. However, these server components 
>> were again based on OSGi.
>>
>> The http question (I.e. using the HTTP service) is actually a bit of 
>> a red herring: there is nothing stopping each endpoint implementing 
>> an http service and also being a client of other endpoints to pass 
>> messages. My suggestion would be to use this approach,  as it has so 
>> many side benefits, including:
>>
>> * good availability of tools such as Jackson for message serialisation
>> * open protocols that will allow for different endpoint 
>> implementations in future
>> * (by and large) avoiding problems that custom socket protocols will 
>> run into with firewall tunnelling etc
>>
>> …the list here is pretty long.
>
> I would like to discuss the Remote Service specification before 
> returning to discussing protocols and RS implementations.
>
> The specification explicitly allows the runtime selection of a 
> distribution provider to be used to export a remote service, via a 
> standardized service property called the 'config type' (property 
> name:  service.exported.configs).    As an example, the 
> 'ecf.generic.server' distribution provider is the config type used in 
> this tutorial [1].
>
> I feel that one of the most useful aspects of the config type and the 
> underlying distribution system is that it abstracts away the 
> implementation specifics, i.e. the transport protocol (e.g. http, jms, 
> tcp, mqtt, private, etc), as well as the serialization 
> mechanism/format (json, binary, xml, etc).   It does this for both the 
> remote service creator/implementer and for the remote service consumer.
>
> IMHO, there are there are many advantages of doing this.   A big one 
> is modularity and reuse.  Why should it be necessary for every remote 
> service author to implement a brand new protocol and serialization, in 
> order to remote their micro service?   Another one is flexibility:  
> any given reasonably sophisticated app has changing requirements over 
> time, and sometimes these changing requirements require flexibility in 
> the underlying libraries. Another big advantage is getting access to 
> all the simplicity, dynamics, versioning, and other capabilities of 
> OSGi services provided by the OSGi service registry and libs built on 
> top of it (e.g. DS, etc). There are of course many others.
>
> Of course, there are disadvantages in reusing distribution systems or 
> any set of libraries:   Perhaps the protocol or serialization 
> mechanism that's widely implemented and popular today doesn't meet 
> your application's performance needs.   Perhaps there are more 
> appropriate or new protocols, or improved implementations. Perhaps you 
> have unique security requirements.     Perhaps you have timing or 
> reliability requirements that are difficult to address. There are lots 
> of application-specific reasons for favoring or disfavoring 
> non-functional requirements like performance, ease of deployment and 
> management, security (e.g. firewalls), network reliability/failure 
> handling, etc.
>
> But in my view, the Remote Services specification adopts a nice 
> balance here, by allowing potentially many distribution providers, 
> down to the level of the individual service instance set at *service 
> instance registration time*.   Even if one needs to create a new, 
> entire distribution system, there are still many advantages of using 
> OSGI Remote Services...e.g. the service registry's graceful handling 
> of dynamics inherent in a network-based service, versioning, 
> discovery, and the ability for service creators and perhaps more 
> importantly service consumers to ignore some of the subtle and 
> difficult aspects of distribution (e.g. partial failure, timing, 
> etc).   I do think that standardizing these advantages and having many 
> implementations will over time make it easier to build, use, 
> understand, and manage distributed applications.
>
> Returning to RS implementations:   The approach ECF has taken (since 
> well before the Distributed OSGi rfc and RS/RSA standardization) has 
> been protocol neutral.   As such we have gone to the trouble of 
> creating extensible APIs for various types of network communication:   
> e.g. filetransfer, instant messaging, group messaging, voice over ip, 
> network service discovery, and remote services distribution (of 
> course).   For each of these types, we have multiple protocol 
> implementations, some created by us (in open source), some created by 
> others (in open source or not).   This is true, of course, for our 
> OSGi Remote Services implementation, and it's distribution and 
> discovery subsystems...as I've said we have a number of providers that 
> are based upon tcp and we have providers that are based upon http and 
> REST.   But more importantly (I think), we are trying to support the 
> creation of hopefully many other/new providers, created by 
> others...possibly extending those that already exist...that meet their 
> needs for app-specific non-functional requirements, new/popular 
> application areas (like IoT) for distributed applications.
>
> The way that we do this with a multi-provider/multi-protocol approach 
> is to:
>
> 0) Fully implement the latest specs an test for compliance
> 1) Define stable, extensible, open APIs for distribution and discovery 
> (e.g. 'remote services api' -> distribution, 'discovery api' -> 
> discovery), and maintain/enhance these APIs.
> 2) Re-use existing work to create stable, extensible, open provider 
> implementations for these types (e.g. ecf generic, http/REST, r-osgi, 
> jms, mqtt, etc)
> 3) Do what we can in open cooperation with the community to document 
> and provide examples for both 1 and 2, as well as document how to 
> create new providers [2], and analyze the runtime characteristics of 
> existing providers.
> 4) Engage the community (commercial and open source) to drive 0-3 
> forword in the directions that are most needed, and get support for 
> these efforts via new committers, contributors, collaborators, and 
> supporters [3].
> 5) Provide the support for the community in doing all of the above
>
> Are we resource-limited?  Yes, unless you consider the community. Is 
> there more to do? (particularly around documenting individual 
> providers)?  Yes.   My comment is that in today's software world I 
> don't think these answers are particularly uncommon ;-).  [4]
>
> Scott
>
> [1] 
> https://wiki.eclipse.org/Tutorial:_Building_your_first_OSGi_Remote_Service
> See section:  Service Host: Register and Export the Service
> [2] 
> https://wiki.eclipse.org/Tutorial:_Creating_a_RESTful_Remote_Service_Provider
> [3] https://projects.eclipse.org/projects/rt.ecf/who
> [4] https://www.eclipse.org/ecf/NewAndNoteworthy_3.10.0.html
>
>
>
>>
>>
>>
>> On 02/06/2015 16:08, "Scott Lewis" <sl...@composent.com> wrote:
>>
>>> On 6/2/2015 2:16 AM, Simon Kitching wrote:
>>>> Hi Scott,
>>>>
>>>> Maybe I shouldn't hijack a thread on the Felix list to ask about ECF,
>>>> but we are already talking about remote-services implementations, and
>>>> you are here so I'll do it :-)
>>>>
>>>> Link [1] has instructions on using ECF's remote services on Karaf
>>>> which lead to this karaf features file:
>>>> http://download.eclipse.org/rt/ecf/3.8.0/site.p2/karaf-features.xml.
>>>> That file appears to indicate that 49 bundles are needed (the majority
>>>> from eclipse.org). Plus bundles to support whatever discovery provider
>>>> is desired. That doesn't seem reasonable; other remote-services
>>>> implementations have 1-3 bundles. Have I understood correctly?
>>> For the latest the correct URL is:
>>> http://download.eclipse.org/rt/ecf/latest/site.p2/karaf-features.xml
>>>
>>> ECF's implementation is quite modular, and so is made up of more
>>> bundles, generally much smaller and finer-grained.
>>>
>>>> <stuff deleted>
>>>>
>>>> Looking at the commit-log for the osgi code, I see that you are the
>>>> only committer since at least the start of 2014. Is this correct?
>>> I have done the majority of the commits from the start of 2014, but am
>>> not the only committer.   The source code history is not a very good
>>> indicator of overall contributions for a project in this point in it's
>>> life-cycle.   Also, much of the recent coding work on ECF (by myself 
>>> and
>>> others) has taken place on our github repos[4].
>>>
>>>> <stuff deleted>
>>>> Regarding building from source : I see no Maven, Ant, Groovy or bnd
>>>> buildfiles in the sourcecode - just eclipse project files. Could you
>>>> point me to instructions on how to build artifacts from the
>>>> source-code (ideally outside of the Eclipse IDE)?
>>> We currently use Eclipse + buckminster to build, and are part of the
>>> Eclipse Simultaneous Release (which also uses buckminster). There are
>>> contributors from Red Hat that are working adding to use Tycho to
>>> build.  FWIW, we also produce a maven repos as part of our build [A].
>>>
>>>> And is there any easy way of seeing what the dependencies of the
>>>> projects under osgi/bundles are (the kind of info visible in a maven
>>>> pom-file)?
>>>  From Eclipse, certainly.  When Red Hat's contribution is in place you
>>> will be able to do with maven-pom file as well.
>>>
>>>> Is there a list of the implemented "distribution providers" (network
>>>> protocols for the actual remote method invocation) available, with
>>>> details on the features and performance of each choice?
>>> No, regrettably we don't have that.  I've opened an enhancement request
>>> to that effect [B].  There was some performance analysis work
>>> contributed some time ago, but unfortunately I think it's likely pretty
>>> obsolete by now.
>>>
>>>> I couldn't find one. In particular, I needed a high-performance
>>>> protocol - one that (a) keeps network sockets open rather than opening
>>>> a new connection for each call, and (b) uses a high-performance
>>>> serialization mechanism, ie not xml. I couldn't find any page with
>>>> that kind of information in it. The closest I found was:
>>>> *
>>>> https://wiki.eclipse.org/Discovery_and_Distribution_Provider_Configuration_Options 
>>>>
>>>> *
>>>> https://wiki.eclipse.org/Comparison_of_Discovery_and_Distribution_Providers 
>>>>
>>>> but neither of those pages are very useful. In particular, "r-osgi"
>>>> and "ecf generic" are not described in any way.
>>> For the details about everything there's always 'use the source'. :-)
>>>
>>> I can tell you that r-osgi, ecf generic, JMS, MQTT distribution
>>> providers all
>>>
>>> 1) keep network sockets open as you describe;
>>> 2) use java object serialization as the default
>>>
>>> I can also tell you that all of those providers, and the distribution
>>> API that they implement (known to us as the 'remote services api'), are
>>> customizable/extensible for both 1 and 2...meaning that if you wish, 
>>> you
>>> can replace both the serialization as well as the socket-level 
>>> transport
>>> with something more suitable for your use case without having to
>>> implement a distribution provider from scratch.   As a an example, that
>>> is how websockets support was added in the 3.10 release cycle.
>>>
>>> I propose moving this to ecf-dev or direct correspondence if you want
>>> more details.
>>>
>>> Scott
>>>
>>> [A] http://build.ecf-project.org/maven/
>>> [B] https://bugs.eclipse.org/bugs/show_bug.cgi?id=469155
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: users-unsubscribe@felix.apache.org
>>> For additional commands, e-mail: users-help@felix.apache.org
>>>
>> *** DISCLAIMER *** This message, including attachments, is intended 
>> solely for the addressee indicated in this message and is strictly 
>> confidential or otherwise privileged. If you are not the intended 
>> recipient (or responsible for delivery of the message to such person) 
>> : - (1) please immediately (i) notify the sender by reply email and 
>> (ii) delete this message and attachments, - (2) any use, copy or 
>> dissemination of this transmission is strictly prohibited. If you or 
>> your employer does not consent to Internet email messages of this 
>> kind, please advise Myriad Group AG by reply e-mail immediately. 
>> Opinions, conclusions and other information expressed in this message 
>> are not given or endorsed by Myriad Group AG unless otherwise 
>> indicated by an authorized representative independent of this message.
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@felix.apache.org
>> For additional commands, e-mail: users-help@felix.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@felix.apache.org
> For additional commands, e-mail: users-help@felix.apache.org
>


---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@felix.apache.org
For additional commands, e-mail: users-help@felix.apache.org