You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@celix.apache.org by Björn Petri <bj...@sundevil.de> on 2013/12/04 18:07:30 UTC
Re: Remote service using shared mem
I updated the functionality of the discovery component to ensure that
all threads perform its update routine, when a service was added/stopped
or the discovery bundle is stopped.
I would be happy if you find the time to give it a short check.
Regards,
Bjoern
Am 2013-10-03 14:07, schrieb Björn Petri:
> Hi Alexander,
>
> introducing the lock in the discovery_stop function was done with
> purpose. The unlocking and locking should enable all
> discovery-polling-threads to update their local list of services and
> in the case of the stopping discovery bundle it should allow the
> thread to leave it's loop (the loop control variable should be set to
> false before performing the unlock and lock). So, it should actually
> not wait on the join - I'll take a look at the code and check what is
> going on there.
>
> Regards,
> Björn
>
>
>
>
> Am 2013-10-03 11:59, schrieb Alexander Broekhuis:
>> Hi Bjoern,
>>
>> I have some problems with the discovery now. If I try to stop the
>> discovery
>> bundle, it tries to join the discovery thread, but somehow the lock
>> isn't
>> released and the thread doesn't exit.
>>
>> Looking in the code, the discovery_stop now has an unlock directly
>> followed
>> by a lock. This seems to be the problem. Did a small error sneak in?
>> Or is
>> there some reason for the unlock/lock?
>>
>> I'll do some more testing later on this week, so I'll get back to
>> you :)
>>
>>
>> 2013/10/2 Björn Petri <bj...@sundevil.de>
>>
>>>
>>> Alexander,
>>>
>>> I updated the rs shared memory implementation to
>>>
>>> (1) use System-V shared memory routines instead of ones from APR,
>>> (2) make use of your install_bundles macro,
>>> (3) perform some necessary cleanup when the exported service is
>>> stopped
>>> (This is also missing in the "standard"-rsa implementation). See
>>> remoteServiceAdmin_**removeExportedService function for details,
>>> (4) get rid of the linking of the example_proxy against the
>>> rsa_shm,
>>> (5) and fixed some minor bugs.
>>>
>>> looking forward to your feedback,
>>> Bjoern
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> Am 2013-09-27 16:11, schrieb Alexander Broekhuis:
>>>
>>> 2013/9/27 Björn Petri <bj...@sundevil.de>
>>>>
>>>> On 09/18/2013 06:29 PM, Alexander Broekhuis wrote:
>>>>>
>>>>> I checked it, and there are some small leftovers in the code.
>>>>> The proxy
>>>>>> still includes the rsa and also links with the library.
>>>>>>
>>>>>>
>>>>> How do you want to get rid of this dependency? The proxy needs to
>>>>> include
>>>>> definition of the remote_proxy_service as well as the rsa does.
>>>>> Do I miss
>>>>> something?
>>>>>
>>>>>
>>>> Since the proxy now uses the function pointer and only the service
>>>> struct
>>>> of the RSA it only needs the header files (as far as I can tell).
>>>> Also,
>>>> with the concept of bundles, and Celix not supporting code-sharing
>>>> at this
>>>> point, bundles (actually the library in it) should never link to a
>>>> library
>>>> of any other bundle.
>>>> Libraries are loaded locally, so the symbols aren't shared with
>>>> others. So
>>>> this means that if you link with another bundle, at runtime there
>>>> will be
>>>> unresolved references. Hence the need for services (structs with
>>>> function
>>>> pointers) etc.
>>>>
>>>
>>>
Re: Remote service using shared mem
Posted by Gerrit Binnenmars <ge...@gmail.com>.
Hello Bjorn and Alexander,
Both of you, thanks for your work and cooperation to bring the remote
shm patch to Celix.
Greetings Gerrit
> Am 2013-12-05 20:25, schrieb Alexander Broekhuis:
>> Hi,
>>
>> 2013/12/5 Björn Petri <bj...@sundevil.de>
>>
>>>
>>> Hi Alexander,
>>>
>>> thanks for your feedback. Concerning your issues:
>>>
>>> * I am fine with replacing semtimedop again through semop, as the
>>> function
>>> calls are almost identical, I can do that right away.
>>>
>>
>> Would be great :). This doesn't introduce any problems?
>
> Not as long as all discovery bundles are working fine. As soon as one
> discovery bundle is not shutdown properly, the semtimedop still allows
> all other discovery bundles to shutdown.
>
> But as I do not expect the discovery bundle to 'not shutdown
> properly', replacing semtimedop with semop will not bring in any
> changed behavior.
>
> I've put a new patch in place, which has also been updated concerning
> the latest svn changes and the export of the RSA-Headers (CELIX-99).
>
> Regards,
> Bjoern
Re: Remote service using shared mem
Posted by Björn Petri <bj...@sundevil.de>.
Am 2013-12-05 20:25, schrieb Alexander Broekhuis:
> Hi,
>
> 2013/12/5 Björn Petri <bj...@sundevil.de>
>
>>
>> Hi Alexander,
>>
>> thanks for your feedback. Concerning your issues:
>>
>> * I am fine with replacing semtimedop again through semop, as the
>> function
>> calls are almost identical, I can do that right away.
>>
>
> Would be great :). This doesn't introduce any problems?
Not as long as all discovery bundles are working fine. As soon as one
discovery bundle is not shutdown properly, the semtimedop still allows
all other discovery bundles to shutdown.
But as I do not expect the discovery bundle to 'not shutdown properly',
replacing semtimedop with semop will not bring in any changed behavior.
I've put a new patch in place, which has also been updated concerning
the latest svn changes and the export of the RSA-Headers (CELIX-99).
Regards,
Bjoern
Re: Remote service using shared mem
Posted by Alexander Broekhuis <a....@gmail.com>.
>
> I've pushed a fix for the proxy callback methods with the correct json
> handling. If all is ok now you should be able to use the calculator example
> as is. If there are any changes needed, please say so, so we can keep the
> HTTP and SHM RSA aligned.
>
I've done an additional commit which might be useful for the SHM RSA. I've
renamed the current HTTP implementation to a more descriptive directory
(remote_service_admin_http) and reused the existing directory to place all
common headers.
These headers include the public (service) and private headers needed to
implement an RSA.
Specific details can easily be placed in an implementation specific header
(see the http implementation as an example).
It might be useful to use these common headers for the SHM RSA as well,
there is no need anymore to have duplicates.
For both issues you can take a look at [1] and [2].
[1]: https://issues.apache.org/jira/browse/CELIX-98
[2]: https://issues.apache.org/jira/browse/CELIX-99
--
Met vriendelijke groet,
Alexander Broekhuis
Re: Remote service using shared mem
Posted by Alexander Broekhuis <a....@gmail.com>.
>
> * Same for the JSON format changes. As far as I remember, there is only a
>> line per function which specifies the JSON format.
>>
>
> I am already working on those locally, if you have a moment I'll push some
> changes so that the proxy/endpoint can be used by both the HTTP and the SHM
> RSA.
>
I've pushed a fix for the proxy callback methods with the correct json
handling. If all is ok now you should be able to use the calculator example
as is. If there are any changes needed, please say so, so we can keep the
HTTP and SHM RSA aligned.
--
Met vriendelijke groet,
Alexander Broekhuis
Re: Remote service using shared mem
Posted by Alexander Broekhuis <a....@gmail.com>.
Hi,
2013/12/5 Björn Petri <bj...@sundevil.de>
>
> Hi Alexander,
>
> thanks for your feedback. Concerning your issues:
>
> * I am fine with replacing semtimedop again through semop, as the function
> calls are almost identical, I can do that right away.
>
Would be great :). This doesn't introduce any problems?
>
> * Same for the JSON format changes. As far as I remember, there is only a
> line per function which specifies the JSON format.
>
I am already working on those locally, if you have a moment I'll push some
changes so that the proxy/endpoint can be used by both the HTTP and the SHM
RSA.
>
> I'll provide you with the changes asap. Let me know when something else
> should be done.
>
I'll do, thx for the quick repsonse!
--
Met vriendelijke groet,
Alexander Broekhuis
Re: Remote service using shared mem
Posted by Björn Petri <bj...@sundevil.de>.
Hi Alexander,
thanks for your feedback. Concerning your issues:
* I am fine with replacing semtimedop again through semop, as the
function calls are almost identical, I can do that right away.
* Same for the JSON format changes. As far as I remember, there is only
a line per function which specifies the JSON format.
I'll provide you with the changes asap. Let me know when something else
should be done.
Regards,
Björn
Am 2013-12-05 17:17, schrieb Alexander Broekhuis:
> Hi Bjoern,
>
> I'm now trying to apply the patch and have some new issues, I've
> added some
> remarks in between:
>
> * It is created against an older revision, some small changes that
> easily
> can be fixed.
> For a final patch I'd like this to be solved, but isn't a big
> issue.
> * It now uses "semtimedop", which is a GLibC function (and not posix)
> so
> now it doesn't work on OSX.
> While I think we should work towards a broader support with these
> kind of
> things, I can live with the first version being Linux (GLibC) only.
> But
> this does mean I cannot test it at this moment.. So someone should
> definitely still do this (Pepijn?)
> * Calculator Endpoint/Proxy uses the older format for the JSON data
> (which
> is incompatible with the Amdatu RSA).
> This is a small change in the encoding/decoding, but isn't a big
> issue.
> * The SHM RSA uses setHandler/setCallback instead of
> setEndpointProperties.
> The existing HTTP RSA doesn't use these yet.
> At this moment this breaks the Calculator example when used with
> the HTTP
> RSA. Probably not that difficult to port back to the HTTP RSA. I
> prefer the
> newer method, so definitely in favour of applying those changes to
> the HTTP
> RSA as well.
>
> Some other remarks/updates not yet mentioned on this list:
> Yesterday we already talked about how to donate this code. Since this
> code
> introduces new bundles it makes the most sense to handle this using a
> Code
> Grant and IP Clearance.
> We already discussed that Pepijn is willing to do the paper work, so
> hopefully it al doesn't take that much time.
>
> So to summarize (in order):
> 1) We need a working set of bundles (at least tested on Linux,
> preferably
> on OSX as well). Existing code should still work as well.
> 2) After that a code grant with Issue number and MD5 sum needs to be
> filled
> in and filed.
> 3) Finally the IP Clearance should be handled.
>
> I have some time to do some work, so I can help out a bit. For now
> I'll
> work on getting the setHandler/setCallback method in the HTTP RSA as
> well,
> if that is in place, ideally you shouldn't need to update those for
> the SHM
> RSA.
>
> Anything I forgot or overlooking?
>
>
>
> 2013/12/4 Björn Petri <bj...@sundevil.de>
>
>>
>>
>> I updated the functionality of the discovery component to ensure
>> that all
>> threads perform its update routine, when a service was added/stopped
>> or the
>> discovery bundle is stopped.
>>
>> I would be happy if you find the time to give it a short check.
>>
>> Regards,
>> Bjoern
>>
>>
>>
>>
>>
>> Am 2013-10-03 14:07, schrieb Björn Petri:
>>
>> Hi Alexander,
>>>
>>> introducing the lock in the discovery_stop function was done with
>>> purpose. The unlocking and locking should enable all
>>> discovery-polling-threads to update their local list of services
>>> and
>>> in the case of the stopping discovery bundle it should allow the
>>> thread to leave it's loop (the loop control variable should be set
>>> to
>>> false before performing the unlock and lock). So, it should
>>> actually
>>> not wait on the join - I'll take a look at the code and check what
>>> is
>>> going on there.
>>>
>>> Regards,
>>> Björn
>>>
>>>
>>>
>>>
>>> Am 2013-10-03 11:59, schrieb Alexander Broekhuis:
>>>
>>>> Hi Bjoern,
>>>>
>>>> I have some problems with the discovery now. If I try to stop the
>>>> discovery
>>>> bundle, it tries to join the discovery thread, but somehow the
>>>> lock isn't
>>>> released and the thread doesn't exit.
>>>>
>>>> Looking in the code, the discovery_stop now has an unlock directly
>>>> followed
>>>> by a lock. This seems to be the problem. Did a small error sneak
>>>> in? Or
>>>> is
>>>> there some reason for the unlock/lock?
>>>>
>>>> I'll do some more testing later on this week, so I'll get back to
>>>> you :)
>>>>
>>>>
>>>> 2013/10/2 Björn Petri <bj...@sundevil.de>
>>>>
>>>>
>>>>> Alexander,
>>>>>
>>>>> I updated the rs shared memory implementation to
>>>>>
>>>>> (1) use System-V shared memory routines instead of ones from APR,
>>>>> (2) make use of your install_bundles macro,
>>>>> (3) perform some necessary cleanup when the exported service is
>>>>> stopped
>>>>> (This is also missing in the "standard"-rsa implementation). See
>>>>> remoteServiceAdmin_**removeExportedService function for details,
>>>>> (4) get rid of the linking of the example_proxy against the
>>>>> rsa_shm,
>>>>> (5) and fixed some minor bugs.
>>>>>
>>>>> looking forward to your feedback,
>>>>> Bjoern
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> Am 2013-09-27 16:11, schrieb Alexander Broekhuis:
>>>>>
>>>>> 2013/9/27 Björn Petri <bj...@sundevil.de>
>>>>>
>>>>>>
>>>>>> On 09/18/2013 06:29 PM, Alexander Broekhuis wrote:
>>>>>>
>>>>>>>
>>>>>>> I checked it, and there are some small leftovers in the code.
>>>>>>> The
>>>>>>> proxy
>>>>>>>
>>>>>>>> still includes the rsa and also links with the library.
>>>>>>>>
>>>>>>>>
>>>>>>>> How do you want to get rid of this dependency? The proxy
>>>>>>>> needs to
>>>>>>> include
>>>>>>> definition of the remote_proxy_service as well as the rsa does.
>>>>>>> Do I
>>>>>>> miss
>>>>>>> something?
>>>>>>>
>>>>>>>
>>>>>>> Since the proxy now uses the function pointer and only the
>>>>>>> service
>>>>>> struct
>>>>>> of the RSA it only needs the header files (as far as I can
>>>>>> tell). Also,
>>>>>> with the concept of bundles, and Celix not supporting
>>>>>> code-sharing at
>>>>>> this
>>>>>> point, bundles (actually the library in it) should never link to
>>>>>> a
>>>>>> library
>>>>>> of any other bundle.
>>>>>> Libraries are loaded locally, so the symbols aren't shared with
>>>>>> others. So
>>>>>> this means that if you link with another bundle, at runtime
>>>>>> there will
>>>>>> be
>>>>>> unresolved references. Hence the need for services (structs with
>>>>>> function
>>>>>> pointers) etc.
>>>>>>
>>>>>>
>>>>>
>>>>>
>>
Re: Remote service using shared mem
Posted by Alexander Broekhuis <a....@gmail.com>.
Hi Bjoern,
I'm now trying to apply the patch and have some new issues, I've added some
remarks in between:
* It is created against an older revision, some small changes that easily
can be fixed.
For a final patch I'd like this to be solved, but isn't a big issue.
* It now uses "semtimedop", which is a GLibC function (and not posix) so
now it doesn't work on OSX.
While I think we should work towards a broader support with these kind of
things, I can live with the first version being Linux (GLibC) only. But
this does mean I cannot test it at this moment.. So someone should
definitely still do this (Pepijn?)
* Calculator Endpoint/Proxy uses the older format for the JSON data (which
is incompatible with the Amdatu RSA).
This is a small change in the encoding/decoding, but isn't a big issue.
* The SHM RSA uses setHandler/setCallback instead of setEndpointProperties.
The existing HTTP RSA doesn't use these yet.
At this moment this breaks the Calculator example when used with the HTTP
RSA. Probably not that difficult to port back to the HTTP RSA. I prefer the
newer method, so definitely in favour of applying those changes to the HTTP
RSA as well.
Some other remarks/updates not yet mentioned on this list:
Yesterday we already talked about how to donate this code. Since this code
introduces new bundles it makes the most sense to handle this using a Code
Grant and IP Clearance.
We already discussed that Pepijn is willing to do the paper work, so
hopefully it al doesn't take that much time.
So to summarize (in order):
1) We need a working set of bundles (at least tested on Linux, preferably
on OSX as well). Existing code should still work as well.
2) After that a code grant with Issue number and MD5 sum needs to be filled
in and filed.
3) Finally the IP Clearance should be handled.
I have some time to do some work, so I can help out a bit. For now I'll
work on getting the setHandler/setCallback method in the HTTP RSA as well,
if that is in place, ideally you shouldn't need to update those for the SHM
RSA.
Anything I forgot or overlooking?
2013/12/4 Björn Petri <bj...@sundevil.de>
>
>
> I updated the functionality of the discovery component to ensure that all
> threads perform its update routine, when a service was added/stopped or the
> discovery bundle is stopped.
>
> I would be happy if you find the time to give it a short check.
>
> Regards,
> Bjoern
>
>
>
>
>
> Am 2013-10-03 14:07, schrieb Björn Petri:
>
> Hi Alexander,
>>
>> introducing the lock in the discovery_stop function was done with
>> purpose. The unlocking and locking should enable all
>> discovery-polling-threads to update their local list of services and
>> in the case of the stopping discovery bundle it should allow the
>> thread to leave it's loop (the loop control variable should be set to
>> false before performing the unlock and lock). So, it should actually
>> not wait on the join - I'll take a look at the code and check what is
>> going on there.
>>
>> Regards,
>> Björn
>>
>>
>>
>>
>> Am 2013-10-03 11:59, schrieb Alexander Broekhuis:
>>
>>> Hi Bjoern,
>>>
>>> I have some problems with the discovery now. If I try to stop the
>>> discovery
>>> bundle, it tries to join the discovery thread, but somehow the lock isn't
>>> released and the thread doesn't exit.
>>>
>>> Looking in the code, the discovery_stop now has an unlock directly
>>> followed
>>> by a lock. This seems to be the problem. Did a small error sneak in? Or
>>> is
>>> there some reason for the unlock/lock?
>>>
>>> I'll do some more testing later on this week, so I'll get back to you :)
>>>
>>>
>>> 2013/10/2 Björn Petri <bj...@sundevil.de>
>>>
>>>
>>>> Alexander,
>>>>
>>>> I updated the rs shared memory implementation to
>>>>
>>>> (1) use System-V shared memory routines instead of ones from APR,
>>>> (2) make use of your install_bundles macro,
>>>> (3) perform some necessary cleanup when the exported service is stopped
>>>> (This is also missing in the "standard"-rsa implementation). See
>>>> remoteServiceAdmin_**removeExportedService function for details,
>>>> (4) get rid of the linking of the example_proxy against the rsa_shm,
>>>> (5) and fixed some minor bugs.
>>>>
>>>> looking forward to your feedback,
>>>> Bjoern
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> Am 2013-09-27 16:11, schrieb Alexander Broekhuis:
>>>>
>>>> 2013/9/27 Björn Petri <bj...@sundevil.de>
>>>>
>>>>>
>>>>> On 09/18/2013 06:29 PM, Alexander Broekhuis wrote:
>>>>>
>>>>>>
>>>>>> I checked it, and there are some small leftovers in the code. The
>>>>>> proxy
>>>>>>
>>>>>>> still includes the rsa and also links with the library.
>>>>>>>
>>>>>>>
>>>>>>> How do you want to get rid of this dependency? The proxy needs to
>>>>>> include
>>>>>> definition of the remote_proxy_service as well as the rsa does. Do I
>>>>>> miss
>>>>>> something?
>>>>>>
>>>>>>
>>>>>> Since the proxy now uses the function pointer and only the service
>>>>> struct
>>>>> of the RSA it only needs the header files (as far as I can tell). Also,
>>>>> with the concept of bundles, and Celix not supporting code-sharing at
>>>>> this
>>>>> point, bundles (actually the library in it) should never link to a
>>>>> library
>>>>> of any other bundle.
>>>>> Libraries are loaded locally, so the symbols aren't shared with
>>>>> others. So
>>>>> this means that if you link with another bundle, at runtime there will
>>>>> be
>>>>> unresolved references. Hence the need for services (structs with
>>>>> function
>>>>> pointers) etc.
>>>>>
>>>>>
>>>>
>>>>
>
--
Met vriendelijke groet,
Alexander Broekhuis