Support for latest Axis2/C and Python runtimes

[Jean-Sebastien Delfino <js...@apache.org>]

I'm on vacation, so if I find time in the next few days I'm going to do 
some coding in sca-cpp and try to get Web Services and Python components 
working again with the latest Axis2/C and Python runtimes.

I'd also like to try and compare sca-cpp's performance with the httpd 
multi threaded 'worker' and 'prefork' modules... not sure I'll get to it 
though, as I have a lot of other non computer related stuff planned for 
the vacation :)

-- 
Jean-Sebastien

Memory management and performance improvements, was: Support for latest Axis2/C and Python runtimes

[Jean-Sebastien Delfino <js...@apache.org>]

Jean-Sebastien Delfino wrote:
> I'm on vacation, so if I find time in the next few days I'm going to do 
> some coding in sca-cpp
...

> 
> I'd also like to try and compare sca-cpp's performance with the httpd 
> multi threaded 'worker' and 'prefork' modules... not sure I'll get to it 
> though, as I have a lot of other non computer related stuff planned for 
> the vacation :)
> 

So I found some time over the holiday break to experiment with different 
threading and memory management schemes, and was able to make a few 
improvements to the C++ SCA runtime.

The SCA runtime now works with both the pre-fork (pool of mono-thread 
processes) and worker (pool of multi-threaded processes) HTTPD MPMs.

That way you can get both super-fast dispatch in multi-threaded HTTPD 
nodes (dedicated to SCA component wiring/routing for example), and 
robustness against application failures in pre-forked HTTPD nodes 
running application code.

To speed up multi-threaded execution, there's no mutex locks anymore on 
the main processing path. To eliminate these locks I've done two things:

- Replaced the reference counting scheme I was using before for memory 
collection (which required locks around the ref counters) by a pool 
based scheme using HTTPD/APR pools (with fast sequential allocs from a 
pool private to the HTTP request processed by a given thread, no frees, 
until HTTPD frees the whole pool at once after processing the request.)

- Added simple memory pool based and lock free implementations of string 
and input/output streams, replacing the slower STL string and stream 
equivalents (which use locks around their reference counters as well.)

With these changes, performance is now getting really good in both 
pre-fork and multi-threaded worker HTTPD servers.

On my home Ubuntu server (Core Duo 2.66 Ghz) a plain HTTP static GET 
loopback takes 0.16 msec (tested with a loop hitting a multi-threaded 
HTTPD in 10 concurrent threads).

An ATOM POST to an SCA component, with SCA wiring/routing to the 
component, parsing of the ATOM XML entry, invocation of the component 
implementation (written in Python or Scheme, that doesn't seem to make a 
difference) takes 0.19 sec.

So that's 0.03 msec for the SCA runtime, on top of the HTTPD 0.16 msec 
baseline... getting pretty fast I think :)

Next, I'd like to measure invocation of SCA components written in C++, 
and see how the runtime can fly with streaming big payloads (as the new 
string and stream implementations should also help minimize memory copies.)

-- 
Jean-Sebastien

ASN.1 support, was: Support for latest Axis2/C and Python runtimes

[Jean-Sebastien Delfino <js...@apache.org>]

Giorgio Zoppi wrote:
...
>  It could become an useful runtime for
> embedded components (i.e. in telco projects),
> going beyond sca or soa platforms, if we provide a binary protocol for
> comunications between parts (i.e some sort of rpc using asn.1 for
> object serialization).

Sounds good...

I did a lot of ASN.1 in a previous life -- to the point of waking up in 
the middle of the night with ASN.1 encoded nightmares :) so I'm not sure 
I want to go back there myself right now on my Tuscany spare coding 
time... But it'd certainly be interesting, so if you want to give it a 
shot I can try to help.

I've looked around and asked some of my old telco-time pals what they'd 
use for ASN.1 now. Here are a few options you could start with:

- ASN1C [1], nice and simple.

- The builtin support for ASN.1 in Erlang [2]. Looks pretty solid, no 
surprise given Erlang's roots.

- The ASN.1 code in openssl [3]. I've not seen a compiler but I may have 
missed it. The runtime seems incomplete, missing REAL and Embedded PDV 
types, but they're not used a lot AFAIK.

There's probably more options. At this point I'd hesitate between [1] 
and [2]. Integrating [1] seems really easy. [2] would require support 
for Erlang component implementations similar to what Wojtek did with 
Tuscany Java, but should be feasible and opens more perspectives.

It'd be fun to try this out with a real application of ASN.1 like H.323, 
some CDR (call detail record) formats, or another useful module from the 
ITU ASN.1 module database [4]...

[1] http://lionet.info/asn1c/
[2] http://www.erlang.org/doc/apps/asn1/index.html
[3] http://www.openssl.org/
[4] http://www.itu.int/ITU-T/recommendations/fl.aspx?lang=1

Hope this helps
-- 
Jean-Sebastien

Re: Support for latest Axis2/C and Python runtimes

[Giorgio Zoppi <gi...@gmail.com>]

>
> Thanks Giorgio. I looked at your SDO runtime patch in JIRA 3181 and it looks
> fine to me. I think you could just commit it to the SDO trunk.

As soon i can i  submit it.

> With respect to the Axis2C and the SCA runtime, I've just made little
> progress as I've spent more time investigating the Python integration.

I've just reviewed your code and you did a great work. It seems all
C++ gets restructured.
I'll give it a spin this night. It could become an useful runtime for
embedded components (i.e. in telco projects),
going beyond sca or soa platforms, if we provide a binary protocol for
comunications between parts (i.e some sort of rpc using asn.1 for
object serialization).

Cheers,
Giorgio.
-- 
Quiero ser el rayo de sol que cada día te despierta
para hacerte respirar y vivir en me.
"Favola -Moda".

Re: Support for latest Axis2/C and Python runtimes

[Jean-Sebastien Delfino <js...@apache.org>]

Giorgio Zoppi wrote:
> 2009/12/26 Jean-Sebastien Delfino <js...@apache.org>:
>> I'm on vacation, so if I find time in the next few days I'm going to do some
>> coding in sca-cpp and try to get Web Services and Python components working
>> again with the latest Axis2/C and Python runtimes.
> 
> It should be a JIRA that I did some time ago. TUSCANY-3180 I guess, a
> sort of initial work.
> 
> 

Thanks Giorgio. I looked at your SDO runtime patch in JIRA 3181 and it 
looks fine to me. I think you could just commit it to the SDO trunk.

With respect to the Axis2C and the SCA runtime, I've just made little 
progress as I've spent more time investigating the Python integration.

The Python support is working again, and the new integration with Python 
2.6 is actually more straightforward than before. I've been able to wrap 
the function objects used in the rest of the SCA runtime in Python 
Callable objects and have also implemented simpler conversions between 
value types and PyObjects. In fact, that's not much code at all, about 
250 locs to hook the interpreter [1], 150 to capture the Python standard 
out and err [2], and 50 to integrate with the server [3].

With respect to Axis2C, I've started to wrap it in a utility component 
that could be used as a sort of 'SOAP gateway SCA component' instead of 
a full blown SCA binding. I'd like to experiment a bit with that 
approach and see how that compares with a real SOAP binding. I'll try to 
give an update and commit some of this code if I find more time to play 
with it this weekend or the next one...

[1] 
http://svn.apache.org/repos/asf/tuscany/sca-cpp/trunk/modules/python/eval.hpp
[2] 
http://svn.apache.org/repos/asf/tuscany/sca-cpp/trunk/modules/python/io.hpp
[3] 
http://svn.apache.org/repos/asf/tuscany/sca-cpp/trunk/modules/server/mod-python.hpp

-- 
Jean-Sebastien

Re: Support for latest Axis2/C and Python runtimes

[Giorgio Zoppi <gi...@gmail.com>]

2009/12/26 Jean-Sebastien Delfino <js...@apache.org>:
> I'm on vacation, so if I find time in the next few days I'm going to do some
> coding in sca-cpp and try to get Web Services and Python components working
> again with the latest Axis2/C and Python runtimes.

It should be a JIRA that I did some time ago. TUSCANY-3180 I guess, a
sort of initial work.


-- 
Quiero ser el rayo de sol que cada día te despierta
para hacerte respirar y vivir en me.
"Favola -Moda".