You are viewing a plain text version of this content. The canonical link for it is here.
Posted to c-dev@axis.apache.org by "Lefrancois, Carl" <Ca...@axa-canada.com> on 2008/12/11 16:56:12 UTC
RE : Multi-threading
Hello Patrick,
Manjula kindly provided a best-practice example for how to do
multi-threading with Axis2/C [1] (at least on the client side) if I
remember correctly, it was one environment per thread, and the stubs
were per call? Well in his example it is svc_client and not stub, but
if I understand correctly, those two concepts are one-to-one, meaning
one stub has one svc_client.
>From my point of view it seems the troubles you are encountering are
based in areas where Axis does little or nothing to help the programmer.
I have not seen any code to manage thread creation and resources acces
in the Axis2/C project. Problems like resource deadlock on the
configuration file are outside the scope of what Axis2/C has to offer.
What Axis does well is freeing resources (once we figure out how to set
everything up right!) so I am a little confused as to where exactly the
limitations are. You say the callback system provided is not good in
terms of freeing resources, but have you tried freeing your resources
from another function which itself waits for the callback to occur?
(either error callback or success callback) I think this is the way
Axis was designed with as implied by Dimuthu: wait in a loop in your
main thread while the callbacks are outstanding, do no cleanup in the
callback itself, let that thread exit completely and after it is done,
then from your main thread detect that the callback ocurred and do the
cleanup there.
For environment vs stub issues, there is no alternative but to take
ownership of this problem directly and implement some synchronisation
outside the scope of Axis2/C. Your code synchronises creation of
threads and initialisation to avoid having deadlock problems. Maybe
there is some improvement to be made to Axis here?
My reason for responding though is really to comment on this phrase:
"Threads are a rather expensive resource to use for just waiting on an
IO completion". It may be my lack of understanding, but I am pretty
sure that -- at least in the win32 tcp/ip stack -- once your thread goes
into asynchronous communication on a socket, you do not see it again
until there is some result. This means if there is a timeout your
thread is inactive for a long time. How can one thread wait on more
than one asychronous communication? I admit this would be a far better
solution, however from my understanding of winsock2 it is not possible.
Seen this way, one thread per socket communication is maybe expensive in
resources, but it is the only way to ensure your main thread continues
to operate in a timely fashion.
Hth
Carl
[1] http://marc.info/?l=axis-c-user&m=118404667311058&w=2
-----Message d'origine-----
De : Patrick van Beem [mailto:patrick.van.beem@quintiq.com]
Envoyé : jeudi, décembre 11, 2008 05:46
À : axis-c-dev@ws.apache.org
Objet : Multi-threading
Hello,
I'm experiencing serious limitations while using axis in a
multi-threading environment. I know axis is not fully designed (yet?)
for multi-threading, but I think some 'small' changes might make it more
usable. One can't deny multi-threading. Here my 2 cents:
* Callback and resources
In a multi-threading environment, an asynchronous job is often
responsible for freeing it's own resources (in this case, the
environment, the stub and possible extra application dependent data). In
the axis framework, this can't be done in the callback, since the
framework will use the stub after the callback is finished. This makes
it impossible for an application to free resources after a call has
finished (because you never know exactly when the call (including the
framework part of it) is finished. This makes the asynchronous
implementation of axis useless in a job-oriented multi-threading
environment. Improvements might include adding an extra callback to free
resources, that is made when axis has freed all its resources and does
not need any resources the user might free.
* Environment vs stub
Since every thread needs it's own environment and stub in axis and (as
stated above) in a multi-threaded environment a job is often responsible
of it's own resources, every job needs to allocate and initialize it's
environment and stub for each call (or keep one set initialized for
every thread, but that's not feasible when one thread can be used for
multiple purposes / jobs and/or threads can be added dynamically). And
free it after the job is finished. Apart from the (slight) performance
drawback, this poses another problem: Part of the stub initialization
includes reading the configuration file. When two threads try to do this
simultaneously, the last one who tries this, fails. Solutions to this
problem might include:
- Create a clone function for the stub, so all threads can clone from
template stub, instead of reading from disk.
- Separate the state information from the stub and put it in the
environment, so the stub can be used read-only and shared by threads.
* Async model
The asynchronous call implementation of axis is based on creating new
threads that just wait on a response. Threads are a rather expensive
resource to use for just waiting on an IO completion (and then
performing some small task). It might be better that the waiting on all
outstanding IO is done by one single thread. The work after the IO
completed can then be done by either that thread, or a (small and
static) thread pool. That way, no threads have to be created / deleted
on the flow, not more than one thread is waiting on IO and no high
amount of threads will exist when a lot of asynchronous calls exist in
parallel. My axis knowledge is not enough to see solutions for this
within axis right now.
What do you all think about these points?
Regards,
--
Patrick van Beem
Sr. Software engineer
Quintiq
T +31 (0) 73 691 07 39
F +31 (0) 73 691 07 54
M +31 (0) 06 15 01 65 83
E patrick.van.beem@quintiq.com
I www.quintiq.com
This message contains information that may be privileged or confidential
and is the property of Quintiq. It is only intended for the person to
whom it is addressed. If you are not the intended recipient, you are not
authorized to read, print, retain, copy, disseminate, distribute or use
this message or any part thereof. If you have received this message in
error, please notify the sender immediately and delete all copies of
this message. Please note that e-mails are susceptible to change,
therefore they are not binding.
_____
"Ce message est confidentiel, à l'usage exclusif du destinataire
ci-dessus et son contenu ne représente en aucun cas un engagement de la
part de AXA, sauf en cas de stipulation expresse et par écrit de la part
de AXA. Toute publication, utilisation ou diffusion, même partielle,
doit être autorisée préalablement. Si vous n'êtes pas destinataire de ce
message, merci d'en avertir immédiatement l'expéditeur."
"This e-mail message is confidential, for the exclusive use of the
addressee and its contents shall not constitute a commitment by AXA,
except as otherwise specifically provided in writing by AXA. Any
unauthorized disclosure, use or dissemination, either whole or partial,
is prohibited. If you are not the intended recipient of the message,
please notify the sender immediately."
Re: RE : Multi-threading
Posted by Damitha Kumarage <da...@gmail.com>.
Patrick van Beem wrote:
> Hello Carl,
>
>
>> What Axis does well is freeing resources (once we figure out how to set
>> everything up right!) so I am a little confused as to where exactly the
>> limitations are. You say the callback system provided is not good in
>> terms of freeing resources, but have you tried freeing your resources
>> from another function which itself waits for the callback to occur?
>> (either error callback or success callback) I think this is the way
>> Axis was designed with as implied by Dimuthu: wait in a loop in your
>> main thread while the callbacks are outstanding, do no cleanup in the
>> callback itself, let that thread exit completely and after it is done,
>> then from your main thread detect that the callback ocurred and do the
>> cleanup there.
>>
>
> Correct. But I think the design is missing one thing. If I allocate the stub and env and then do an async call, I'm not allowed to free those two resources in the callback, because they're used by the axis framework. But if I signal the main thread from the callback, to free the resources, the callback might be switched out directly after this signal, and the main thread might free the resources before the callback ended and the axis framework used them. As you indicate, the only safe way is to wait until the thread is finished. But the axis framework does not provide an api to find out which thread is processing you request. And it shouldn't, because the thread mechanism is an implementation detail of the axis framework. Future versions might re-use the thread or even use no threading at all for asynchronous calls. So the only safe way to free resources is for the axis framework to signal the caller that the resources are no longer needed. A (second?) callback is the most used (elegant) way to do this.
Yes, this problem exists in the current implementation. A second
callback as you said is the ideal solution.
thanks,
Damitha
> Right now, the framework does not provide a safe way of freeing resources in async calls.
>
>
>> My reason for responding though is really to comment on this phrase:
>> "Threads are a rather expensive resource to use for just waiting on an
>> IO completion". It may be my lack of understanding, but I am pretty
>> sure that -- at least in the win32 tcp/ip stack -- once your thread goes
>> into asynchronous communication on a socket, you do not see it again
>> until there is some result. This means if there is a timeout your
>> thread is inactive for a long time.
>>
>
> Correct. So if I've got a couple of hundred outstanding calls, they all consume precious memory. In our case, this is a lot of memory, since we have a heavy server applications with a greedy memory allocation strategy per thread (for performance) and a rather large default stacks. Of course, both can be optimized for the 'just waiting on io-completion'-threads...
> CPU-wise, it's no problem.
>
>
>> How can one thread wait on more
>> than one asychronous communication? I admit this would be a far better
>> solution, however from my understanding of winsock2 it is not possible.
>>
>
> With the fd_set in winsock and the select() function, you can wait at a maximum of 64 (current implementation) sockets at once. With I/O Completion Ports you can use one thread for an infinite number of ports (though a pool of threads might be a good idea if the number of sockets grows large). This is also used by the well known boost (C++) library. Mechanisms like these would be a much better implementation. But I think they don't fit well in the modular (transportation) design of axis, since they require knowledge about the lower level transportation on a higher level.
>
>
>> Seen this way, one thread per socket communication is maybe expensive in
>> resources, but it is the only way to ensure your main thread continues
>> to operate in a timely fashion.
>>
>
> But prone to explode with a log of async calls. As a 'workaround' I've now my own static-sized thread pool that perform synchronous calls. If there are more async calls then threads in the pool, they're queued.
>
> Thank you for your input.
>
>
--
__________________________________________________________________
Damitha Kumarage
http://people.apache.org/
__________________________________________________________________
RE: Multi-threading
Posted by Bill Mitchell <bm...@austin.rr.com>.
Patrick, thanks for your other note addressing Supun. I, too, was a little
puzzled trying to understand his response, as I had not experienced any
instances of crashes/failures from double frees in Axis2c (provided I
avoided libxml). Thinking about Supun's comments, I concluded that this
must be a difference between sync and async I/O. From your description, the
implementation architecture I chose matches yours: a pool of threads
performing synchronous I/O. Thus I fortuitously avoided the problem that
Carl and you ran into with asynchronous calls, and that you worked around
when you went to synchronous calls.
Best regards,
Bill Mitchell
wtmitchell3@acm.org
-----Original Message-----
From: Patrick van Beem [mailto:patrick.van.beem@quintiq.com]
Sent: Tuesday, December 16, 2008 6:20 AM
To: Apache AXIS C Developers List
Subject: Re: RE : Multi-threading
Hello Carl,
> What Axis does well is freeing resources (once we figure out how to set
> everything up right!) so I am a little confused as to where exactly the
> limitations are. You say the callback system provided is not good in
> terms of freeing resources, but have you tried freeing your resources
> from another function which itself waits for the callback to occur?
> (either error callback or success callback) I think this is the way
> Axis was designed with as implied by Dimuthu: wait in a loop in your
> main thread while the callbacks are outstanding, do no cleanup in the
> callback itself, let that thread exit completely and after it is done,
> then from your main thread detect that the callback ocurred and do the
> cleanup there.
Correct. But I think the design is missing one thing. If I allocate the stub
and env and then do an async call, I'm not allowed to free those two
resources in the callback, because they're used by the axis framework. But
if I signal the main thread from the callback, to free the resources, the
callback might be switched out directly after this signal, and the main
thread might free the resources before the callback ended and the axis
framework used them. As you indicate, the only safe way is to wait until the
thread is finished. But the axis framework does not provide an api to find
out which thread is processing you request. And it shouldn't, because the
thread mechanism is an implementation detail of the axis framework. Future
versions might re-use the thread or even use no threading at all for
asynchronous calls. So the only safe way to free resources is for the axis
framework to signal the caller that the resources are no longer needed. A
(second?) callback is the most used (elegant) way to do this. Right now, the
framework does not provide a safe way of freeing resources in async calls.
> My reason for responding though is really to comment on this phrase:
> "Threads are a rather expensive resource to use for just waiting on an
> IO completion". It may be my lack of understanding, but I am pretty
> sure that -- at least in the win32 tcp/ip stack -- once your thread goes
> into asynchronous communication on a socket, you do not see it again
> until there is some result. This means if there is a timeout your
> thread is inactive for a long time.
Correct. So if I've got a couple of hundred outstanding calls, they all
consume precious memory. In our case, this is a lot of memory, since we have
a heavy server applications with a greedy memory allocation strategy per
thread (for performance) and a rather large default stacks. Of course, both
can be optimized for the 'just waiting on io-completion'-threads...
CPU-wise, it's no problem.
> How can one thread wait on more
> than one asychronous communication? I admit this would be a far better
> solution, however from my understanding of winsock2 it is not possible.
With the fd_set in winsock and the select() function, you can wait at a
maximum of 64 (current implementation) sockets at once. With I/O Completion
Ports you can use one thread for an infinite number of ports (though a pool
of threads might be a good idea if the number of sockets grows large). This
is also used by the well known boost (C++) library. Mechanisms like these
would be a much better implementation. But I think they don't fit well in
the modular (transportation) design of axis, since they require knowledge
about the lower level transportation on a higher level.
> Seen this way, one thread per socket communication is maybe expensive in
> resources, but it is the only way to ensure your main thread continues
> to operate in a timely fashion.
But prone to explode with a log of async calls. As a 'workaround' I've now
my own static-sized thread pool that perform synchronous calls. If there are
more async calls then threads in the pool, they're queued.
Thank you for your input.
--
Patrick van Beem
Sr. Software engineer
Quintiq
T +31 (0) 73 691 07 39
F +31 (0) 73 691 07 54
M +31 (0) 06 15 01 65 83
E patrick.van.beem@quintiq.com
I www.quintiq.com
This message contains information that may be privileged or confidential and
is the property of Quintiq. It is only intended for the person to whom it is
addressed. If you are not the intended recipient, you are not authorized to
read, print, retain, copy, disseminate, distribute or use this message or
any part thereof. If you have received this message in error, please notify
the sender immediately and delete all copies of this message. Please note
that e-mails are susceptible to change, therefore they are not binding.
Re: RE : Multi-threading
Posted by Patrick van Beem <pa...@quintiq.com>.
Hello Carl,
> What Axis does well is freeing resources (once we figure out how to set
> everything up right!) so I am a little confused as to where exactly the
> limitations are. You say the callback system provided is not good in
> terms of freeing resources, but have you tried freeing your resources
> from another function which itself waits for the callback to occur?
> (either error callback or success callback) I think this is the way
> Axis was designed with as implied by Dimuthu: wait in a loop in your
> main thread while the callbacks are outstanding, do no cleanup in the
> callback itself, let that thread exit completely and after it is done,
> then from your main thread detect that the callback ocurred and do the
> cleanup there.
Correct. But I think the design is missing one thing. If I allocate the stub and env and then do an async call, I'm not allowed to free those two resources in the callback, because they're used by the axis framework. But if I signal the main thread from the callback, to free the resources, the callback might be switched out directly after this signal, and the main thread might free the resources before the callback ended and the axis framework used them. As you indicate, the only safe way is to wait until the thread is finished. But the axis framework does not provide an api to find out which thread is processing you request. And it shouldn't, because the thread mechanism is an implementation detail of the axis framework. Future versions might re-use the thread or even use no threading at all for asynchronous calls. So the only safe way to free resources is for the axis framework to signal the caller that the resources are no longer needed. A (second?) callback is the most used (elegant) way to do this. Right now, the framework does not provide a safe way of freeing resources in async calls.
> My reason for responding though is really to comment on this phrase:
> "Threads are a rather expensive resource to use for just waiting on an
> IO completion". It may be my lack of understanding, but I am pretty
> sure that -- at least in the win32 tcp/ip stack -- once your thread goes
> into asynchronous communication on a socket, you do not see it again
> until there is some result. This means if there is a timeout your
> thread is inactive for a long time.
Correct. So if I've got a couple of hundred outstanding calls, they all consume precious memory. In our case, this is a lot of memory, since we have a heavy server applications with a greedy memory allocation strategy per thread (for performance) and a rather large default stacks. Of course, both can be optimized for the 'just waiting on io-completion'-threads...
CPU-wise, it's no problem.
> How can one thread wait on more
> than one asychronous communication? I admit this would be a far better
> solution, however from my understanding of winsock2 it is not possible.
With the fd_set in winsock and the select() function, you can wait at a maximum of 64 (current implementation) sockets at once. With I/O Completion Ports you can use one thread for an infinite number of ports (though a pool of threads might be a good idea if the number of sockets grows large). This is also used by the well known boost (C++) library. Mechanisms like these would be a much better implementation. But I think they don't fit well in the modular (transportation) design of axis, since they require knowledge about the lower level transportation on a higher level.
> Seen this way, one thread per socket communication is maybe expensive in
> resources, but it is the only way to ensure your main thread continues
> to operate in a timely fashion.
But prone to explode with a log of async calls. As a 'workaround' I've now my own static-sized thread pool that perform synchronous calls. If there are more async calls then threads in the pool, they're queued.
Thank you for your input.
--
Patrick van Beem
Sr. Software engineer
Quintiq
T +31 (0) 73 691 07 39
F +31 (0) 73 691 07 54
M +31 (0) 06 15 01 65 83
E patrick.van.beem@quintiq.com
I www.quintiq.com
This message contains information that may be privileged or confidential and is the property of Quintiq. It is only intended for the person to whom it is addressed. If you are not the intended recipient, you are not authorized to read, print, retain, copy, disseminate, distribute or use this message or any part thereof. If you have received this message in error, please notify the sender immediately and delete all copies of this message. Please note that e-mails are susceptible to change, therefore they are not binding.
Re: Multi-threading
Posted by Patrick van Beem <pa...@quintiq.com>.
Hello Supun,
> First we need to think why
> we need to execute Axis2/C clients in multiple threads instead of using
> separate clients in different threads.
I had to read that sentence at least three times before I understood it :-)
You mail does shine some light on the subject for me. Both the overhead and performance are an issue for us. And since IO is involved when initializing a stub, performance might be more important.
We selected axis for our soap interface because of the adb code generation (ease of use while adding more and more soap clients). So writing a lot of code for the mep instead of 'just using the framework' is not really an option for us. I think we stay with the 'use our own thread pool and do sync calls from there' implementation of 'async' calls for now.
Thanks for the input.
> One advantage of using a single
> client is the overhead associated with multiple clients. A single
> axis2_svc_client requires multiple axis2 configurations and multiple
> environments. This is the only reasong comes to my mind right now for not
> using different axis2_svc_clients in different threads. If you have any
> other requirements please share with us.
>
> If performance is the concern associated with creating multiple clients,
> there is a solution to the problem. The real entity that do the most
> important work in the client side is axis2_mep_client. axis2_svc_client is a
> wrapper around the mep client for make the job easier for the client
> programmer. But axis2_svc_client is not designed for a multithreaded
> environment. The problems with axis2_svc_client running in multiple threads
> is it keep tracks of the various objects from previous invokations. This
> leads to double free this resources in multiple threading environments.
>
> But if we can use the mep client these problems won't be there. The only
> problem with mep client is it requires quite a bit of coding to make it
> work. If we can provide a simple thread safe method set for directly
> accessing the mep client it will be really useful. So devs what do you
> think?
>
> Supun..
--
Patrick van Beem
Sr. Software engineer
Quintiq
T +31 (0) 73 691 07 39
F +31 (0) 73 691 07 54
M +31 (0) 06 15 01 65 83
E patrick.van.beem@quintiq.com
I www.quintiq.com
This message contains information that may be privileged or confidential and is the property of Quintiq. It is only intended for the person to whom it is addressed. If you are not the intended recipient, you are not authorized to read, print, retain, copy, disseminate, distribute or use this message or any part thereof. If you have received this message in error, please notify the sender immediately and delete all copies of this message. Please note that e-mails are susceptible to change, therefore they are not binding.
Re: Multi-threading
Posted by Manjula Peiris <ma...@wso2.com>.
On Sat, 2008-12-13 at 13:02 +0500, Supun Kamburugamuva wrote:
> Hi all,
>
> This question has being popping up time to time. First we need to
> think why we need to execute Axis2/C clients in multiple threads
> instead of using separate clients in different threads. One advantage
> of using a single client is the overhead associated with multiple
> clients. A single axis2_svc_client requires multiple axis2
> configurations and multiple environments. This is the only reasong
> comes to my mind right now for not using different axis2_svc_clients
> in different threads. If you have any other requirements please share
> with us.
>
> If performance is the concern associated with creating multiple
> clients, there is a solution to the problem. The real entity that do
> the most important work in the client side is axis2_mep_client.
Not mep_client it is op_client.
> axis2_svc_client is a wrapper around the mep client for make the job
> easier for the client programmer. But axis2_svc_client is not designed
> for a multithreaded environment. The problems with axis2_svc_client
> running in multiple threads is it keep tracks of the various objects
> from previous invokations. This leads to double free this resources in
> multiple threading environments.
>
> But if we can use the mep client these problems won't be there. The
> only problem with mep client is it requires quite a bit of coding to
> make it work. If we can provide a simple thread safe method set for
> directly accessing the mep client it will be really useful. So devs
> what do you think?
Actually AFAIK, even each svc_client_send_receive creates a new
op_client and this op_client uses the configurations created from
svc_client. So svc_client is like a container which keeps the
configurations for the use of op_clients. What we need to do is we
should provide methods to application clients to create configurations
of their own. So they can pass these to op_client and do the work.
>
> Supun..
>
> On Thu, Dec 11, 2008 at 10:37 PM, Bill Mitchell
> <bm...@austin.rr.com> wrote:
> Patrick, when building a multi-threaded Axis2C client I too
> was
> concerned about the multiple environments. Although your
> statement is correct in a sense that each thread needs its own
> environment/stub, these environments can in fact share much of
> the underlying structures. In practice, each thread needs its
> own error stack, but it can certainly share the allocator and
> logger. And the configuration information is associated with
> the allocator. There is a primitive function
> axutil_env_create_with_error_log_thread_pool() that lets you
> share the substructures already created for the "global"
> environment
> created once for the application. This way the configuration
> information is read only once. Axutil_env_free_masked() lets
> each thread free just its error stack upon termination,
> leaving
> the allocator et.al. intact.
>
> I was not dealing with asynchronous operation in my
> application,
> so I don't know if you might need a separate thread-pool for
> each created environment.
>
> Good luck,
> Bill Mitchell
>
>
> -----Original Message-----
> From: Lefrancois, Carl [mailto:Carl.Lefrancois@axa-canada.com]
> Sent: Thursday, December 11, 2008 9:56 AM
> To: Apache AXIS C Developers List
> Subject: RE : Multi-threading
>
> Hello Patrick,
>
> Manjula kindly provided a best-practice example for how to do
> multi-threading with Axis2/C [1] (at least on the client
> side) if I
> remember correctly, it was one environment per thread, and the
> stubs
> were per call? Well in his example it is svc_client and not
> stub, but
> if I understand correctly, those two concepts are one-to-one,
> meaning
> one stub has one svc_client.
>
> >From my point of view it seems the troubles you are
> encountering are
> based in areas where Axis does little or nothing to help the
> programmer.
> I have not seen any code to manage thread creation and
> resources acces
> in the Axis2/C project. Problems like resource deadlock on
> the
> configuration file are outside the scope of what Axis2/C has
> to offer.
>
> What Axis does well is freeing resources (once we figure out
> how to set
> everything up right!) so I am a little confused as to where
> exactly the
> limitations are. You say the callback system provided is not
> good in
> terms of freeing resources, but have you tried freeing your
> resources
> from another function which itself waits for the callback to
> occur?
> (either error callback or success callback) I think this is
> the way
> Axis was designed with as implied by Dimuthu: wait in a loop
> in your
> main thread while the callbacks are outstanding, do no cleanup
> in the
> callback itself, let that thread exit completely and after it
> is done,
> then from your main thread detect that the callback ocurred
> and do the
> cleanup there.
>
> For environment vs stub issues, there is no alternative but to
> take
> ownership of this problem directly and implement some
> synchronisation
> outside the scope of Axis2/C. Your code synchronises creation
> of
> threads and initialisation to avoid having deadlock problems.
> Maybe
> there is some improvement to be made to Axis here?
>
> My reason for responding though is really to comment on this
> phrase:
> "Threads are a rather expensive resource to use for just
> waiting on an
> IO completion". It may be my lack of understanding, but I am
> pretty
> sure that -- at least in the win32 tcp/ip stack -- once your
> thread goes
> into asynchronous communication on a socket, you do not see it
> again
> until there is some result. This means if there is a timeout
> your
> thread is inactive for a long time. How can one thread wait
> on more
> than one asychronous communication? I admit this would be a
> far better
> solution, however from my understanding of winsock2 it is not
> possible.
>
> Seen this way, one thread per socket communication is maybe
> expensive in
> resources, but it is the only way to ensure your main thread
> continues
> to operate in a timely fashion.
>
> Hth
>
> Carl
>
>
> [1]
> http://marc.info/?l=axis-c-user&m=118404667311058&w=2
>
>
>
>
>
> -----Message d'origine-----
> De : Patrick van Beem [mailto:patrick.van.beem@quintiq.com]
> Envoyé : jeudi, décembre 11, 2008 05:46
> À : axis-c-dev@ws.apache.org
> Objet : Multi-threading
>
>
> Hello,
>
> I'm experiencing serious limitations while using axis in a
> multi-threading environment. I know axis is not fully designed
> (yet?)
> for multi-threading, but I think some 'small' changes might
> make it more
> usable. One can't deny multi-threading. Here my 2 cents:
>
>
> * Callback and resources
> In a multi-threading environment, an asynchronous job is often
> responsible for freeing it's own resources (in this case, the
> environment, the stub and possible extra application dependent
> data). In
> the axis framework, this can't be done in the callback, since
> the
> framework will use the stub after the callback is finished.
> This makes
> it impossible for an application to free resources after a
> call has
> finished (because you never know exactly when the call
> (including the
> framework part of it) is finished. This makes the asynchronous
> implementation of axis useless in a job-oriented
> multi-threading
> environment. Improvements might include adding an extra
> callback to free
> resources, that is made when axis has freed all its resources
> and does
> not need any resources the user might free.
>
>
> * Environment vs stub
> Since every thread needs it's own environment and stub in axis
> and (as
> stated above) in a multi-threaded environment a job is often
> responsible
> of it's own resources, every job needs to allocate and
> initialize it's
> environment and stub for each call (or keep one set
> initialized for
> every thread, but that's not feasible when one thread can be
> used for
> multiple purposes / jobs and/or threads can be added
> dynamically). And
> free it after the job is finished. Apart from the (slight)
> performance
> drawback, this poses another problem: Part of the stub
> initialization
> includes reading the configuration file. When two threads try
> to do this
> simultaneously, the last one who tries this, fails. Solutions
> to this
> problem might include:
> - Create a clone function for the stub, so all threads can
> clone from
> template stub, instead of reading from disk.
> - Separate the state information from the stub and put it in
> the
> environment, so the stub can be used read-only and shared by
> threads.
>
>
> * Async model
> The asynchronous call implementation of axis is based on
> creating new
> threads that just wait on a response. Threads are a rather
> expensive
> resource to use for just waiting on an IO completion (and then
> performing some small task). It might be better that the
> waiting on all
> outstanding IO is done by one single thread. The work after
> the IO
> completed can then be done by either that thread, or a (small
> and
> static) thread pool. That way, no threads have to be created /
> deleted
> on the flow, not more than one thread is waiting on IO and no
> high
> amount of threads will exist when a lot of asynchronous calls
> exist in
> parallel. My axis knowledge is not enough to see solutions for
> this
> within axis right now.
>
>
> What do you all think about these points?
>
> Regards,
>
> --
>
>
> Patrick van Beem
> Sr. Software engineer
>
> Quintiq
>
> T +31 (0) 73 691 07 39
> F +31 (0) 73 691 07 54
> M +31 (0) 06 15 01 65 83
> E patrick.van.beem@quintiq.com
> I www.quintiq.com
>
>
>
> This message contains information that may be privileged or
> confidential
> and is the property of Quintiq. It is only intended for the
> person to
> whom it is addressed. If you are not the intended recipient,
> you are not
> authorized to read, print, retain, copy, disseminate,
> distribute or use
> this message or any part thereof. If you have received this
> message in
> error, please notify the sender immediately and delete all
> copies of
> this message. Please note that e-mails are susceptible to
> change,
> therefore they are not binding.
> _____
>
> "Ce message est confidentiel, à l'usage exclusif du
> destinataire
> ci-dessus et son contenu ne représente en aucun cas un
> engagement de la
> part de AXA, sauf en cas de stipulation expresse et par écrit
> de la part
> de AXA. Toute publication, utilisation ou diffusion, même
> partielle,
> doit être autorisée préalablement. Si vous n'êtes pas
> destinataire de ce
> message, merci d'en avertir immédiatement l'expéditeur."
>
> "This e-mail message is confidential, for the exclusive use of
> the
> addressee and its contents shall not constitute a commitment
> by AXA,
> except as otherwise specifically provided in writing by AXA.
> Any
> unauthorized disclosure, use or dissemination, either whole or
> partial,
> is prohibited. If you are not the intended recipient of the
> message,
> please notify the sender immediately."
>
>
>
>
>
>
> --
> Software Engineer, WSO2 Inc
> http://wso2.org
> Web Services with Axis2/C http://wsaxc.blospot.com
>
Re: Multi-threading
Posted by Supun Kamburugamuva <su...@gmail.com>.
Hi all,
This question has being popping up time to time. First we need to think why
we need to execute Axis2/C clients in multiple threads instead of using
separate clients in different threads. One advantage of using a single
client is the overhead associated with multiple clients. A single
axis2_svc_client requires multiple axis2 configurations and multiple
environments. This is the only reasong comes to my mind right now for not
using different axis2_svc_clients in different threads. If you have any
other requirements please share with us.
If performance is the concern associated with creating multiple clients,
there is a solution to the problem. The real entity that do the most
important work in the client side is axis2_mep_client. axis2_svc_client is a
wrapper around the mep client for make the job easier for the client
programmer. But axis2_svc_client is not designed for a multithreaded
environment. The problems with axis2_svc_client running in multiple threads
is it keep tracks of the various objects from previous invokations. This
leads to double free this resources in multiple threading environments.
But if we can use the mep client these problems won't be there. The only
problem with mep client is it requires quite a bit of coding to make it
work. If we can provide a simple thread safe method set for directly
accessing the mep client it will be really useful. So devs what do you
think?
Supun..
On Thu, Dec 11, 2008 at 10:37 PM, Bill Mitchell <bm...@austin.rr.com>wrote:
> Patrick, when building a multi-threaded Axis2C client I too was
> concerned about the multiple environments. Although your
> statement is correct in a sense that each thread needs its own
> environment/stub, these environments can in fact share much of
> the underlying structures. In practice, each thread needs its
> own error stack, but it can certainly share the allocator and
> logger. And the configuration information is associated with
> the allocator. There is a primitive function
> axutil_env_create_with_error_log_thread_pool() that lets you
> share the substructures already created for the "global" environment
> created once for the application. This way the configuration
> information is read only once. Axutil_env_free_masked() lets
> each thread free just its error stack upon termination, leaving
> the allocator et.al. intact.
>
> I was not dealing with asynchronous operation in my application,
> so I don't know if you might need a separate thread-pool for
> each created environment.
>
> Good luck,
> Bill Mitchell
>
> -----Original Message-----
> From: Lefrancois, Carl [mailto:Carl.Lefrancois@axa-canada.com]
> Sent: Thursday, December 11, 2008 9:56 AM
> To: Apache AXIS C Developers List
> Subject: RE : Multi-threading
>
> Hello Patrick,
>
> Manjula kindly provided a best-practice example for how to do
> multi-threading with Axis2/C [1] (at least on the client side) if I
> remember correctly, it was one environment per thread, and the stubs
> were per call? Well in his example it is svc_client and not stub, but
> if I understand correctly, those two concepts are one-to-one, meaning
> one stub has one svc_client.
>
> From my point of view it seems the troubles you are encountering are
> based in areas where Axis does little or nothing to help the programmer.
> I have not seen any code to manage thread creation and resources acces
> in the Axis2/C project. Problems like resource deadlock on the
> configuration file are outside the scope of what Axis2/C has to offer.
>
> What Axis does well is freeing resources (once we figure out how to set
> everything up right!) so I am a little confused as to where exactly the
> limitations are. You say the callback system provided is not good in
> terms of freeing resources, but have you tried freeing your resources
> from another function which itself waits for the callback to occur?
> (either error callback or success callback) I think this is the way
> Axis was designed with as implied by Dimuthu: wait in a loop in your
> main thread while the callbacks are outstanding, do no cleanup in the
> callback itself, let that thread exit completely and after it is done,
> then from your main thread detect that the callback ocurred and do the
> cleanup there.
>
> For environment vs stub issues, there is no alternative but to take
> ownership of this problem directly and implement some synchronisation
> outside the scope of Axis2/C. Your code synchronises creation of
> threads and initialisation to avoid having deadlock problems. Maybe
> there is some improvement to be made to Axis here?
>
> My reason for responding though is really to comment on this phrase:
> "Threads are a rather expensive resource to use for just waiting on an
> IO completion". It may be my lack of understanding, but I am pretty
> sure that -- at least in the win32 tcp/ip stack -- once your thread goes
> into asynchronous communication on a socket, you do not see it again
> until there is some result. This means if there is a timeout your
> thread is inactive for a long time. How can one thread wait on more
> than one asychronous communication? I admit this would be a far better
> solution, however from my understanding of winsock2 it is not possible.
>
> Seen this way, one thread per socket communication is maybe expensive in
> resources, but it is the only way to ensure your main thread continues
> to operate in a timely fashion.
>
> Hth
>
> Carl
>
>
> [1] http://marc.info/?l=axis-c-user&m=118404667311058&w=2
>
>
>
>
>
> -----Message d'origine-----
> De : Patrick van Beem [mailto:patrick.van.beem@quintiq.com]
> Envoyé : jeudi, décembre 11, 2008 05:46
> À : axis-c-dev@ws.apache.org
> Objet : Multi-threading
>
>
> Hello,
>
> I'm experiencing serious limitations while using axis in a
> multi-threading environment. I know axis is not fully designed (yet?)
> for multi-threading, but I think some 'small' changes might make it more
> usable. One can't deny multi-threading. Here my 2 cents:
>
>
> * Callback and resources
> In a multi-threading environment, an asynchronous job is often
> responsible for freeing it's own resources (in this case, the
> environment, the stub and possible extra application dependent data). In
> the axis framework, this can't be done in the callback, since the
> framework will use the stub after the callback is finished. This makes
> it impossible for an application to free resources after a call has
> finished (because you never know exactly when the call (including the
> framework part of it) is finished. This makes the asynchronous
> implementation of axis useless in a job-oriented multi-threading
> environment. Improvements might include adding an extra callback to free
> resources, that is made when axis has freed all its resources and does
> not need any resources the user might free.
>
>
> * Environment vs stub
> Since every thread needs it's own environment and stub in axis and (as
> stated above) in a multi-threaded environment a job is often responsible
> of it's own resources, every job needs to allocate and initialize it's
> environment and stub for each call (or keep one set initialized for
> every thread, but that's not feasible when one thread can be used for
> multiple purposes / jobs and/or threads can be added dynamically). And
> free it after the job is finished. Apart from the (slight) performance
> drawback, this poses another problem: Part of the stub initialization
> includes reading the configuration file. When two threads try to do this
> simultaneously, the last one who tries this, fails. Solutions to this
> problem might include:
> - Create a clone function for the stub, so all threads can clone from
> template stub, instead of reading from disk.
> - Separate the state information from the stub and put it in the
> environment, so the stub can be used read-only and shared by threads.
>
>
> * Async model
> The asynchronous call implementation of axis is based on creating new
> threads that just wait on a response. Threads are a rather expensive
> resource to use for just waiting on an IO completion (and then
> performing some small task). It might be better that the waiting on all
> outstanding IO is done by one single thread. The work after the IO
> completed can then be done by either that thread, or a (small and
> static) thread pool. That way, no threads have to be created / deleted
> on the flow, not more than one thread is waiting on IO and no high
> amount of threads will exist when a lot of asynchronous calls exist in
> parallel. My axis knowledge is not enough to see solutions for this
> within axis right now.
>
>
> What do you all think about these points?
>
> Regards,
>
> --
>
>
> Patrick van Beem
> Sr. Software engineer
>
> Quintiq
>
> T +31 (0) 73 691 07 39
> F +31 (0) 73 691 07 54
> M +31 (0) 06 15 01 65 83
> E patrick.van.beem@quintiq.com
> I www.quintiq.com
>
>
>
> This message contains information that may be privileged or confidential
> and is the property of Quintiq. It is only intended for the person to
> whom it is addressed. If you are not the intended recipient, you are not
> authorized to read, print, retain, copy, disseminate, distribute or use
> this message or any part thereof. If you have received this message in
> error, please notify the sender immediately and delete all copies of
> this message. Please note that e-mails are susceptible to change,
> therefore they are not binding.
> _____
>
> "Ce message est confidentiel, à l'usage exclusif du destinataire
> ci-dessus et son contenu ne représente en aucun cas un engagement de la
> part de AXA, sauf en cas de stipulation expresse et par écrit de la part
> de AXA. Toute publication, utilisation ou diffusion, même partielle,
> doit être autorisée préalablement. Si vous n'êtes pas destinataire de ce
> message, merci d'en avertir immédiatement l'expéditeur."
>
> "This e-mail message is confidential, for the exclusive use of the
> addressee and its contents shall not constitute a commitment by AXA,
> except as otherwise specifically provided in writing by AXA. Any
> unauthorized disclosure, use or dissemination, either whole or partial,
> is prohibited. If you are not the intended recipient of the message,
> please notify the sender immediately."
>
>
>
--
Software Engineer, WSO2 Inc
http://wso2.org
Web Services with Axis2/C http://wsaxc.blospot.com
Re: Multi-threading
Posted by Damitha Kumarage <da...@gmail.com>.
Hi Patrick
Patrick van Beem wrote:
> No. The current implementation of axis relies on an infinite number of available threads. It assumes that when it requests a thread, it's getting one and it then starts the thread with a thread method / data. But in reality, a thread pool would have a finite number of (re-used) threads. In the case of axis, this would mean that the thread requesting the new thread would block until a new thread is available. This is not what you want. For axis to work with a finite number of threads, the way it uses threads should change. It should not request a thread, but it should submit a job (probably the thread method and data) to the thread pool. The submitting thread can then continue an the thread pool can decide when the job is performed by which thread.
Thanks for the explanation. I agree with it.
thanks,
Damitha
--
__________________________________________________________________
Damitha Kumarage
http://people.apache.org/
__________________________________________________________________
Re: Multi-threading
Posted by Patrick van Beem <pa...@quintiq.com>.
Hello,
--
Patrick van Beem
Sr. Software engineer
Quintiq
T +31 (0) 73 691 07 39
F +31 (0) 73 691 07 54
M +31 (0) 06 15 01 65 83
E patrick.van.beem@quintiq.com
I www.quintiq.com
I was not dealing with asynchronous operation in my application,
>>> so I don't know if you might need a separate thread-pool for
>>> each created environment.
>>>
>>
>> The current implementation of a thread pool in axis is no thread pool but a
> collection of thread creation and deletion methods... So no...
>>
> But when creating environment you can pass your own thread pool. Would
> that not help?
No. The current implementation of axis relies on an infinite number of available threads. It assumes that when it requests a thread, it's getting one and it then starts the thread with a thread method / data. But in reality, a thread pool would have a finite number of (re-used) threads. In the case of axis, this would mean that the thread requesting the new thread would block until a new thread is available. This is not what you want. For axis to work with a finite number of threads, the way it uses threads should change. It should not request a thread, but it should submit a job (probably the thread method and data) to the thread pool. The submitting thread can then continue an the thread pool can decide when the job is performed by which thread.
Regards,
Quintiq Conference "Quintessence 09" Tuesday May 12th, 2009, Country Estate Duin & Kruidberg, near Amsterdam Schiphol, The Netherlands - for more information visit www.quintiq.com
This message contains information that may be privileged or confidential and is the property of Quintiq. It is only intended for the person to whom it is addressed. If you are not the intended recipient, you are not authorized to read, print, retain, copy, disseminate, distribute or use this message or any part thereof. If you have received this message in error, please notify the sender immediately and delete all copies of this message. Please note that e-mails are susceptible to change, therefore they are not binding.
Re: Multi-threading
Posted by Damitha Kumarage <da...@gmail.com>.
Patrick van Beem wrote:
> Hello Bill
>
>
>> Patrick, when building a multi-threaded Axis2C client I too was
>> concerned about the multiple environments. Although your
>> statement is correct in a sense that each thread needs its own
>> environment/stub, these environments can in fact share much of
>> the underlying structures. In practice, each thread needs its
>> own error stack, but it can certainly share the allocator and
>> logger. And the configuration information is associated with
>> the allocator. There is a primitive function
>> axutil_env_create_with_error_log_thread_pool() that lets you
>> share the substructures already created for the "global" environment
>> created once for the application. This way the configuration
>> information is read only once. Axutil_env_free_masked() lets
>> each thread free just its error stack upon termination, leaving
>> the allocator et.al. intact.
>>
>
> Correct. I already use these. But afaik the configuration file is read when creating the stub. And that should be done for each thread / call. So while many resources can indeed be shared, the one needing the most (time-consuming) initialization (I think), can't. I think this would be a great improvement for the future.
>
I think adding this functionality to the stub creation is straight
forward. Just add the stub creation function
axis2_stub_create_with_conf_ctx_and_endpoint_ref_and_client_home(const
axutil_env_t * env, axis2_conf_ctx_t *con_ctx, axis2_endpoint_ref_t *
endpoint_ref, const axis2_char_t * client_home));
which in turn call already available api function
axis2_svc_client_create_with_conf_ctx_and_svc(
const axutil_env_t * env,
const axis2_char_t * client_home,
axis2_conf_ctx_t * conf_ctx,
axis2_svc_t * svc);
>
>> I was not dealing with asynchronous operation in my application,
>> so I don't know if you might need a separate thread-pool for
>> each created environment.
>>
>
> The current implementation of a thread pool in axis is no thread pool but a collection of thread creation and deletion methods... So no...
>
But when creating environment you can pass your own thread pool. Would
that not help?
thanks,
thanks,
Damitha
> Thank you for your input.
>
>
--
__________________________________________________________________
Damitha Kumarage
http://people.apache.org/
__________________________________________________________________
RE: Multi-threading
Posted by Patrick van Beem <pa...@quintiq.com>.
Hello Bill
> Patrick, when building a multi-threaded Axis2C client I too was
> concerned about the multiple environments. Although your
> statement is correct in a sense that each thread needs its own
> environment/stub, these environments can in fact share much of
> the underlying structures. In practice, each thread needs its
> own error stack, but it can certainly share the allocator and
> logger. And the configuration information is associated with
> the allocator. There is a primitive function
> axutil_env_create_with_error_log_thread_pool() that lets you
> share the substructures already created for the "global" environment
> created once for the application. This way the configuration
> information is read only once. Axutil_env_free_masked() lets
> each thread free just its error stack upon termination, leaving
> the allocator et.al. intact.
Correct. I already use these. But afaik the configuration file is read when creating the stub. And that should be done for each thread / call. So while many resources can indeed be shared, the one needing the most (time-consuming) initialization (I think), can't. I think this would be a great improvement for the future.
> I was not dealing with asynchronous operation in my application,
> so I don't know if you might need a separate thread-pool for
> each created environment.
The current implementation of a thread pool in axis is no thread pool but a collection of thread creation and deletion methods... So no...
Thank you for your input.
--
Patrick van Beem
Sr. Software engineer
Quintiq
T +31 (0) 73 691 07 39
F +31 (0) 73 691 07 54
M +31 (0) 06 15 01 65 83
E patrick.van.beem@quintiq.com
I www.quintiq.com
This message contains information that may be privileged or confidential and is the property of Quintiq. It is only intended for the person to whom it is addressed. If you are not the intended recipient, you are not authorized to read, print, retain, copy, disseminate, distribute or use this message or any part thereof. If you have received this message in error, please notify the sender immediately and delete all copies of this message. Please note that e-mails are susceptible to change, therefore they are not binding.
RE: Multi-threading
Posted by Bill Mitchell <bm...@austin.rr.com>.
Patrick, when building a multi-threaded Axis2C client I too was
concerned about the multiple environments. Although your
statement is correct in a sense that each thread needs its own
environment/stub, these environments can in fact share much of
the underlying structures. In practice, each thread needs its
own error stack, but it can certainly share the allocator and
logger. And the configuration information is associated with
the allocator. There is a primitive function
axutil_env_create_with_error_log_thread_pool() that lets you
share the substructures already created for the "global" environment
created once for the application. This way the configuration
information is read only once. Axutil_env_free_masked() lets
each thread free just its error stack upon termination, leaving
the allocator et.al. intact.
I was not dealing with asynchronous operation in my application,
so I don't know if you might need a separate thread-pool for
each created environment.
Good luck,
Bill Mitchell
-----Original Message-----
From: Lefrancois, Carl [mailto:Carl.Lefrancois@axa-canada.com]
Sent: Thursday, December 11, 2008 9:56 AM
To: Apache AXIS C Developers List
Subject: RE : Multi-threading
Hello Patrick,
Manjula kindly provided a best-practice example for how to do
multi-threading with Axis2/C [1] (at least on the client side) if I
remember correctly, it was one environment per thread, and the stubs
were per call? Well in his example it is svc_client and not stub, but
if I understand correctly, those two concepts are one-to-one, meaning
one stub has one svc_client.
>From my point of view it seems the troubles you are encountering are
based in areas where Axis does little or nothing to help the programmer.
I have not seen any code to manage thread creation and resources acces
in the Axis2/C project. Problems like resource deadlock on the
configuration file are outside the scope of what Axis2/C has to offer.
What Axis does well is freeing resources (once we figure out how to set
everything up right!) so I am a little confused as to where exactly the
limitations are. You say the callback system provided is not good in
terms of freeing resources, but have you tried freeing your resources
from another function which itself waits for the callback to occur?
(either error callback or success callback) I think this is the way
Axis was designed with as implied by Dimuthu: wait in a loop in your
main thread while the callbacks are outstanding, do no cleanup in the
callback itself, let that thread exit completely and after it is done,
then from your main thread detect that the callback ocurred and do the
cleanup there.
For environment vs stub issues, there is no alternative but to take
ownership of this problem directly and implement some synchronisation
outside the scope of Axis2/C. Your code synchronises creation of
threads and initialisation to avoid having deadlock problems. Maybe
there is some improvement to be made to Axis here?
My reason for responding though is really to comment on this phrase:
"Threads are a rather expensive resource to use for just waiting on an
IO completion". It may be my lack of understanding, but I am pretty
sure that -- at least in the win32 tcp/ip stack -- once your thread goes
into asynchronous communication on a socket, you do not see it again
until there is some result. This means if there is a timeout your
thread is inactive for a long time. How can one thread wait on more
than one asychronous communication? I admit this would be a far better
solution, however from my understanding of winsock2 it is not possible.
Seen this way, one thread per socket communication is maybe expensive in
resources, but it is the only way to ensure your main thread continues
to operate in a timely fashion.
Hth
Carl
[1] http://marc.info/?l=axis-c-user&m=118404667311058&w=2
-----Message d'origine-----
De : Patrick van Beem [mailto:patrick.van.beem@quintiq.com]
Envoyé : jeudi, décembre 11, 2008 05:46
À : axis-c-dev@ws.apache.org
Objet : Multi-threading
Hello,
I'm experiencing serious limitations while using axis in a
multi-threading environment. I know axis is not fully designed (yet?)
for multi-threading, but I think some 'small' changes might make it more
usable. One can't deny multi-threading. Here my 2 cents:
* Callback and resources
In a multi-threading environment, an asynchronous job is often
responsible for freeing it's own resources (in this case, the
environment, the stub and possible extra application dependent data). In
the axis framework, this can't be done in the callback, since the
framework will use the stub after the callback is finished. This makes
it impossible for an application to free resources after a call has
finished (because you never know exactly when the call (including the
framework part of it) is finished. This makes the asynchronous
implementation of axis useless in a job-oriented multi-threading
environment. Improvements might include adding an extra callback to free
resources, that is made when axis has freed all its resources and does
not need any resources the user might free.
* Environment vs stub
Since every thread needs it's own environment and stub in axis and (as
stated above) in a multi-threaded environment a job is often responsible
of it's own resources, every job needs to allocate and initialize it's
environment and stub for each call (or keep one set initialized for
every thread, but that's not feasible when one thread can be used for
multiple purposes / jobs and/or threads can be added dynamically). And
free it after the job is finished. Apart from the (slight) performance
drawback, this poses another problem: Part of the stub initialization
includes reading the configuration file. When two threads try to do this
simultaneously, the last one who tries this, fails. Solutions to this
problem might include:
- Create a clone function for the stub, so all threads can clone from
template stub, instead of reading from disk.
- Separate the state information from the stub and put it in the
environment, so the stub can be used read-only and shared by threads.
* Async model
The asynchronous call implementation of axis is based on creating new
threads that just wait on a response. Threads are a rather expensive
resource to use for just waiting on an IO completion (and then
performing some small task). It might be better that the waiting on all
outstanding IO is done by one single thread. The work after the IO
completed can then be done by either that thread, or a (small and
static) thread pool. That way, no threads have to be created / deleted
on the flow, not more than one thread is waiting on IO and no high
amount of threads will exist when a lot of asynchronous calls exist in
parallel. My axis knowledge is not enough to see solutions for this
within axis right now.
What do you all think about these points?
Regards,
--
Patrick van Beem
Sr. Software engineer
Quintiq
T +31 (0) 73 691 07 39
F +31 (0) 73 691 07 54
M +31 (0) 06 15 01 65 83
E patrick.van.beem@quintiq.com
I www.quintiq.com
This message contains information that may be privileged or confidential
and is the property of Quintiq. It is only intended for the person to
whom it is addressed. If you are not the intended recipient, you are not
authorized to read, print, retain, copy, disseminate, distribute or use
this message or any part thereof. If you have received this message in
error, please notify the sender immediately and delete all copies of
this message. Please note that e-mails are susceptible to change,
therefore they are not binding.
_____
"Ce message est confidentiel, à l'usage exclusif du destinataire
ci-dessus et son contenu ne représente en aucun cas un engagement de la
part de AXA, sauf en cas de stipulation expresse et par écrit de la part
de AXA. Toute publication, utilisation ou diffusion, même partielle,
doit être autorisée préalablement. Si vous n'êtes pas destinataire de ce
message, merci d'en avertir immédiatement l'expéditeur."
"This e-mail message is confidential, for the exclusive use of the
addressee and its contents shall not constitute a commitment by AXA,
except as otherwise specifically provided in writing by AXA. Any
unauthorized disclosure, use or dissemination, either whole or partial,
is prohibited. If you are not the intended recipient of the message,
please notify the sender immediately."