You are viewing a plain text version of this content. The canonical link for it is here.
Posted to c-user@axis.apache.org by Raghavendra SM <ra...@aylus.com> on 2008/03/05 13:04:31 UTC
Core dumps in Axis libraries
All,
For a particular scenario, our module core dumps in some axis library
functions consistently but with different back traces. I'm attaching the
dumps.log file that has as many as 4 different back-traces produced for
the same scenario.
Also, please find attached the log that has axis generated log for both
successful and erroneous times, axis.log.
To give you all a brief background,
+ we are using axis2c-1.0.0
+ the core dump appears during a particular scenario only. Eg: while
trying to add a list of already existing users to my database. For each
user there is a SOAP request generated and the requests are sequential,
i.e the second request is sent only after the response for previous
request is received.
+ if I increase the log level of my own module that uses axis2c, there
is no dump. On increasing the log level, my module writes quite a bit of
information to my log file.
Since the back trace is pointing to the libraries(for which there is no
source code given), how can this be debugged and fixed?
Any inputs will be of great help. Please reply.
Regards,
~raghav
Re: Core dumps in Axis libraries
Posted by Samisa Abeysinghe <sa...@wso2.com>.
Manjula Peiris wrote:
> On Wed, 2008-03-05 at 18:09 +0530, Raghavendra SM wrote:
>
>> Thanks Manjula,
>>
>> But thats not a viable option for us. We would still want to continue
>> using axis2C-1.0 for some more time.
>>
>
> IMHO, changing from Axis2/C 1.0 to 1.3 is not a big issue. Because the
> client API and the service API did not change much. Devs please correct
> me if i am wrong.
That is correct. We added new methods, but did not change any of the
existing methods. So it should be safe. However, you may have to
recompile your clients/services.
Samisa...
---------------------------------------------------------------------
To unsubscribe, e-mail: axis-c-user-unsubscribe@ws.apache.org
For additional commands, e-mail: axis-c-user-help@ws.apache.org
RE: Core dumps in Axis libraries
Posted by Manjula Peiris <ma...@wso2.com>.
On Wed, 2008-03-05 at 18:09 +0530, Raghavendra SM wrote:
> Thanks Manjula,
>
> But thats not a viable option for us. We would still want to continue
> using axis2C-1.0 for some more time.
IMHO, changing from Axis2/C 1.0 to 1.3 is not a big issue. Because the
client API and the service API did not change much. Devs please correct
me if i am wrong. There are lot of new features, bug fixes and
performance improvements in 1.3 release.
-Manjula
>
>
> Are these dumps well known? And have they been specifically fixed in
> axis2c-1.3?
>
> Regards,
> ~raghav
>
>
> -----Original Message-----
> From: Manjula Peiris [mailto:manjula@wso2.com]
> Sent: Wednesday, March 05, 2008 6:05 PM
> To: Apache AXIS C User List
> Subject: Re: Core dumps in Axis libraries
>
> Hi raghav,
>
> Axis2/C 1.3 was recently released. There are lots of new features and
> bug fixes in that release. Please try that.
> http://ws.apache.org/axis2/c/download.cgi
>
> -Thanks,
> Manjula.
>
>
> On Wed, 2008-03-05 at 17:34 +0530, Raghavendra SM wrote:
> > All,
> >
> > For a particular scenario, our module core dumps in some axis library
> > functions consistently but with different back traces. I'm attaching
> the
> > dumps.log file that has as many as 4 different back-traces produced
> for
> > the same scenario.
> > Also, please find attached the log that has axis generated log for
> both
> > successful and erroneous times, axis.log.
> >
> > To give you all a brief background,
> > + we are using axis2c-1.0.0
> > + the core dump appears during a particular scenario only. Eg: while
> > trying to add a list of already existing users to my database. For
> each
> > user there is a SOAP request generated and the requests are
> sequential,
> > i.e the second request is sent only after the response for previous
> > request is received.
> > + if I increase the log level of my own module that uses axis2c, there
>
> > is no dump. On increasing the log level, my module writes quite a bit
> of
> > information to my log file.
> >
> > Since the back trace is pointing to the libraries(for which there is
> no
> > source code given), how can this be debugged and fixed?
> >
> > Any inputs will be of great help. Please reply.
> >
> >
> > Regards,
> > ~raghav
> >
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: axis-c-user-unsubscribe@ws.apache.org
> > For additional commands, e-mail: axis-c-user-help@ws.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: axis-c-user-unsubscribe@ws.apache.org
> For additional commands, e-mail: axis-c-user-help@ws.apache.org
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: axis-c-user-unsubscribe@ws.apache.org
> For additional commands, e-mail: axis-c-user-help@ws.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: axis-c-user-unsubscribe@ws.apache.org
For additional commands, e-mail: axis-c-user-help@ws.apache.org
RE: Core dumps in Axis libraries
Posted by Senaka Fernando <se...@wso2.com>.
Hi again Raghav,
Forgot to mention.
This is of course if you use mod_axis2. If not I don't think this might
affect you rather. Anyway the FastCGI issue may not be relevant at all. I
came across that while googling for what could cause a SIGTERM in
accept(2).
Also, is "http_server_main.cpp" something that you've written on your own?
If so, the issue might be in there sometimes.
Regards,
Senaka
> Hi Raghav,
>
> Are you connecting to a fast CGI server?
>
> If so, I think that [1] would help.
>
> [1] http://search.cpan.org/src/JURACH/FCGI-ProcManager-0.17/ProcManager.pm
>
> Regards,
> Senaka
>
>> As expected, when I put a delay of about 50 milliseconds between the
>> requests there is no dump.
>>
>> But we would still like to know the following,
>> + Isn't Axis capable of handling such huge bombardment of requests (of
>> the order of 1000's) one after another without any delay between the
>> requests? Has anyone tried such client before?
>> + Can we make out anything out of the valgrind report or the axis logs
>> that were attached?
>>
>> Regards,
>> ~raghav
>>
>>
>> -----Original Message-----
>> From: Senaka Fernando [mailto:senaka@wso2.com]
>> Sent: Thursday, March 06, 2008 4:22 PM
>> To: Apache AXIS C User List
>> Subject: RE: Core dumps in Axis libraries
>>
>> Hi Raghav,
>>
>> Can you please try this with --enable-guthilla=yes, and send the
>> dump.log,
>> and also the appropriate log_files generated by the server/client.
>>
>> There also can be issues with your "condition variable" being used in
>> multiple threads.
>>
>> It also seems that if you pause between your requests you might get this
>> solved. According to what you say, when you increase the log_level,
>> obviously you waste some time there which might be the solution rather.
>> You can use the AXIS2_SLEEP() and AXIS2_USLEEP() inside Axis2/C to pause
>> between requests. You can get an idea on how to use these functions by
>> taking a look at the Axis2/C user_guide/client samples.
>>
>> Regards,
>> Senaka
>>
>>> Thanks,
>>>
>>> As suggested, I tried using axis2c-1.3.0 both with and without
>>> --enable-guthilla=yes. But I still see different dumps consistently.
>>> Please find the attached file with back-traces.
>>>
>>> Are you using the same service client instance from multiple threads?
>>>>>> We have a axis thread pool with default number of threads and a set
>>> of our own threads(about 10, lets call them "database threads").
>>> + On receiving a HTTP soap request, axis thread invokes the
>> appropriate
>>> function that is bound to our logic.
>>> + Now, the axis thread requests a "database thread" to perform the
>>> appropriate query and waits on a condition variable.
>>> + our "db thread" performs the necessary query and signals on that
>>> condition variable so that axis thread shall take the control over.
>>> + axis thread will create the om output and dispatch as usual.
>>>
>>> My doubts:
>>> + does this design has any apparent drawback when there is a flurry of
>>> back-to-back soap requests?
>>> + Is there a need to increase the axis thread pool size? If yes, how
>> to
>>> do it?
>>> + As told before, the dump is seen during a particular scenario only.
>>> i.e: while trying to add a list of already existing users to my
>>> database. For each user there is a SOAP request generated and the
>>> requests are sequential, i.e the second request is sent only after the
>>> response for previous request is received.
>>> + if I increase the log level of my own module that uses axis2c, there
>>> is no dump. On increasing the log level, my module writes quite a bit
>> of
>>> information to my log file.
>>>
>>> Your help is appreciated.
>>>
>>> Thanks & Regards,
>>> ~raghav
>>>
>>>
>>> -----Original Message-----
>>> From: Samisa Abeysinghe [mailto:samisa@wso2.com]
>>> Sent: Wednesday, March 05, 2008 7:25 PM
>>> To: Apache AXIS C User List
>>> Subject: Re: Core dumps in Axis libraries
>>>
>>> Raghavendra SM wrote:
>>>> Thanks Manjula,
>>>>
>>>> But thats not a viable option for us. We would still want to continue
>>>> using axis2C-1.0 for some more time.
>>>>
>>>> Are these dumps well known?
>>>
>>> No.
>>>
>>> Are you using the same service client instance from multiple threads?
>>>
>>> Samisa...
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: axis-c-user-unsubscribe@ws.apache.org
>>> For additional commands, e-mail: axis-c-user-help@ws.apache.org
>>>
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: axis-c-user-unsubscribe@ws.apache.org
>>> For additional commands, e-mail: axis-c-user-help@ws.apache.org
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: axis-c-user-unsubscribe@ws.apache.org
>> For additional commands, e-mail: axis-c-user-help@ws.apache.org
>>
>>
>>
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: axis-c-user-unsubscribe@ws.apache.org
> For additional commands, e-mail: axis-c-user-help@ws.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: axis-c-user-unsubscribe@ws.apache.org
For additional commands, e-mail: axis-c-user-help@ws.apache.org
RE: Core dumps in Axis libraries
Posted by Senaka Fernando <se...@wso2.com>.
Hi Raghav,
Are you connecting to a fast CGI server?
If so, I think that [1] would help.
[1] http://search.cpan.org/src/JURACH/FCGI-ProcManager-0.17/ProcManager.pm
Regards,
Senaka
> As expected, when I put a delay of about 50 milliseconds between the
> requests there is no dump.
>
> But we would still like to know the following,
> + Isn't Axis capable of handling such huge bombardment of requests (of
> the order of 1000's) one after another without any delay between the
> requests? Has anyone tried such client before?
> + Can we make out anything out of the valgrind report or the axis logs
> that were attached?
>
> Regards,
> ~raghav
>
>
> -----Original Message-----
> From: Senaka Fernando [mailto:senaka@wso2.com]
> Sent: Thursday, March 06, 2008 4:22 PM
> To: Apache AXIS C User List
> Subject: RE: Core dumps in Axis libraries
>
> Hi Raghav,
>
> Can you please try this with --enable-guthilla=yes, and send the
> dump.log,
> and also the appropriate log_files generated by the server/client.
>
> There also can be issues with your "condition variable" being used in
> multiple threads.
>
> It also seems that if you pause between your requests you might get this
> solved. According to what you say, when you increase the log_level,
> obviously you waste some time there which might be the solution rather.
> You can use the AXIS2_SLEEP() and AXIS2_USLEEP() inside Axis2/C to pause
> between requests. You can get an idea on how to use these functions by
> taking a look at the Axis2/C user_guide/client samples.
>
> Regards,
> Senaka
>
>> Thanks,
>>
>> As suggested, I tried using axis2c-1.3.0 both with and without
>> --enable-guthilla=yes. But I still see different dumps consistently.
>> Please find the attached file with back-traces.
>>
>> Are you using the same service client instance from multiple threads?
>>>>> We have a axis thread pool with default number of threads and a set
>> of our own threads(about 10, lets call them "database threads").
>> + On receiving a HTTP soap request, axis thread invokes the
> appropriate
>> function that is bound to our logic.
>> + Now, the axis thread requests a "database thread" to perform the
>> appropriate query and waits on a condition variable.
>> + our "db thread" performs the necessary query and signals on that
>> condition variable so that axis thread shall take the control over.
>> + axis thread will create the om output and dispatch as usual.
>>
>> My doubts:
>> + does this design has any apparent drawback when there is a flurry of
>> back-to-back soap requests?
>> + Is there a need to increase the axis thread pool size? If yes, how
> to
>> do it?
>> + As told before, the dump is seen during a particular scenario only.
>> i.e: while trying to add a list of already existing users to my
>> database. For each user there is a SOAP request generated and the
>> requests are sequential, i.e the second request is sent only after the
>> response for previous request is received.
>> + if I increase the log level of my own module that uses axis2c, there
>> is no dump. On increasing the log level, my module writes quite a bit
> of
>> information to my log file.
>>
>> Your help is appreciated.
>>
>> Thanks & Regards,
>> ~raghav
>>
>>
>> -----Original Message-----
>> From: Samisa Abeysinghe [mailto:samisa@wso2.com]
>> Sent: Wednesday, March 05, 2008 7:25 PM
>> To: Apache AXIS C User List
>> Subject: Re: Core dumps in Axis libraries
>>
>> Raghavendra SM wrote:
>>> Thanks Manjula,
>>>
>>> But thats not a viable option for us. We would still want to continue
>>> using axis2C-1.0 for some more time.
>>>
>>> Are these dumps well known?
>>
>> No.
>>
>> Are you using the same service client instance from multiple threads?
>>
>> Samisa...
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: axis-c-user-unsubscribe@ws.apache.org
>> For additional commands, e-mail: axis-c-user-help@ws.apache.org
>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: axis-c-user-unsubscribe@ws.apache.org
>> For additional commands, e-mail: axis-c-user-help@ws.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: axis-c-user-unsubscribe@ws.apache.org
> For additional commands, e-mail: axis-c-user-help@ws.apache.org
>
>
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: axis-c-user-unsubscribe@ws.apache.org
For additional commands, e-mail: axis-c-user-help@ws.apache.org
Re: Solution to Core Dump Issue Raised in User List [was: Re: Core
dumps in Axis libraries]
Posted by Senaka Fernando <se...@wso2.com>.
> There are two sides to this problem. One is that, if it is failing for
> some scenario, we should be able to re-produce it. So without some code
> to re-produce the fault, we cannot come to a conclusion that it fails.
> If accepts fails, and it crashes, then there needs to be someuser
> scenario that makes this happen. If we cannot find a user scenario that
> makes accept to fail, then even though the bug is there, it will never
> be exposed.
>
> Anyway we can ask the user to try with your fixes for accept failure
> case and test to see if the problem persists.
Yes, that can be done. Actually as I said before, this is just a wild
guess that an issue in accept(2) could've created a segfault. Reason is
that there is no way to figure out what could be the reason from the
valgrind log.
Of course if this is a SIGTERM inside accept, he will have to use a signal
handler, there was an example in the Fast CGI Issue fix that I sent, and
return the appropriate error to the rest of the source following
accept(2). Then our code will pick it up correctly from there onwards with
this fix in place.
Also, adding a signal handler is a user-specific thing which we need not
attend to because, AFAIK, Axis2/C implementation will not cause a SIGTERM
inside accept(2) and that should be through the intervention of some
external source.
Regards,
Senaka
>
> Samisa...
>
> Senaka Fernando wrote:
>> Hi again Samisa,
>>
>> And, also, I don't have any similar database accessing logic etc.
>> Therefore, it is hard to conclude without seeing the actual code. Also,
>> the user seems to be using a modified version of "http_server_main.c",
>> it
>> is actually a cpp file if you go through that portion which I
>> highlighted
>> in his log. May be some issues due to not handling concurrency properly
>> could also lead him into trouble.
>>
>> However, what I can say is that there was a bug if accept(2) was to
>> fail,
>> according to the tests I did. And, according to literature available
>> online that can fail. Plus, we do have a mechanism of retrying, if it
>> does
>> fail which did not work due to the segfault.
>>
>> Regards,
>> Senaka
>>
>>
>>>> Were you able to re-create the problem without passing -1 to accept?
>>>>
>>> No I was not able to make accept fail without passing -1. Added a
>>> printf
>>> there to print any failure in accept and it didn't. However, ran into
>>> the
>>> segfault issue when accept actually failed, which I had to do manually.
>>>
>>> I did test with 1000 back-2-back mtom requests in 5 terminals, and also
>>> 1000 repeated calls to an echo client that sends 1000 back-2-back
>>> requests, and in 4 terminals.
>>>
>>> None of these cases failed. There were occasional hangs but still it
>>> didn't result in any failure on either client or server side.
>>>
>>> However, I didn't have any way to test multiple requests made to a
>>> single
>>> port from multiple machines. That may be a different scenario.
>>>
>>> Regards,
>>> Senaka
>>>
>>>
>>>> Samisa...
>>>>
>>>> Senaka Fernando wrote:
>>>>
>>>>>> Senaka Fernando wrote:
>>>>>>
>>>>>>
>>>>>>> Hi Samisa,
>>>>>>>
>>>>>>> I fixed two bugs regarding stream handling inside
>>>>>>> simple_http_svr_conn.c.
>>>>>>> This fixes an issue of this nature.
>>>>>>>
>>>>>>> Say, we have two server threads that try to serve requests offered
>>>>>>> to
>>>>>>> the
>>>>>>> axis2 server. Now, according to what is intended, one must serve
>>>>>>> and
>>>>>>> the
>>>>>>> other must wait until it can serve. The previous implementation
>>>>>>> caused
>>>>>>> this to fail with a segfault. But, with the fix, it will rather not
>>>>>>> crash
>>>>>>> but, loop until it gets a chance to serve the request.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>> Did you test before the fix and did it seg fault for you? Can you
>>>>>> please
>>>>>> explain the scenario under which it seg faulted? If it is a problem,
>>>>>> I
>>>>>> too should be able to re-create it.
>>>>>>
>>>>>> And how did you solve the problem? By adding a sleep?
>>>>>>
>>>>>>
>>>>> I made the accept(2) call fail. By passing a -1 to it. Once that
>>>>> failed,
>>>>> there were two places that segfaulted which I already fixed, few
>>>>> minutes
>>>>> ago. No, I didn't add a sleep(). I simply prevented the segfault by
>>>>> adding
>>>>> necessary is_null checks before calling methods. Then, the server
>>>>> just
>>>>> kept on spawning threads and freeing them as it can't serve the
>>>>> request.
>>>>> This is the expected behaviour in http_svr_thread.c I believe.
>>>>>
>>>>> In a real world scenario, a situation similar to a -1 passed to
>>>>> accept(2)
>>>>> will be temporal. So a respawned thread down the line should server
>>>>> the
>>>>> request without any problem.
>>>>>
>>>>> Passing -1 to accept was done by,
>>>>>
>>>>> Index: util/src/network_handler.c
>>>>> ===================================================================
>>>>> --- util/src/network_handler.c (revision 634578)
>>>>> +++ util/src/network_handler.c (working copy)
>>>>> @@ -221,7 +221,7 @@
>>>>> AXIS2_ENV_CHECK(env, AXIS2_CRITICAL_FAILURE);
>>>>>
>>>>> cli_len = sizeof(cli_addr);
>>>>> - cli_socket = accept(svr_socket, (struct sockaddr *) &cli_addr,
>>>>> &cli_len);
>>>>> + cli_socket = accept(-1, (struct sockaddr *) &cli_addr,
>>>>> &cli_len);
>>>>> if (cli_socket < 0)
>>>>> AXIS2_LOG_ERROR(env->log, AXIS2_LOG_SI,
>>>>> "[Axis2][network_handler] Socket accept \
>>>>> ---
>>>>>
>>>>> I'm not sure whether this is what really lead to the core dump, or
>>>>> was
>>>>> it
>>>>> something else.
>>>>>
>>>>> This was all that I could try on based on the less informative
>>>>> valgrind.log that was sent. :(...
>>>>>
>>>>> Also i did test 1000 back-to-back requests and they all were served
>>>>> without any issues.
>>>>>
>>>>> Regards,
>>>>> Senaka
>>>>>
>>>>>
>>>>>
>>>>>> Samisa...
>>>>>>
>>>>>>
>>>>>>
>>>>>>> Could this solve the core dump issue?
>>>>>>>
>>>>>>> Regards,
>>>>>>> Senaka
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>> Raghavendra SM wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>> As expected, when I put a delay of about 50 milliseconds between
>>>>>>>>> the
>>>>>>>>> requests there is no dump.
>>>>>>>>>
>>>>>>>>> But we would still like to know the following,
>>>>>>>>> + Isn't Axis capable of handling such huge bombardment of
>>>>>>>>> requests
>>>>>>>>> (of
>>>>>>>>> the order of 1000's) one after another without any delay between
>>>>>>>>> the
>>>>>>>>> requests? Has anyone tried such client before?
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>> Yes it is supposed to. We have done performance testing that
>>>>>>>> proved
>>>>>>>> that
>>>>>>>> Axis2/C can serve more than 300 rps.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>> + Can we make out anything out of the valgrind report or the axis
>>>>>>>>> logs
>>>>>>>>> that were attached?
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>> Unfortunately not. When valgrind runs, it makes the system slow,
>>>>>>>> that
>>>>>>>> prevents the system crashing in situations like that you are
>>>>>>>> experiencing. To solve this kind of a problem, it is a must that
>>>>>>>> you
>>>>>>>> provide us with some sample code that can be used to reproduce the
>>>>>>>> problem. Without having a way to re-produce the situation, there
>>>>>>>> is
>>>>>>>> no
>>>>>>>> way to fix this problem - it could well be a problem in Axis2/C as
>>>>>>>> well
>>>>>>>> as it could also be in the code that you use.
>>>>>>>>
>>>>>>>> Samisa...
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>> Regards,
>>>>>>>>> ~raghav
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> -----Original Message-----
>>>>>>>>> From: Senaka Fernando [mailto:senaka@wso2.com]
>>>>>>>>> Sent: Thursday, March 06, 2008 4:22 PM
>>>>>>>>> To: Apache AXIS C User List
>>>>>>>>> Subject: RE: Core dumps in Axis libraries
>>>>>>>>>
>>>>>>>>> Hi Raghav,
>>>>>>>>>
>>>>>>>>> Can you please try this with --enable-guthilla=yes, and send the
>>>>>>>>> dump.log,
>>>>>>>>> and also the appropriate log_files generated by the
>>>>>>>>> server/client.
>>>>>>>>>
>>>>>>>>> There also can be issues with your "condition variable" being
>>>>>>>>> used
>>>>>>>>> in
>>>>>>>>> multiple threads.
>>>>>>>>>
>>>>>>>>> It also seems that if you pause between your requests you might
>>>>>>>>> get
>>>>>>>>> this
>>>>>>>>> solved. According to what you say, when you increase the
>>>>>>>>> log_level,
>>>>>>>>> obviously you waste some time there which might be the solution
>>>>>>>>> rather.
>>>>>>>>> You can use the AXIS2_SLEEP() and AXIS2_USLEEP() inside Axis2/C
>>>>>>>>> to
>>>>>>>>> pause
>>>>>>>>> between requests. You can get an idea on how to use these
>>>>>>>>> functions
>>>>>>>>> by
>>>>>>>>> taking a look at the Axis2/C user_guide/client samples.
>>>>>>>>>
>>>>>>>>> Regards,
>>>>>>>>> Senaka
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>>
>>>>>>>>>> As suggested, I tried using axis2c-1.3.0 both with and without
>>>>>>>>>> --enable-guthilla=yes. But I still see different dumps
>>>>>>>>>> consistently.
>>>>>>>>>> Please find the attached file with back-traces.
>>>>>>>>>>
>>>>>>>>>> Are you using the same service client instance from multiple
>>>>>>>>>> threads?
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>>>> We have a axis thread pool with default number of threads and
>>>>>>>>>>>>> a
>>>>>>>>>>>>> set
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>> of our own threads(about 10, lets call them "database threads").
>>>>>>>>>> + On receiving a HTTP soap request, axis thread invokes the
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>> appropriate
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> function that is bound to our logic.
>>>>>>>>>> + Now, the axis thread requests a "database thread" to perform
>>>>>>>>>> the
>>>>>>>>>> appropriate query and waits on a condition variable.
>>>>>>>>>> + our "db thread" performs the necessary query and signals on
>>>>>>>>>> that
>>>>>>>>>> condition variable so that axis thread shall take the control
>>>>>>>>>> over.
>>>>>>>>>> + axis thread will create the om output and dispatch as usual.
>>>>>>>>>>
>>>>>>>>>> My doubts:
>>>>>>>>>> + does this design has any apparent drawback when there is a
>>>>>>>>>> flurry
>>>>>>>>>> of
>>>>>>>>>> back-to-back soap requests?
>>>>>>>>>> + Is there a need to increase the axis thread pool size? If yes,
>>>>>>>>>> how
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>> to
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> do it?
>>>>>>>>>> + As told before, the dump is seen during a particular scenario
>>>>>>>>>> only.
>>>>>>>>>> i.e: while trying to add a list of already existing users to my
>>>>>>>>>> database. For each user there is a SOAP request generated and
>>>>>>>>>> the
>>>>>>>>>> requests are sequential, i.e the second request is sent only
>>>>>>>>>> after
>>>>>>>>>> the
>>>>>>>>>> response for previous request is received.
>>>>>>>>>> + if I increase the log level of my own module that uses axis2c,
>>>>>>>>>> there
>>>>>>>>>> is no dump. On increasing the log level, my module writes quite
>>>>>>>>>> a
>>>>>>>>>> bit
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>> of
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> information to my log file.
>>>>>>>>>>
>>>>>>>>>> Your help is appreciated.
>>>>>>>>>>
>>>>>>>>>> Thanks & Regards,
>>>>>>>>>> ~raghav
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> -----Original Message-----
>>>>>>>>>> From: Samisa Abeysinghe [mailto:samisa@wso2.com]
>>>>>>>>>> Sent: Wednesday, March 05, 2008 7:25 PM
>>>>>>>>>> To: Apache AXIS C User List
>>>>>>>>>> Subject: Re: Core dumps in Axis libraries
>>>>>>>>>>
>>>>>>>>>> Raghavendra SM wrote:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> Thanks Manjula,
>>>>>>>>>>>
>>>>>>>>>>> But thats not a viable option for us. We would still want to
>>>>>>>>>>> continue
>>>>>>>>>>> using axis2C-1.0 for some more time.
>>>>>>>>>>>
>>>>>>>>>>> Are these dumps well known?
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>> No.
>>>>>>>>>>
>>>>>>>>>> Are you using the same service client instance from multiple
>>>>>>>>>> threads?
>>>>>>>>>>
>>>>>>>>>> Samisa...
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>> To unsubscribe, e-mail: axis-c-user-unsubscribe@ws.apache.org
>>>>>>>>>> For additional commands, e-mail: axis-c-user-help@ws.apache.org
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>> To unsubscribe, e-mail: axis-c-user-unsubscribe@ws.apache.org
>>>>>>>>>> For additional commands, e-mail: axis-c-user-help@ws.apache.org
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>> To unsubscribe, e-mail: axis-c-user-unsubscribe@ws.apache.org
>>>>>>>>> For additional commands, e-mail: axis-c-user-help@ws.apache.org
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>> To unsubscribe, e-mail: axis-c-user-unsubscribe@ws.apache.org
>>>>>>>>> For additional commands, e-mail: axis-c-user-help@ws.apache.org
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>> --
>>>>>>>> Samisa Abeysinghe
>>>>>>>> Software Architect; WSO2 Inc.
>>>>>>>>
>>>>>>>> http://www.wso2.com/ - "Oxygenating the Web Service Platform."
>>>>>>>>
>>>>>>>>
>>>>>>>> ---------------------------------------------------------------------
>>>>>>>> To unsubscribe, e-mail: axis-c-user-unsubscribe@ws.apache.org
>>>>>>>> For additional commands, e-mail: axis-c-user-help@ws.apache.org
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>> ---------------------------------------------------------------------
>>>>>>> To unsubscribe, e-mail: axis-c-dev-unsubscribe@ws.apache.org
>>>>>>> For additional commands, e-mail: axis-c-dev-help@ws.apache.org
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>> --
>>>>>> Samisa Abeysinghe
>>>>>> Software Architect; WSO2 Inc.
>>>>>>
>>>>>> http://www.wso2.com/ - "Oxygenating the Web Service Platform."
>>>>>>
>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: axis-c-dev-unsubscribe@ws.apache.org
>>>>>> For additional commands, e-mail: axis-c-dev-help@ws.apache.org
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: axis-c-dev-unsubscribe@ws.apache.org
>>>>> For additional commands, e-mail: axis-c-dev-help@ws.apache.org
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>> --
>>>> Samisa Abeysinghe
>>>> Software Architect; WSO2 Inc.
>>>>
>>>> http://www.wso2.com/ - "Oxygenating the Web Service Platform."
>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: axis-c-dev-unsubscribe@ws.apache.org
>>>> For additional commands, e-mail: axis-c-dev-help@ws.apache.org
>>>>
>>>>
>>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: axis-c-dev-unsubscribe@ws.apache.org
>>> For additional commands, e-mail: axis-c-dev-help@ws.apache.org
>>>
>>>
>>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: axis-c-dev-unsubscribe@ws.apache.org
>> For additional commands, e-mail: axis-c-dev-help@ws.apache.org
>>
>>
>>
>>
>
>
> --
> Samisa Abeysinghe
> Software Architect; WSO2 Inc.
>
> http://www.wso2.com/ - "Oxygenating the Web Service Platform."
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: axis-c-dev-unsubscribe@ws.apache.org
> For additional commands, e-mail: axis-c-dev-help@ws.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: axis-c-dev-unsubscribe@ws.apache.org
For additional commands, e-mail: axis-c-dev-help@ws.apache.org
Re: Solution to Core Dump Issue Raised in User List [was: Re: Core
dumps in Axis libraries]
Posted by Samisa Abeysinghe <sa...@wso2.com>.
There are two sides to this problem. One is that, if it is failing for
some scenario, we should be able to re-produce it. So without some code
to re-produce the fault, we cannot come to a conclusion that it fails.
If accepts fails, and it crashes, then there needs to be someuser
scenario that makes this happen. If we cannot find a user scenario that
makes accept to fail, then even though the bug is there, it will never
be exposed.
Anyway we can ask the user to try with your fixes for accept failure
case and test to see if the problem persists.
Samisa...
Senaka Fernando wrote:
> Hi again Samisa,
>
> And, also, I don't have any similar database accessing logic etc.
> Therefore, it is hard to conclude without seeing the actual code. Also,
> the user seems to be using a modified version of "http_server_main.c", it
> is actually a cpp file if you go through that portion which I highlighted
> in his log. May be some issues due to not handling concurrency properly
> could also lead him into trouble.
>
> However, what I can say is that there was a bug if accept(2) was to fail,
> according to the tests I did. And, according to literature available
> online that can fail. Plus, we do have a mechanism of retrying, if it does
> fail which did not work due to the segfault.
>
> Regards,
> Senaka
>
>
>>> Were you able to re-create the problem without passing -1 to accept?
>>>
>> No I was not able to make accept fail without passing -1. Added a printf
>> there to print any failure in accept and it didn't. However, ran into the
>> segfault issue when accept actually failed, which I had to do manually.
>>
>> I did test with 1000 back-2-back mtom requests in 5 terminals, and also
>> 1000 repeated calls to an echo client that sends 1000 back-2-back
>> requests, and in 4 terminals.
>>
>> None of these cases failed. There were occasional hangs but still it
>> didn't result in any failure on either client or server side.
>>
>> However, I didn't have any way to test multiple requests made to a single
>> port from multiple machines. That may be a different scenario.
>>
>> Regards,
>> Senaka
>>
>>
>>> Samisa...
>>>
>>> Senaka Fernando wrote:
>>>
>>>>> Senaka Fernando wrote:
>>>>>
>>>>>
>>>>>> Hi Samisa,
>>>>>>
>>>>>> I fixed two bugs regarding stream handling inside
>>>>>> simple_http_svr_conn.c.
>>>>>> This fixes an issue of this nature.
>>>>>>
>>>>>> Say, we have two server threads that try to serve requests offered to
>>>>>> the
>>>>>> axis2 server. Now, according to what is intended, one must serve and
>>>>>> the
>>>>>> other must wait until it can serve. The previous implementation
>>>>>> caused
>>>>>> this to fail with a segfault. But, with the fix, it will rather not
>>>>>> crash
>>>>>> but, loop until it gets a chance to serve the request.
>>>>>>
>>>>>>
>>>>>>
>>>>> Did you test before the fix and did it seg fault for you? Can you
>>>>> please
>>>>> explain the scenario under which it seg faulted? If it is a problem, I
>>>>> too should be able to re-create it.
>>>>>
>>>>> And how did you solve the problem? By adding a sleep?
>>>>>
>>>>>
>>>> I made the accept(2) call fail. By passing a -1 to it. Once that
>>>> failed,
>>>> there were two places that segfaulted which I already fixed, few
>>>> minutes
>>>> ago. No, I didn't add a sleep(). I simply prevented the segfault by
>>>> adding
>>>> necessary is_null checks before calling methods. Then, the server just
>>>> kept on spawning threads and freeing them as it can't serve the
>>>> request.
>>>> This is the expected behaviour in http_svr_thread.c I believe.
>>>>
>>>> In a real world scenario, a situation similar to a -1 passed to
>>>> accept(2)
>>>> will be temporal. So a respawned thread down the line should server the
>>>> request without any problem.
>>>>
>>>> Passing -1 to accept was done by,
>>>>
>>>> Index: util/src/network_handler.c
>>>> ===================================================================
>>>> --- util/src/network_handler.c (revision 634578)
>>>> +++ util/src/network_handler.c (working copy)
>>>> @@ -221,7 +221,7 @@
>>>> AXIS2_ENV_CHECK(env, AXIS2_CRITICAL_FAILURE);
>>>>
>>>> cli_len = sizeof(cli_addr);
>>>> - cli_socket = accept(svr_socket, (struct sockaddr *) &cli_addr,
>>>> &cli_len);
>>>> + cli_socket = accept(-1, (struct sockaddr *) &cli_addr, &cli_len);
>>>> if (cli_socket < 0)
>>>> AXIS2_LOG_ERROR(env->log, AXIS2_LOG_SI,
>>>> "[Axis2][network_handler] Socket accept \
>>>> ---
>>>>
>>>> I'm not sure whether this is what really lead to the core dump, or was
>>>> it
>>>> something else.
>>>>
>>>> This was all that I could try on based on the less informative
>>>> valgrind.log that was sent. :(...
>>>>
>>>> Also i did test 1000 back-to-back requests and they all were served
>>>> without any issues.
>>>>
>>>> Regards,
>>>> Senaka
>>>>
>>>>
>>>>
>>>>> Samisa...
>>>>>
>>>>>
>>>>>
>>>>>> Could this solve the core dump issue?
>>>>>>
>>>>>> Regards,
>>>>>> Senaka
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>> Raghavendra SM wrote:
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>> As expected, when I put a delay of about 50 milliseconds between
>>>>>>>> the
>>>>>>>> requests there is no dump.
>>>>>>>>
>>>>>>>> But we would still like to know the following,
>>>>>>>> + Isn't Axis capable of handling such huge bombardment of requests
>>>>>>>> (of
>>>>>>>> the order of 1000's) one after another without any delay between
>>>>>>>> the
>>>>>>>> requests? Has anyone tried such client before?
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>> Yes it is supposed to. We have done performance testing that proved
>>>>>>> that
>>>>>>> Axis2/C can serve more than 300 rps.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>> + Can we make out anything out of the valgrind report or the axis
>>>>>>>> logs
>>>>>>>> that were attached?
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>> Unfortunately not. When valgrind runs, it makes the system slow,
>>>>>>> that
>>>>>>> prevents the system crashing in situations like that you are
>>>>>>> experiencing. To solve this kind of a problem, it is a must that you
>>>>>>> provide us with some sample code that can be used to reproduce the
>>>>>>> problem. Without having a way to re-produce the situation, there is
>>>>>>> no
>>>>>>> way to fix this problem - it could well be a problem in Axis2/C as
>>>>>>> well
>>>>>>> as it could also be in the code that you use.
>>>>>>>
>>>>>>> Samisa...
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>> Regards,
>>>>>>>> ~raghav
>>>>>>>>
>>>>>>>>
>>>>>>>> -----Original Message-----
>>>>>>>> From: Senaka Fernando [mailto:senaka@wso2.com]
>>>>>>>> Sent: Thursday, March 06, 2008 4:22 PM
>>>>>>>> To: Apache AXIS C User List
>>>>>>>> Subject: RE: Core dumps in Axis libraries
>>>>>>>>
>>>>>>>> Hi Raghav,
>>>>>>>>
>>>>>>>> Can you please try this with --enable-guthilla=yes, and send the
>>>>>>>> dump.log,
>>>>>>>> and also the appropriate log_files generated by the server/client.
>>>>>>>>
>>>>>>>> There also can be issues with your "condition variable" being used
>>>>>>>> in
>>>>>>>> multiple threads.
>>>>>>>>
>>>>>>>> It also seems that if you pause between your requests you might get
>>>>>>>> this
>>>>>>>> solved. According to what you say, when you increase the log_level,
>>>>>>>> obviously you waste some time there which might be the solution
>>>>>>>> rather.
>>>>>>>> You can use the AXIS2_SLEEP() and AXIS2_USLEEP() inside Axis2/C to
>>>>>>>> pause
>>>>>>>> between requests. You can get an idea on how to use these functions
>>>>>>>> by
>>>>>>>> taking a look at the Axis2/C user_guide/client samples.
>>>>>>>>
>>>>>>>> Regards,
>>>>>>>> Senaka
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>>
>>>>>>>>> As suggested, I tried using axis2c-1.3.0 both with and without
>>>>>>>>> --enable-guthilla=yes. But I still see different dumps
>>>>>>>>> consistently.
>>>>>>>>> Please find the attached file with back-traces.
>>>>>>>>>
>>>>>>>>> Are you using the same service client instance from multiple
>>>>>>>>> threads?
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>>>> We have a axis thread pool with default number of threads and a
>>>>>>>>>>>> set
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>> of our own threads(about 10, lets call them "database threads").
>>>>>>>>> + On receiving a HTTP soap request, axis thread invokes the
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>> appropriate
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>> function that is bound to our logic.
>>>>>>>>> + Now, the axis thread requests a "database thread" to perform the
>>>>>>>>> appropriate query and waits on a condition variable.
>>>>>>>>> + our "db thread" performs the necessary query and signals on that
>>>>>>>>> condition variable so that axis thread shall take the control
>>>>>>>>> over.
>>>>>>>>> + axis thread will create the om output and dispatch as usual.
>>>>>>>>>
>>>>>>>>> My doubts:
>>>>>>>>> + does this design has any apparent drawback when there is a
>>>>>>>>> flurry
>>>>>>>>> of
>>>>>>>>> back-to-back soap requests?
>>>>>>>>> + Is there a need to increase the axis thread pool size? If yes,
>>>>>>>>> how
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>> to
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>> do it?
>>>>>>>>> + As told before, the dump is seen during a particular scenario
>>>>>>>>> only.
>>>>>>>>> i.e: while trying to add a list of already existing users to my
>>>>>>>>> database. For each user there is a SOAP request generated and the
>>>>>>>>> requests are sequential, i.e the second request is sent only after
>>>>>>>>> the
>>>>>>>>> response for previous request is received.
>>>>>>>>> + if I increase the log level of my own module that uses axis2c,
>>>>>>>>> there
>>>>>>>>> is no dump. On increasing the log level, my module writes quite a
>>>>>>>>> bit
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>> of
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>> information to my log file.
>>>>>>>>>
>>>>>>>>> Your help is appreciated.
>>>>>>>>>
>>>>>>>>> Thanks & Regards,
>>>>>>>>> ~raghav
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> -----Original Message-----
>>>>>>>>> From: Samisa Abeysinghe [mailto:samisa@wso2.com]
>>>>>>>>> Sent: Wednesday, March 05, 2008 7:25 PM
>>>>>>>>> To: Apache AXIS C User List
>>>>>>>>> Subject: Re: Core dumps in Axis libraries
>>>>>>>>>
>>>>>>>>> Raghavendra SM wrote:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> Thanks Manjula,
>>>>>>>>>>
>>>>>>>>>> But thats not a viable option for us. We would still want to
>>>>>>>>>> continue
>>>>>>>>>> using axis2C-1.0 for some more time.
>>>>>>>>>>
>>>>>>>>>> Are these dumps well known?
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>> No.
>>>>>>>>>
>>>>>>>>> Are you using the same service client instance from multiple
>>>>>>>>> threads?
>>>>>>>>>
>>>>>>>>> Samisa...
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>> To unsubscribe, e-mail: axis-c-user-unsubscribe@ws.apache.org
>>>>>>>>> For additional commands, e-mail: axis-c-user-help@ws.apache.org
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>> To unsubscribe, e-mail: axis-c-user-unsubscribe@ws.apache.org
>>>>>>>>> For additional commands, e-mail: axis-c-user-help@ws.apache.org
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>> ---------------------------------------------------------------------
>>>>>>>> To unsubscribe, e-mail: axis-c-user-unsubscribe@ws.apache.org
>>>>>>>> For additional commands, e-mail: axis-c-user-help@ws.apache.org
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> ---------------------------------------------------------------------
>>>>>>>> To unsubscribe, e-mail: axis-c-user-unsubscribe@ws.apache.org
>>>>>>>> For additional commands, e-mail: axis-c-user-help@ws.apache.org
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>> --
>>>>>>> Samisa Abeysinghe
>>>>>>> Software Architect; WSO2 Inc.
>>>>>>>
>>>>>>> http://www.wso2.com/ - "Oxygenating the Web Service Platform."
>>>>>>>
>>>>>>>
>>>>>>> ---------------------------------------------------------------------
>>>>>>> To unsubscribe, e-mail: axis-c-user-unsubscribe@ws.apache.org
>>>>>>> For additional commands, e-mail: axis-c-user-help@ws.apache.org
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: axis-c-dev-unsubscribe@ws.apache.org
>>>>>> For additional commands, e-mail: axis-c-dev-help@ws.apache.org
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>> --
>>>>> Samisa Abeysinghe
>>>>> Software Architect; WSO2 Inc.
>>>>>
>>>>> http://www.wso2.com/ - "Oxygenating the Web Service Platform."
>>>>>
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: axis-c-dev-unsubscribe@ws.apache.org
>>>>> For additional commands, e-mail: axis-c-dev-help@ws.apache.org
>>>>>
>>>>>
>>>>>
>>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: axis-c-dev-unsubscribe@ws.apache.org
>>>> For additional commands, e-mail: axis-c-dev-help@ws.apache.org
>>>>
>>>>
>>>>
>>>>
>>>>
>>> --
>>> Samisa Abeysinghe
>>> Software Architect; WSO2 Inc.
>>>
>>> http://www.wso2.com/ - "Oxygenating the Web Service Platform."
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: axis-c-dev-unsubscribe@ws.apache.org
>>> For additional commands, e-mail: axis-c-dev-help@ws.apache.org
>>>
>>>
>>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: axis-c-dev-unsubscribe@ws.apache.org
>> For additional commands, e-mail: axis-c-dev-help@ws.apache.org
>>
>>
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: axis-c-dev-unsubscribe@ws.apache.org
> For additional commands, e-mail: axis-c-dev-help@ws.apache.org
>
>
>
>
--
Samisa Abeysinghe
Software Architect; WSO2 Inc.
http://www.wso2.com/ - "Oxygenating the Web Service Platform."
---------------------------------------------------------------------
To unsubscribe, e-mail: axis-c-dev-unsubscribe@ws.apache.org
For additional commands, e-mail: axis-c-dev-help@ws.apache.org
Re: Solution to Core Dump Issue Raised in User List [was: Re: Core
dumps in Axis libraries]
Posted by Senaka Fernando <se...@wso2.com>.
Hi again Samisa,
And, also, I don't have any similar database accessing logic etc.
Therefore, it is hard to conclude without seeing the actual code. Also,
the user seems to be using a modified version of "http_server_main.c", it
is actually a cpp file if you go through that portion which I highlighted
in his log. May be some issues due to not handling concurrency properly
could also lead him into trouble.
However, what I can say is that there was a bug if accept(2) was to fail,
according to the tests I did. And, according to literature available
online that can fail. Plus, we do have a mechanism of retrying, if it does
fail which did not work due to the segfault.
Regards,
Senaka
>> Were you able to re-create the problem without passing -1 to accept?
>
> No I was not able to make accept fail without passing -1. Added a printf
> there to print any failure in accept and it didn't. However, ran into the
> segfault issue when accept actually failed, which I had to do manually.
>
> I did test with 1000 back-2-back mtom requests in 5 terminals, and also
> 1000 repeated calls to an echo client that sends 1000 back-2-back
> requests, and in 4 terminals.
>
> None of these cases failed. There were occasional hangs but still it
> didn't result in any failure on either client or server side.
>
> However, I didn't have any way to test multiple requests made to a single
> port from multiple machines. That may be a different scenario.
>
> Regards,
> Senaka
>
>>
>> Samisa...
>>
>> Senaka Fernando wrote:
>>>> Senaka Fernando wrote:
>>>>
>>>>> Hi Samisa,
>>>>>
>>>>> I fixed two bugs regarding stream handling inside
>>>>> simple_http_svr_conn.c.
>>>>> This fixes an issue of this nature.
>>>>>
>>>>> Say, we have two server threads that try to serve requests offered to
>>>>> the
>>>>> axis2 server. Now, according to what is intended, one must serve and
>>>>> the
>>>>> other must wait until it can serve. The previous implementation
>>>>> caused
>>>>> this to fail with a segfault. But, with the fix, it will rather not
>>>>> crash
>>>>> but, loop until it gets a chance to serve the request.
>>>>>
>>>>>
>>>> Did you test before the fix and did it seg fault for you? Can you
>>>> please
>>>> explain the scenario under which it seg faulted? If it is a problem, I
>>>> too should be able to re-create it.
>>>>
>>>> And how did you solve the problem? By adding a sleep?
>>>>
>>>
>>> I made the accept(2) call fail. By passing a -1 to it. Once that
>>> failed,
>>> there were two places that segfaulted which I already fixed, few
>>> minutes
>>> ago. No, I didn't add a sleep(). I simply prevented the segfault by
>>> adding
>>> necessary is_null checks before calling methods. Then, the server just
>>> kept on spawning threads and freeing them as it can't serve the
>>> request.
>>> This is the expected behaviour in http_svr_thread.c I believe.
>>>
>>> In a real world scenario, a situation similar to a -1 passed to
>>> accept(2)
>>> will be temporal. So a respawned thread down the line should server the
>>> request without any problem.
>>>
>>> Passing -1 to accept was done by,
>>>
>>> Index: util/src/network_handler.c
>>> ===================================================================
>>> --- util/src/network_handler.c (revision 634578)
>>> +++ util/src/network_handler.c (working copy)
>>> @@ -221,7 +221,7 @@
>>> AXIS2_ENV_CHECK(env, AXIS2_CRITICAL_FAILURE);
>>>
>>> cli_len = sizeof(cli_addr);
>>> - cli_socket = accept(svr_socket, (struct sockaddr *) &cli_addr,
>>> &cli_len);
>>> + cli_socket = accept(-1, (struct sockaddr *) &cli_addr, &cli_len);
>>> if (cli_socket < 0)
>>> AXIS2_LOG_ERROR(env->log, AXIS2_LOG_SI,
>>> "[Axis2][network_handler] Socket accept \
>>> ---
>>>
>>> I'm not sure whether this is what really lead to the core dump, or was
>>> it
>>> something else.
>>>
>>> This was all that I could try on based on the less informative
>>> valgrind.log that was sent. :(...
>>>
>>> Also i did test 1000 back-to-back requests and they all were served
>>> without any issues.
>>>
>>> Regards,
>>> Senaka
>>>
>>>
>>>> Samisa...
>>>>
>>>>
>>>>> Could this solve the core dump issue?
>>>>>
>>>>> Regards,
>>>>> Senaka
>>>>>
>>>>>
>>>>>
>>>>>> Raghavendra SM wrote:
>>>>>>
>>>>>>
>>>>>>> As expected, when I put a delay of about 50 milliseconds between
>>>>>>> the
>>>>>>> requests there is no dump.
>>>>>>>
>>>>>>> But we would still like to know the following,
>>>>>>> + Isn't Axis capable of handling such huge bombardment of requests
>>>>>>> (of
>>>>>>> the order of 1000's) one after another without any delay between
>>>>>>> the
>>>>>>> requests? Has anyone tried such client before?
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>> Yes it is supposed to. We have done performance testing that proved
>>>>>> that
>>>>>> Axis2/C can serve more than 300 rps.
>>>>>>
>>>>>>
>>>>>>
>>>>>>> + Can we make out anything out of the valgrind report or the axis
>>>>>>> logs
>>>>>>> that were attached?
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>> Unfortunately not. When valgrind runs, it makes the system slow,
>>>>>> that
>>>>>> prevents the system crashing in situations like that you are
>>>>>> experiencing. To solve this kind of a problem, it is a must that you
>>>>>> provide us with some sample code that can be used to reproduce the
>>>>>> problem. Without having a way to re-produce the situation, there is
>>>>>> no
>>>>>> way to fix this problem - it could well be a problem in Axis2/C as
>>>>>> well
>>>>>> as it could also be in the code that you use.
>>>>>>
>>>>>> Samisa...
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>> Regards,
>>>>>>> ~raghav
>>>>>>>
>>>>>>>
>>>>>>> -----Original Message-----
>>>>>>> From: Senaka Fernando [mailto:senaka@wso2.com]
>>>>>>> Sent: Thursday, March 06, 2008 4:22 PM
>>>>>>> To: Apache AXIS C User List
>>>>>>> Subject: RE: Core dumps in Axis libraries
>>>>>>>
>>>>>>> Hi Raghav,
>>>>>>>
>>>>>>> Can you please try this with --enable-guthilla=yes, and send the
>>>>>>> dump.log,
>>>>>>> and also the appropriate log_files generated by the server/client.
>>>>>>>
>>>>>>> There also can be issues with your "condition variable" being used
>>>>>>> in
>>>>>>> multiple threads.
>>>>>>>
>>>>>>> It also seems that if you pause between your requests you might get
>>>>>>> this
>>>>>>> solved. According to what you say, when you increase the log_level,
>>>>>>> obviously you waste some time there which might be the solution
>>>>>>> rather.
>>>>>>> You can use the AXIS2_SLEEP() and AXIS2_USLEEP() inside Axis2/C to
>>>>>>> pause
>>>>>>> between requests. You can get an idea on how to use these functions
>>>>>>> by
>>>>>>> taking a look at the Axis2/C user_guide/client samples.
>>>>>>>
>>>>>>> Regards,
>>>>>>> Senaka
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>> Thanks,
>>>>>>>>
>>>>>>>> As suggested, I tried using axis2c-1.3.0 both with and without
>>>>>>>> --enable-guthilla=yes. But I still see different dumps
>>>>>>>> consistently.
>>>>>>>> Please find the attached file with back-traces.
>>>>>>>>
>>>>>>>> Are you using the same service client instance from multiple
>>>>>>>> threads?
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>>>> We have a axis thread pool with default number of threads and a
>>>>>>>>>>> set
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>> of our own threads(about 10, lets call them "database threads").
>>>>>>>> + On receiving a HTTP soap request, axis thread invokes the
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>> appropriate
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>> function that is bound to our logic.
>>>>>>>> + Now, the axis thread requests a "database thread" to perform the
>>>>>>>> appropriate query and waits on a condition variable.
>>>>>>>> + our "db thread" performs the necessary query and signals on that
>>>>>>>> condition variable so that axis thread shall take the control
>>>>>>>> over.
>>>>>>>> + axis thread will create the om output and dispatch as usual.
>>>>>>>>
>>>>>>>> My doubts:
>>>>>>>> + does this design has any apparent drawback when there is a
>>>>>>>> flurry
>>>>>>>> of
>>>>>>>> back-to-back soap requests?
>>>>>>>> + Is there a need to increase the axis thread pool size? If yes,
>>>>>>>> how
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>> to
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>> do it?
>>>>>>>> + As told before, the dump is seen during a particular scenario
>>>>>>>> only.
>>>>>>>> i.e: while trying to add a list of already existing users to my
>>>>>>>> database. For each user there is a SOAP request generated and the
>>>>>>>> requests are sequential, i.e the second request is sent only after
>>>>>>>> the
>>>>>>>> response for previous request is received.
>>>>>>>> + if I increase the log level of my own module that uses axis2c,
>>>>>>>> there
>>>>>>>> is no dump. On increasing the log level, my module writes quite a
>>>>>>>> bit
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>> of
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>> information to my log file.
>>>>>>>>
>>>>>>>> Your help is appreciated.
>>>>>>>>
>>>>>>>> Thanks & Regards,
>>>>>>>> ~raghav
>>>>>>>>
>>>>>>>>
>>>>>>>> -----Original Message-----
>>>>>>>> From: Samisa Abeysinghe [mailto:samisa@wso2.com]
>>>>>>>> Sent: Wednesday, March 05, 2008 7:25 PM
>>>>>>>> To: Apache AXIS C User List
>>>>>>>> Subject: Re: Core dumps in Axis libraries
>>>>>>>>
>>>>>>>> Raghavendra SM wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>> Thanks Manjula,
>>>>>>>>>
>>>>>>>>> But thats not a viable option for us. We would still want to
>>>>>>>>> continue
>>>>>>>>> using axis2C-1.0 for some more time.
>>>>>>>>>
>>>>>>>>> Are these dumps well known?
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>> No.
>>>>>>>>
>>>>>>>> Are you using the same service client instance from multiple
>>>>>>>> threads?
>>>>>>>>
>>>>>>>> Samisa...
>>>>>>>>
>>>>>>>>
>>>>>>>> ---------------------------------------------------------------------
>>>>>>>> To unsubscribe, e-mail: axis-c-user-unsubscribe@ws.apache.org
>>>>>>>> For additional commands, e-mail: axis-c-user-help@ws.apache.org
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> ---------------------------------------------------------------------
>>>>>>>> To unsubscribe, e-mail: axis-c-user-unsubscribe@ws.apache.org
>>>>>>>> For additional commands, e-mail: axis-c-user-help@ws.apache.org
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>> ---------------------------------------------------------------------
>>>>>>> To unsubscribe, e-mail: axis-c-user-unsubscribe@ws.apache.org
>>>>>>> For additional commands, e-mail: axis-c-user-help@ws.apache.org
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> ---------------------------------------------------------------------
>>>>>>> To unsubscribe, e-mail: axis-c-user-unsubscribe@ws.apache.org
>>>>>>> For additional commands, e-mail: axis-c-user-help@ws.apache.org
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>> --
>>>>>> Samisa Abeysinghe
>>>>>> Software Architect; WSO2 Inc.
>>>>>>
>>>>>> http://www.wso2.com/ - "Oxygenating the Web Service Platform."
>>>>>>
>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: axis-c-user-unsubscribe@ws.apache.org
>>>>>> For additional commands, e-mail: axis-c-user-help@ws.apache.org
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: axis-c-dev-unsubscribe@ws.apache.org
>>>>> For additional commands, e-mail: axis-c-dev-help@ws.apache.org
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>> --
>>>> Samisa Abeysinghe
>>>> Software Architect; WSO2 Inc.
>>>>
>>>> http://www.wso2.com/ - "Oxygenating the Web Service Platform."
>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: axis-c-dev-unsubscribe@ws.apache.org
>>>> For additional commands, e-mail: axis-c-dev-help@ws.apache.org
>>>>
>>>>
>>>>
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: axis-c-dev-unsubscribe@ws.apache.org
>>> For additional commands, e-mail: axis-c-dev-help@ws.apache.org
>>>
>>>
>>>
>>>
>>
>>
>> --
>> Samisa Abeysinghe
>> Software Architect; WSO2 Inc.
>>
>> http://www.wso2.com/ - "Oxygenating the Web Service Platform."
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: axis-c-dev-unsubscribe@ws.apache.org
>> For additional commands, e-mail: axis-c-dev-help@ws.apache.org
>>
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: axis-c-dev-unsubscribe@ws.apache.org
> For additional commands, e-mail: axis-c-dev-help@ws.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: axis-c-dev-unsubscribe@ws.apache.org
For additional commands, e-mail: axis-c-dev-help@ws.apache.org
Re: Solution to Core Dump Issue Raised in User List [was: Re: Core
dumps in Axis libraries]
Posted by Senaka Fernando <se...@wso2.com>.
> Were you able to re-create the problem without passing -1 to accept?
No I was not able to make accept fail without passing -1. Added a printf
there to print any failure in accept and it didn't. However, ran into the
segfault issue when accept actually failed, which I had to do manually.
I did test with 1000 back-2-back mtom requests in 5 terminals, and also
1000 repeated calls to an echo client that sends 1000 back-2-back
requests, and in 4 terminals.
None of these cases failed. There were occasional hangs but still it
didn't result in any failure on either client or server side.
However, I didn't have any way to test multiple requests made to a single
port from multiple machines. That may be a different scenario.
Regards,
Senaka
>
> Samisa...
>
> Senaka Fernando wrote:
>>> Senaka Fernando wrote:
>>>
>>>> Hi Samisa,
>>>>
>>>> I fixed two bugs regarding stream handling inside
>>>> simple_http_svr_conn.c.
>>>> This fixes an issue of this nature.
>>>>
>>>> Say, we have two server threads that try to serve requests offered to
>>>> the
>>>> axis2 server. Now, according to what is intended, one must serve and
>>>> the
>>>> other must wait until it can serve. The previous implementation caused
>>>> this to fail with a segfault. But, with the fix, it will rather not
>>>> crash
>>>> but, loop until it gets a chance to serve the request.
>>>>
>>>>
>>> Did you test before the fix and did it seg fault for you? Can you
>>> please
>>> explain the scenario under which it seg faulted? If it is a problem, I
>>> too should be able to re-create it.
>>>
>>> And how did you solve the problem? By adding a sleep?
>>>
>>
>> I made the accept(2) call fail. By passing a -1 to it. Once that failed,
>> there were two places that segfaulted which I already fixed, few minutes
>> ago. No, I didn't add a sleep(). I simply prevented the segfault by
>> adding
>> necessary is_null checks before calling methods. Then, the server just
>> kept on spawning threads and freeing them as it can't serve the request.
>> This is the expected behaviour in http_svr_thread.c I believe.
>>
>> In a real world scenario, a situation similar to a -1 passed to
>> accept(2)
>> will be temporal. So a respawned thread down the line should server the
>> request without any problem.
>>
>> Passing -1 to accept was done by,
>>
>> Index: util/src/network_handler.c
>> ===================================================================
>> --- util/src/network_handler.c (revision 634578)
>> +++ util/src/network_handler.c (working copy)
>> @@ -221,7 +221,7 @@
>> AXIS2_ENV_CHECK(env, AXIS2_CRITICAL_FAILURE);
>>
>> cli_len = sizeof(cli_addr);
>> - cli_socket = accept(svr_socket, (struct sockaddr *) &cli_addr,
>> &cli_len);
>> + cli_socket = accept(-1, (struct sockaddr *) &cli_addr, &cli_len);
>> if (cli_socket < 0)
>> AXIS2_LOG_ERROR(env->log, AXIS2_LOG_SI,
>> "[Axis2][network_handler] Socket accept \
>> ---
>>
>> I'm not sure whether this is what really lead to the core dump, or was
>> it
>> something else.
>>
>> This was all that I could try on based on the less informative
>> valgrind.log that was sent. :(...
>>
>> Also i did test 1000 back-to-back requests and they all were served
>> without any issues.
>>
>> Regards,
>> Senaka
>>
>>
>>> Samisa...
>>>
>>>
>>>> Could this solve the core dump issue?
>>>>
>>>> Regards,
>>>> Senaka
>>>>
>>>>
>>>>
>>>>> Raghavendra SM wrote:
>>>>>
>>>>>
>>>>>> As expected, when I put a delay of about 50 milliseconds between the
>>>>>> requests there is no dump.
>>>>>>
>>>>>> But we would still like to know the following,
>>>>>> + Isn't Axis capable of handling such huge bombardment of requests
>>>>>> (of
>>>>>> the order of 1000's) one after another without any delay between the
>>>>>> requests? Has anyone tried such client before?
>>>>>>
>>>>>>
>>>>>>
>>>>> Yes it is supposed to. We have done performance testing that proved
>>>>> that
>>>>> Axis2/C can serve more than 300 rps.
>>>>>
>>>>>
>>>>>
>>>>>> + Can we make out anything out of the valgrind report or the axis
>>>>>> logs
>>>>>> that were attached?
>>>>>>
>>>>>>
>>>>>>
>>>>> Unfortunately not. When valgrind runs, it makes the system slow, that
>>>>> prevents the system crashing in situations like that you are
>>>>> experiencing. To solve this kind of a problem, it is a must that you
>>>>> provide us with some sample code that can be used to reproduce the
>>>>> problem. Without having a way to re-produce the situation, there is
>>>>> no
>>>>> way to fix this problem - it could well be a problem in Axis2/C as
>>>>> well
>>>>> as it could also be in the code that you use.
>>>>>
>>>>> Samisa...
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>> Regards,
>>>>>> ~raghav
>>>>>>
>>>>>>
>>>>>> -----Original Message-----
>>>>>> From: Senaka Fernando [mailto:senaka@wso2.com]
>>>>>> Sent: Thursday, March 06, 2008 4:22 PM
>>>>>> To: Apache AXIS C User List
>>>>>> Subject: RE: Core dumps in Axis libraries
>>>>>>
>>>>>> Hi Raghav,
>>>>>>
>>>>>> Can you please try this with --enable-guthilla=yes, and send the
>>>>>> dump.log,
>>>>>> and also the appropriate log_files generated by the server/client.
>>>>>>
>>>>>> There also can be issues with your "condition variable" being used
>>>>>> in
>>>>>> multiple threads.
>>>>>>
>>>>>> It also seems that if you pause between your requests you might get
>>>>>> this
>>>>>> solved. According to what you say, when you increase the log_level,
>>>>>> obviously you waste some time there which might be the solution
>>>>>> rather.
>>>>>> You can use the AXIS2_SLEEP() and AXIS2_USLEEP() inside Axis2/C to
>>>>>> pause
>>>>>> between requests. You can get an idea on how to use these functions
>>>>>> by
>>>>>> taking a look at the Axis2/C user_guide/client samples.
>>>>>>
>>>>>> Regards,
>>>>>> Senaka
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>> Thanks,
>>>>>>>
>>>>>>> As suggested, I tried using axis2c-1.3.0 both with and without
>>>>>>> --enable-guthilla=yes. But I still see different dumps
>>>>>>> consistently.
>>>>>>> Please find the attached file with back-traces.
>>>>>>>
>>>>>>> Are you using the same service client instance from multiple
>>>>>>> threads?
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>>>> We have a axis thread pool with default number of threads and a
>>>>>>>>>> set
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>> of our own threads(about 10, lets call them "database threads").
>>>>>>> + On receiving a HTTP soap request, axis thread invokes the
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>> appropriate
>>>>>>
>>>>>>
>>>>>>
>>>>>>> function that is bound to our logic.
>>>>>>> + Now, the axis thread requests a "database thread" to perform the
>>>>>>> appropriate query and waits on a condition variable.
>>>>>>> + our "db thread" performs the necessary query and signals on that
>>>>>>> condition variable so that axis thread shall take the control over.
>>>>>>> + axis thread will create the om output and dispatch as usual.
>>>>>>>
>>>>>>> My doubts:
>>>>>>> + does this design has any apparent drawback when there is a flurry
>>>>>>> of
>>>>>>> back-to-back soap requests?
>>>>>>> + Is there a need to increase the axis thread pool size? If yes,
>>>>>>> how
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>> to
>>>>>>
>>>>>>
>>>>>>
>>>>>>> do it?
>>>>>>> + As told before, the dump is seen during a particular scenario
>>>>>>> only.
>>>>>>> i.e: while trying to add a list of already existing users to my
>>>>>>> database. For each user there is a SOAP request generated and the
>>>>>>> requests are sequential, i.e the second request is sent only after
>>>>>>> the
>>>>>>> response for previous request is received.
>>>>>>> + if I increase the log level of my own module that uses axis2c,
>>>>>>> there
>>>>>>> is no dump. On increasing the log level, my module writes quite a
>>>>>>> bit
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>> of
>>>>>>
>>>>>>
>>>>>>
>>>>>>> information to my log file.
>>>>>>>
>>>>>>> Your help is appreciated.
>>>>>>>
>>>>>>> Thanks & Regards,
>>>>>>> ~raghav
>>>>>>>
>>>>>>>
>>>>>>> -----Original Message-----
>>>>>>> From: Samisa Abeysinghe [mailto:samisa@wso2.com]
>>>>>>> Sent: Wednesday, March 05, 2008 7:25 PM
>>>>>>> To: Apache AXIS C User List
>>>>>>> Subject: Re: Core dumps in Axis libraries
>>>>>>>
>>>>>>> Raghavendra SM wrote:
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>> Thanks Manjula,
>>>>>>>>
>>>>>>>> But thats not a viable option for us. We would still want to
>>>>>>>> continue
>>>>>>>> using axis2C-1.0 for some more time.
>>>>>>>>
>>>>>>>> Are these dumps well known?
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>> No.
>>>>>>>
>>>>>>> Are you using the same service client instance from multiple
>>>>>>> threads?
>>>>>>>
>>>>>>> Samisa...
>>>>>>>
>>>>>>>
>>>>>>> ---------------------------------------------------------------------
>>>>>>> To unsubscribe, e-mail: axis-c-user-unsubscribe@ws.apache.org
>>>>>>> For additional commands, e-mail: axis-c-user-help@ws.apache.org
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> ---------------------------------------------------------------------
>>>>>>> To unsubscribe, e-mail: axis-c-user-unsubscribe@ws.apache.org
>>>>>>> For additional commands, e-mail: axis-c-user-help@ws.apache.org
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: axis-c-user-unsubscribe@ws.apache.org
>>>>>> For additional commands, e-mail: axis-c-user-help@ws.apache.org
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: axis-c-user-unsubscribe@ws.apache.org
>>>>>> For additional commands, e-mail: axis-c-user-help@ws.apache.org
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>> --
>>>>> Samisa Abeysinghe
>>>>> Software Architect; WSO2 Inc.
>>>>>
>>>>> http://www.wso2.com/ - "Oxygenating the Web Service Platform."
>>>>>
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: axis-c-user-unsubscribe@ws.apache.org
>>>>> For additional commands, e-mail: axis-c-user-help@ws.apache.org
>>>>>
>>>>>
>>>>>
>>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: axis-c-dev-unsubscribe@ws.apache.org
>>>> For additional commands, e-mail: axis-c-dev-help@ws.apache.org
>>>>
>>>>
>>>>
>>>>
>>>>
>>> --
>>> Samisa Abeysinghe
>>> Software Architect; WSO2 Inc.
>>>
>>> http://www.wso2.com/ - "Oxygenating the Web Service Platform."
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: axis-c-dev-unsubscribe@ws.apache.org
>>> For additional commands, e-mail: axis-c-dev-help@ws.apache.org
>>>
>>>
>>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: axis-c-dev-unsubscribe@ws.apache.org
>> For additional commands, e-mail: axis-c-dev-help@ws.apache.org
>>
>>
>>
>>
>
>
> --
> Samisa Abeysinghe
> Software Architect; WSO2 Inc.
>
> http://www.wso2.com/ - "Oxygenating the Web Service Platform."
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: axis-c-dev-unsubscribe@ws.apache.org
> For additional commands, e-mail: axis-c-dev-help@ws.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: axis-c-dev-unsubscribe@ws.apache.org
For additional commands, e-mail: axis-c-dev-help@ws.apache.org
Re: Solution to Core Dump Issue Raised in User List [was: Re: Core
dumps in Axis libraries]
Posted by Samisa Abeysinghe <sa...@wso2.com>.
Were you able to re-create the problem without passing -1 to accept?
Samisa...
Senaka Fernando wrote:
>> Senaka Fernando wrote:
>>
>>> Hi Samisa,
>>>
>>> I fixed two bugs regarding stream handling inside
>>> simple_http_svr_conn.c.
>>> This fixes an issue of this nature.
>>>
>>> Say, we have two server threads that try to serve requests offered to
>>> the
>>> axis2 server. Now, according to what is intended, one must serve and the
>>> other must wait until it can serve. The previous implementation caused
>>> this to fail with a segfault. But, with the fix, it will rather not
>>> crash
>>> but, loop until it gets a chance to serve the request.
>>>
>>>
>> Did you test before the fix and did it seg fault for you? Can you please
>> explain the scenario under which it seg faulted? If it is a problem, I
>> too should be able to re-create it.
>>
>> And how did you solve the problem? By adding a sleep?
>>
>
> I made the accept(2) call fail. By passing a -1 to it. Once that failed,
> there were two places that segfaulted which I already fixed, few minutes
> ago. No, I didn't add a sleep(). I simply prevented the segfault by adding
> necessary is_null checks before calling methods. Then, the server just
> kept on spawning threads and freeing them as it can't serve the request.
> This is the expected behaviour in http_svr_thread.c I believe.
>
> In a real world scenario, a situation similar to a -1 passed to accept(2)
> will be temporal. So a respawned thread down the line should server the
> request without any problem.
>
> Passing -1 to accept was done by,
>
> Index: util/src/network_handler.c
> ===================================================================
> --- util/src/network_handler.c (revision 634578)
> +++ util/src/network_handler.c (working copy)
> @@ -221,7 +221,7 @@
> AXIS2_ENV_CHECK(env, AXIS2_CRITICAL_FAILURE);
>
> cli_len = sizeof(cli_addr);
> - cli_socket = accept(svr_socket, (struct sockaddr *) &cli_addr,
> &cli_len);
> + cli_socket = accept(-1, (struct sockaddr *) &cli_addr, &cli_len);
> if (cli_socket < 0)
> AXIS2_LOG_ERROR(env->log, AXIS2_LOG_SI,
> "[Axis2][network_handler] Socket accept \
> ---
>
> I'm not sure whether this is what really lead to the core dump, or was it
> something else.
>
> This was all that I could try on based on the less informative
> valgrind.log that was sent. :(...
>
> Also i did test 1000 back-to-back requests and they all were served
> without any issues.
>
> Regards,
> Senaka
>
>
>> Samisa...
>>
>>
>>> Could this solve the core dump issue?
>>>
>>> Regards,
>>> Senaka
>>>
>>>
>>>
>>>> Raghavendra SM wrote:
>>>>
>>>>
>>>>> As expected, when I put a delay of about 50 milliseconds between the
>>>>> requests there is no dump.
>>>>>
>>>>> But we would still like to know the following,
>>>>> + Isn't Axis capable of handling such huge bombardment of requests (of
>>>>> the order of 1000's) one after another without any delay between the
>>>>> requests? Has anyone tried such client before?
>>>>>
>>>>>
>>>>>
>>>> Yes it is supposed to. We have done performance testing that proved
>>>> that
>>>> Axis2/C can serve more than 300 rps.
>>>>
>>>>
>>>>
>>>>> + Can we make out anything out of the valgrind report or the axis logs
>>>>> that were attached?
>>>>>
>>>>>
>>>>>
>>>> Unfortunately not. When valgrind runs, it makes the system slow, that
>>>> prevents the system crashing in situations like that you are
>>>> experiencing. To solve this kind of a problem, it is a must that you
>>>> provide us with some sample code that can be used to reproduce the
>>>> problem. Without having a way to re-produce the situation, there is no
>>>> way to fix this problem - it could well be a problem in Axis2/C as well
>>>> as it could also be in the code that you use.
>>>>
>>>> Samisa...
>>>>
>>>>
>>>>
>>>>
>>>>> Regards,
>>>>> ~raghav
>>>>>
>>>>>
>>>>> -----Original Message-----
>>>>> From: Senaka Fernando [mailto:senaka@wso2.com]
>>>>> Sent: Thursday, March 06, 2008 4:22 PM
>>>>> To: Apache AXIS C User List
>>>>> Subject: RE: Core dumps in Axis libraries
>>>>>
>>>>> Hi Raghav,
>>>>>
>>>>> Can you please try this with --enable-guthilla=yes, and send the
>>>>> dump.log,
>>>>> and also the appropriate log_files generated by the server/client.
>>>>>
>>>>> There also can be issues with your "condition variable" being used in
>>>>> multiple threads.
>>>>>
>>>>> It also seems that if you pause between your requests you might get
>>>>> this
>>>>> solved. According to what you say, when you increase the log_level,
>>>>> obviously you waste some time there which might be the solution
>>>>> rather.
>>>>> You can use the AXIS2_SLEEP() and AXIS2_USLEEP() inside Axis2/C to
>>>>> pause
>>>>> between requests. You can get an idea on how to use these functions by
>>>>> taking a look at the Axis2/C user_guide/client samples.
>>>>>
>>>>> Regards,
>>>>> Senaka
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>> Thanks,
>>>>>>
>>>>>> As suggested, I tried using axis2c-1.3.0 both with and without
>>>>>> --enable-guthilla=yes. But I still see different dumps consistently.
>>>>>> Please find the attached file with back-traces.
>>>>>>
>>>>>> Are you using the same service client instance from multiple threads?
>>>>>>
>>>>>>
>>>>>>
>>>>>>>>> We have a axis thread pool with default number of threads and a
>>>>>>>>> set
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>> of our own threads(about 10, lets call them "database threads").
>>>>>> + On receiving a HTTP soap request, axis thread invokes the
>>>>>>
>>>>>>
>>>>>>
>>>>> appropriate
>>>>>
>>>>>
>>>>>
>>>>>> function that is bound to our logic.
>>>>>> + Now, the axis thread requests a "database thread" to perform the
>>>>>> appropriate query and waits on a condition variable.
>>>>>> + our "db thread" performs the necessary query and signals on that
>>>>>> condition variable so that axis thread shall take the control over.
>>>>>> + axis thread will create the om output and dispatch as usual.
>>>>>>
>>>>>> My doubts:
>>>>>> + does this design has any apparent drawback when there is a flurry
>>>>>> of
>>>>>> back-to-back soap requests?
>>>>>> + Is there a need to increase the axis thread pool size? If yes, how
>>>>>>
>>>>>>
>>>>>>
>>>>> to
>>>>>
>>>>>
>>>>>
>>>>>> do it?
>>>>>> + As told before, the dump is seen during a particular scenario only.
>>>>>> i.e: while trying to add a list of already existing users to my
>>>>>> database. For each user there is a SOAP request generated and the
>>>>>> requests are sequential, i.e the second request is sent only after
>>>>>> the
>>>>>> response for previous request is received.
>>>>>> + if I increase the log level of my own module that uses axis2c,
>>>>>> there
>>>>>> is no dump. On increasing the log level, my module writes quite a bit
>>>>>>
>>>>>>
>>>>>>
>>>>> of
>>>>>
>>>>>
>>>>>
>>>>>> information to my log file.
>>>>>>
>>>>>> Your help is appreciated.
>>>>>>
>>>>>> Thanks & Regards,
>>>>>> ~raghav
>>>>>>
>>>>>>
>>>>>> -----Original Message-----
>>>>>> From: Samisa Abeysinghe [mailto:samisa@wso2.com]
>>>>>> Sent: Wednesday, March 05, 2008 7:25 PM
>>>>>> To: Apache AXIS C User List
>>>>>> Subject: Re: Core dumps in Axis libraries
>>>>>>
>>>>>> Raghavendra SM wrote:
>>>>>>
>>>>>>
>>>>>>
>>>>>>> Thanks Manjula,
>>>>>>>
>>>>>>> But thats not a viable option for us. We would still want to
>>>>>>> continue
>>>>>>> using axis2C-1.0 for some more time.
>>>>>>>
>>>>>>> Are these dumps well known?
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>> No.
>>>>>>
>>>>>> Are you using the same service client instance from multiple threads?
>>>>>>
>>>>>> Samisa...
>>>>>>
>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: axis-c-user-unsubscribe@ws.apache.org
>>>>>> For additional commands, e-mail: axis-c-user-help@ws.apache.org
>>>>>>
>>>>>>
>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: axis-c-user-unsubscribe@ws.apache.org
>>>>>> For additional commands, e-mail: axis-c-user-help@ws.apache.org
>>>>>>
>>>>>>
>>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: axis-c-user-unsubscribe@ws.apache.org
>>>>> For additional commands, e-mail: axis-c-user-help@ws.apache.org
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: axis-c-user-unsubscribe@ws.apache.org
>>>>> For additional commands, e-mail: axis-c-user-help@ws.apache.org
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>> --
>>>> Samisa Abeysinghe
>>>> Software Architect; WSO2 Inc.
>>>>
>>>> http://www.wso2.com/ - "Oxygenating the Web Service Platform."
>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: axis-c-user-unsubscribe@ws.apache.org
>>>> For additional commands, e-mail: axis-c-user-help@ws.apache.org
>>>>
>>>>
>>>>
>>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: axis-c-dev-unsubscribe@ws.apache.org
>>> For additional commands, e-mail: axis-c-dev-help@ws.apache.org
>>>
>>>
>>>
>>>
>>>
>> --
>> Samisa Abeysinghe
>> Software Architect; WSO2 Inc.
>>
>> http://www.wso2.com/ - "Oxygenating the Web Service Platform."
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: axis-c-dev-unsubscribe@ws.apache.org
>> For additional commands, e-mail: axis-c-dev-help@ws.apache.org
>>
>>
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: axis-c-dev-unsubscribe@ws.apache.org
> For additional commands, e-mail: axis-c-dev-help@ws.apache.org
>
>
>
>
--
Samisa Abeysinghe
Software Architect; WSO2 Inc.
http://www.wso2.com/ - "Oxygenating the Web Service Platform."
---------------------------------------------------------------------
To unsubscribe, e-mail: axis-c-dev-unsubscribe@ws.apache.org
For additional commands, e-mail: axis-c-dev-help@ws.apache.org
Re: Solution to Core Dump Issue Raised in User List [was: Re: Core
dumps in Axis libraries]
Posted by Senaka Fernando <se...@wso2.com>.
> Senaka Fernando wrote:
>> Hi Samisa,
>>
>> I fixed two bugs regarding stream handling inside
>> simple_http_svr_conn.c.
>> This fixes an issue of this nature.
>>
>> Say, we have two server threads that try to serve requests offered to
>> the
>> axis2 server. Now, according to what is intended, one must serve and the
>> other must wait until it can serve. The previous implementation caused
>> this to fail with a segfault. But, with the fix, it will rather not
>> crash
>> but, loop until it gets a chance to serve the request.
>>
>
> Did you test before the fix and did it seg fault for you? Can you please
> explain the scenario under which it seg faulted? If it is a problem, I
> too should be able to re-create it.
>
> And how did you solve the problem? By adding a sleep?
I made the accept(2) call fail. By passing a -1 to it. Once that failed,
there were two places that segfaulted which I already fixed, few minutes
ago. No, I didn't add a sleep(). I simply prevented the segfault by adding
necessary is_null checks before calling methods. Then, the server just
kept on spawning threads and freeing them as it can't serve the request.
This is the expected behaviour in http_svr_thread.c I believe.
In a real world scenario, a situation similar to a -1 passed to accept(2)
will be temporal. So a respawned thread down the line should server the
request without any problem.
Passing -1 to accept was done by,
Index: util/src/network_handler.c
===================================================================
--- util/src/network_handler.c (revision 634578)
+++ util/src/network_handler.c (working copy)
@@ -221,7 +221,7 @@
AXIS2_ENV_CHECK(env, AXIS2_CRITICAL_FAILURE);
cli_len = sizeof(cli_addr);
- cli_socket = accept(svr_socket, (struct sockaddr *) &cli_addr,
&cli_len);
+ cli_socket = accept(-1, (struct sockaddr *) &cli_addr, &cli_len);
if (cli_socket < 0)
AXIS2_LOG_ERROR(env->log, AXIS2_LOG_SI,
"[Axis2][network_handler] Socket accept \
---
I'm not sure whether this is what really lead to the core dump, or was it
something else.
This was all that I could try on based on the less informative
valgrind.log that was sent. :(...
Also i did test 1000 back-to-back requests and they all were served
without any issues.
Regards,
Senaka
>
> Samisa...
>
>> Could this solve the core dump issue?
>>
>> Regards,
>> Senaka
>>
>>
>>> Raghavendra SM wrote:
>>>
>>>> As expected, when I put a delay of about 50 milliseconds between the
>>>> requests there is no dump.
>>>>
>>>> But we would still like to know the following,
>>>> + Isn't Axis capable of handling such huge bombardment of requests (of
>>>> the order of 1000's) one after another without any delay between the
>>>> requests? Has anyone tried such client before?
>>>>
>>>>
>>> Yes it is supposed to. We have done performance testing that proved
>>> that
>>> Axis2/C can serve more than 300 rps.
>>>
>>>
>>>> + Can we make out anything out of the valgrind report or the axis logs
>>>> that were attached?
>>>>
>>>>
>>> Unfortunately not. When valgrind runs, it makes the system slow, that
>>> prevents the system crashing in situations like that you are
>>> experiencing. To solve this kind of a problem, it is a must that you
>>> provide us with some sample code that can be used to reproduce the
>>> problem. Without having a way to re-produce the situation, there is no
>>> way to fix this problem - it could well be a problem in Axis2/C as well
>>> as it could also be in the code that you use.
>>>
>>> Samisa...
>>>
>>>
>>>
>>>> Regards,
>>>> ~raghav
>>>>
>>>>
>>>> -----Original Message-----
>>>> From: Senaka Fernando [mailto:senaka@wso2.com]
>>>> Sent: Thursday, March 06, 2008 4:22 PM
>>>> To: Apache AXIS C User List
>>>> Subject: RE: Core dumps in Axis libraries
>>>>
>>>> Hi Raghav,
>>>>
>>>> Can you please try this with --enable-guthilla=yes, and send the
>>>> dump.log,
>>>> and also the appropriate log_files generated by the server/client.
>>>>
>>>> There also can be issues with your "condition variable" being used in
>>>> multiple threads.
>>>>
>>>> It also seems that if you pause between your requests you might get
>>>> this
>>>> solved. According to what you say, when you increase the log_level,
>>>> obviously you waste some time there which might be the solution
>>>> rather.
>>>> You can use the AXIS2_SLEEP() and AXIS2_USLEEP() inside Axis2/C to
>>>> pause
>>>> between requests. You can get an idea on how to use these functions by
>>>> taking a look at the Axis2/C user_guide/client samples.
>>>>
>>>> Regards,
>>>> Senaka
>>>>
>>>>
>>>>
>>>>> Thanks,
>>>>>
>>>>> As suggested, I tried using axis2c-1.3.0 both with and without
>>>>> --enable-guthilla=yes. But I still see different dumps consistently.
>>>>> Please find the attached file with back-traces.
>>>>>
>>>>> Are you using the same service client instance from multiple threads?
>>>>>
>>>>>
>>>>>>>> We have a axis thread pool with default number of threads and a
>>>>>>>> set
>>>>>>>>
>>>>>>>>
>>>>> of our own threads(about 10, lets call them "database threads").
>>>>> + On receiving a HTTP soap request, axis thread invokes the
>>>>>
>>>>>
>>>> appropriate
>>>>
>>>>
>>>>> function that is bound to our logic.
>>>>> + Now, the axis thread requests a "database thread" to perform the
>>>>> appropriate query and waits on a condition variable.
>>>>> + our "db thread" performs the necessary query and signals on that
>>>>> condition variable so that axis thread shall take the control over.
>>>>> + axis thread will create the om output and dispatch as usual.
>>>>>
>>>>> My doubts:
>>>>> + does this design has any apparent drawback when there is a flurry
>>>>> of
>>>>> back-to-back soap requests?
>>>>> + Is there a need to increase the axis thread pool size? If yes, how
>>>>>
>>>>>
>>>> to
>>>>
>>>>
>>>>> do it?
>>>>> + As told before, the dump is seen during a particular scenario only.
>>>>> i.e: while trying to add a list of already existing users to my
>>>>> database. For each user there is a SOAP request generated and the
>>>>> requests are sequential, i.e the second request is sent only after
>>>>> the
>>>>> response for previous request is received.
>>>>> + if I increase the log level of my own module that uses axis2c,
>>>>> there
>>>>> is no dump. On increasing the log level, my module writes quite a bit
>>>>>
>>>>>
>>>> of
>>>>
>>>>
>>>>> information to my log file.
>>>>>
>>>>> Your help is appreciated.
>>>>>
>>>>> Thanks & Regards,
>>>>> ~raghav
>>>>>
>>>>>
>>>>> -----Original Message-----
>>>>> From: Samisa Abeysinghe [mailto:samisa@wso2.com]
>>>>> Sent: Wednesday, March 05, 2008 7:25 PM
>>>>> To: Apache AXIS C User List
>>>>> Subject: Re: Core dumps in Axis libraries
>>>>>
>>>>> Raghavendra SM wrote:
>>>>>
>>>>>
>>>>>> Thanks Manjula,
>>>>>>
>>>>>> But thats not a viable option for us. We would still want to
>>>>>> continue
>>>>>> using axis2C-1.0 for some more time.
>>>>>>
>>>>>> Are these dumps well known?
>>>>>>
>>>>>>
>>>>> No.
>>>>>
>>>>> Are you using the same service client instance from multiple threads?
>>>>>
>>>>> Samisa...
>>>>>
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: axis-c-user-unsubscribe@ws.apache.org
>>>>> For additional commands, e-mail: axis-c-user-help@ws.apache.org
>>>>>
>>>>>
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: axis-c-user-unsubscribe@ws.apache.org
>>>>> For additional commands, e-mail: axis-c-user-help@ws.apache.org
>>>>>
>>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: axis-c-user-unsubscribe@ws.apache.org
>>>> For additional commands, e-mail: axis-c-user-help@ws.apache.org
>>>>
>>>>
>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: axis-c-user-unsubscribe@ws.apache.org
>>>> For additional commands, e-mail: axis-c-user-help@ws.apache.org
>>>>
>>>>
>>>>
>>>>
>>>>
>>> --
>>> Samisa Abeysinghe
>>> Software Architect; WSO2 Inc.
>>>
>>> http://www.wso2.com/ - "Oxygenating the Web Service Platform."
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: axis-c-user-unsubscribe@ws.apache.org
>>> For additional commands, e-mail: axis-c-user-help@ws.apache.org
>>>
>>>
>>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: axis-c-dev-unsubscribe@ws.apache.org
>> For additional commands, e-mail: axis-c-dev-help@ws.apache.org
>>
>>
>>
>>
>
>
> --
> Samisa Abeysinghe
> Software Architect; WSO2 Inc.
>
> http://www.wso2.com/ - "Oxygenating the Web Service Platform."
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: axis-c-dev-unsubscribe@ws.apache.org
> For additional commands, e-mail: axis-c-dev-help@ws.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: axis-c-dev-unsubscribe@ws.apache.org
For additional commands, e-mail: axis-c-dev-help@ws.apache.org
Re: Solution to Core Dump Issue Raised in User List [was: Re: Core
dumps in Axis libraries]
Posted by Samisa Abeysinghe <sa...@wso2.com>.
Senaka Fernando wrote:
> Hi Samisa,
>
> I fixed two bugs regarding stream handling inside simple_http_svr_conn.c.
> This fixes an issue of this nature.
>
> Say, we have two server threads that try to serve requests offered to the
> axis2 server. Now, according to what is intended, one must serve and the
> other must wait until it can serve. The previous implementation caused
> this to fail with a segfault. But, with the fix, it will rather not crash
> but, loop until it gets a chance to serve the request.
>
Did you test before the fix and did it seg fault for you? Can you please
explain the scenario under which it seg faulted? If it is a problem, I
too should be able to re-create it.
And how did you solve the problem? By adding a sleep?
Samisa...
> Could this solve the core dump issue?
>
> Regards,
> Senaka
>
>
>> Raghavendra SM wrote:
>>
>>> As expected, when I put a delay of about 50 milliseconds between the
>>> requests there is no dump.
>>>
>>> But we would still like to know the following,
>>> + Isn't Axis capable of handling such huge bombardment of requests (of
>>> the order of 1000's) one after another without any delay between the
>>> requests? Has anyone tried such client before?
>>>
>>>
>> Yes it is supposed to. We have done performance testing that proved that
>> Axis2/C can serve more than 300 rps.
>>
>>
>>> + Can we make out anything out of the valgrind report or the axis logs
>>> that were attached?
>>>
>>>
>> Unfortunately not. When valgrind runs, it makes the system slow, that
>> prevents the system crashing in situations like that you are
>> experiencing. To solve this kind of a problem, it is a must that you
>> provide us with some sample code that can be used to reproduce the
>> problem. Without having a way to re-produce the situation, there is no
>> way to fix this problem - it could well be a problem in Axis2/C as well
>> as it could also be in the code that you use.
>>
>> Samisa...
>>
>>
>>
>>> Regards,
>>> ~raghav
>>>
>>>
>>> -----Original Message-----
>>> From: Senaka Fernando [mailto:senaka@wso2.com]
>>> Sent: Thursday, March 06, 2008 4:22 PM
>>> To: Apache AXIS C User List
>>> Subject: RE: Core dumps in Axis libraries
>>>
>>> Hi Raghav,
>>>
>>> Can you please try this with --enable-guthilla=yes, and send the
>>> dump.log,
>>> and also the appropriate log_files generated by the server/client.
>>>
>>> There also can be issues with your "condition variable" being used in
>>> multiple threads.
>>>
>>> It also seems that if you pause between your requests you might get this
>>> solved. According to what you say, when you increase the log_level,
>>> obviously you waste some time there which might be the solution rather.
>>> You can use the AXIS2_SLEEP() and AXIS2_USLEEP() inside Axis2/C to pause
>>> between requests. You can get an idea on how to use these functions by
>>> taking a look at the Axis2/C user_guide/client samples.
>>>
>>> Regards,
>>> Senaka
>>>
>>>
>>>
>>>> Thanks,
>>>>
>>>> As suggested, I tried using axis2c-1.3.0 both with and without
>>>> --enable-guthilla=yes. But I still see different dumps consistently.
>>>> Please find the attached file with back-traces.
>>>>
>>>> Are you using the same service client instance from multiple threads?
>>>>
>>>>
>>>>>>> We have a axis thread pool with default number of threads and a set
>>>>>>>
>>>>>>>
>>>> of our own threads(about 10, lets call them "database threads").
>>>> + On receiving a HTTP soap request, axis thread invokes the
>>>>
>>>>
>>> appropriate
>>>
>>>
>>>> function that is bound to our logic.
>>>> + Now, the axis thread requests a "database thread" to perform the
>>>> appropriate query and waits on a condition variable.
>>>> + our "db thread" performs the necessary query and signals on that
>>>> condition variable so that axis thread shall take the control over.
>>>> + axis thread will create the om output and dispatch as usual.
>>>>
>>>> My doubts:
>>>> + does this design has any apparent drawback when there is a flurry of
>>>> back-to-back soap requests?
>>>> + Is there a need to increase the axis thread pool size? If yes, how
>>>>
>>>>
>>> to
>>>
>>>
>>>> do it?
>>>> + As told before, the dump is seen during a particular scenario only.
>>>> i.e: while trying to add a list of already existing users to my
>>>> database. For each user there is a SOAP request generated and the
>>>> requests are sequential, i.e the second request is sent only after the
>>>> response for previous request is received.
>>>> + if I increase the log level of my own module that uses axis2c, there
>>>> is no dump. On increasing the log level, my module writes quite a bit
>>>>
>>>>
>>> of
>>>
>>>
>>>> information to my log file.
>>>>
>>>> Your help is appreciated.
>>>>
>>>> Thanks & Regards,
>>>> ~raghav
>>>>
>>>>
>>>> -----Original Message-----
>>>> From: Samisa Abeysinghe [mailto:samisa@wso2.com]
>>>> Sent: Wednesday, March 05, 2008 7:25 PM
>>>> To: Apache AXIS C User List
>>>> Subject: Re: Core dumps in Axis libraries
>>>>
>>>> Raghavendra SM wrote:
>>>>
>>>>
>>>>> Thanks Manjula,
>>>>>
>>>>> But thats not a viable option for us. We would still want to continue
>>>>> using axis2C-1.0 for some more time.
>>>>>
>>>>> Are these dumps well known?
>>>>>
>>>>>
>>>> No.
>>>>
>>>> Are you using the same service client instance from multiple threads?
>>>>
>>>> Samisa...
>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: axis-c-user-unsubscribe@ws.apache.org
>>>> For additional commands, e-mail: axis-c-user-help@ws.apache.org
>>>>
>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: axis-c-user-unsubscribe@ws.apache.org
>>>> For additional commands, e-mail: axis-c-user-help@ws.apache.org
>>>>
>>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: axis-c-user-unsubscribe@ws.apache.org
>>> For additional commands, e-mail: axis-c-user-help@ws.apache.org
>>>
>>>
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: axis-c-user-unsubscribe@ws.apache.org
>>> For additional commands, e-mail: axis-c-user-help@ws.apache.org
>>>
>>>
>>>
>>>
>>>
>> --
>> Samisa Abeysinghe
>> Software Architect; WSO2 Inc.
>>
>> http://www.wso2.com/ - "Oxygenating the Web Service Platform."
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: axis-c-user-unsubscribe@ws.apache.org
>> For additional commands, e-mail: axis-c-user-help@ws.apache.org
>>
>>
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: axis-c-dev-unsubscribe@ws.apache.org
> For additional commands, e-mail: axis-c-dev-help@ws.apache.org
>
>
>
>
--
Samisa Abeysinghe
Software Architect; WSO2 Inc.
http://www.wso2.com/ - "Oxygenating the Web Service Platform."
---------------------------------------------------------------------
To unsubscribe, e-mail: axis-c-dev-unsubscribe@ws.apache.org
For additional commands, e-mail: axis-c-dev-help@ws.apache.org
Solution to Core Dump Issue Raised in User List [was: Re: Core
dumps in Axis libraries]
Posted by Senaka Fernando <se...@wso2.com>.
Hi Samisa,
I fixed two bugs regarding stream handling inside simple_http_svr_conn.c.
This fixes an issue of this nature.
Say, we have two server threads that try to serve requests offered to the
axis2 server. Now, according to what is intended, one must serve and the
other must wait until it can serve. The previous implementation caused
this to fail with a segfault. But, with the fix, it will rather not crash
but, loop until it gets a chance to serve the request.
Could this solve the core dump issue?
Regards,
Senaka
> Raghavendra SM wrote:
>> As expected, when I put a delay of about 50 milliseconds between the
>> requests there is no dump.
>>
>> But we would still like to know the following,
>> + Isn't Axis capable of handling such huge bombardment of requests (of
>> the order of 1000's) one after another without any delay between the
>> requests? Has anyone tried such client before?
>>
>
> Yes it is supposed to. We have done performance testing that proved that
> Axis2/C can serve more than 300 rps.
>
>> + Can we make out anything out of the valgrind report or the axis logs
>> that were attached?
>>
>
> Unfortunately not. When valgrind runs, it makes the system slow, that
> prevents the system crashing in situations like that you are
> experiencing. To solve this kind of a problem, it is a must that you
> provide us with some sample code that can be used to reproduce the
> problem. Without having a way to re-produce the situation, there is no
> way to fix this problem - it could well be a problem in Axis2/C as well
> as it could also be in the code that you use.
>
> Samisa...
>
>
>> Regards,
>> ~raghav
>>
>>
>> -----Original Message-----
>> From: Senaka Fernando [mailto:senaka@wso2.com]
>> Sent: Thursday, March 06, 2008 4:22 PM
>> To: Apache AXIS C User List
>> Subject: RE: Core dumps in Axis libraries
>>
>> Hi Raghav,
>>
>> Can you please try this with --enable-guthilla=yes, and send the
>> dump.log,
>> and also the appropriate log_files generated by the server/client.
>>
>> There also can be issues with your "condition variable" being used in
>> multiple threads.
>>
>> It also seems that if you pause between your requests you might get this
>> solved. According to what you say, when you increase the log_level,
>> obviously you waste some time there which might be the solution rather.
>> You can use the AXIS2_SLEEP() and AXIS2_USLEEP() inside Axis2/C to pause
>> between requests. You can get an idea on how to use these functions by
>> taking a look at the Axis2/C user_guide/client samples.
>>
>> Regards,
>> Senaka
>>
>>
>>> Thanks,
>>>
>>> As suggested, I tried using axis2c-1.3.0 both with and without
>>> --enable-guthilla=yes. But I still see different dumps consistently.
>>> Please find the attached file with back-traces.
>>>
>>> Are you using the same service client instance from multiple threads?
>>>
>>>>>> We have a axis thread pool with default number of threads and a set
>>>>>>
>>> of our own threads(about 10, lets call them "database threads").
>>> + On receiving a HTTP soap request, axis thread invokes the
>>>
>> appropriate
>>
>>> function that is bound to our logic.
>>> + Now, the axis thread requests a "database thread" to perform the
>>> appropriate query and waits on a condition variable.
>>> + our "db thread" performs the necessary query and signals on that
>>> condition variable so that axis thread shall take the control over.
>>> + axis thread will create the om output and dispatch as usual.
>>>
>>> My doubts:
>>> + does this design has any apparent drawback when there is a flurry of
>>> back-to-back soap requests?
>>> + Is there a need to increase the axis thread pool size? If yes, how
>>>
>> to
>>
>>> do it?
>>> + As told before, the dump is seen during a particular scenario only.
>>> i.e: while trying to add a list of already existing users to my
>>> database. For each user there is a SOAP request generated and the
>>> requests are sequential, i.e the second request is sent only after the
>>> response for previous request is received.
>>> + if I increase the log level of my own module that uses axis2c, there
>>> is no dump. On increasing the log level, my module writes quite a bit
>>>
>> of
>>
>>> information to my log file.
>>>
>>> Your help is appreciated.
>>>
>>> Thanks & Regards,
>>> ~raghav
>>>
>>>
>>> -----Original Message-----
>>> From: Samisa Abeysinghe [mailto:samisa@wso2.com]
>>> Sent: Wednesday, March 05, 2008 7:25 PM
>>> To: Apache AXIS C User List
>>> Subject: Re: Core dumps in Axis libraries
>>>
>>> Raghavendra SM wrote:
>>>
>>>> Thanks Manjula,
>>>>
>>>> But thats not a viable option for us. We would still want to continue
>>>> using axis2C-1.0 for some more time.
>>>>
>>>> Are these dumps well known?
>>>>
>>> No.
>>>
>>> Are you using the same service client instance from multiple threads?
>>>
>>> Samisa...
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: axis-c-user-unsubscribe@ws.apache.org
>>> For additional commands, e-mail: axis-c-user-help@ws.apache.org
>>>
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: axis-c-user-unsubscribe@ws.apache.org
>>> For additional commands, e-mail: axis-c-user-help@ws.apache.org
>>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: axis-c-user-unsubscribe@ws.apache.org
>> For additional commands, e-mail: axis-c-user-help@ws.apache.org
>>
>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: axis-c-user-unsubscribe@ws.apache.org
>> For additional commands, e-mail: axis-c-user-help@ws.apache.org
>>
>>
>>
>>
>
>
> --
> Samisa Abeysinghe
> Software Architect; WSO2 Inc.
>
> http://www.wso2.com/ - "Oxygenating the Web Service Platform."
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: axis-c-user-unsubscribe@ws.apache.org
> For additional commands, e-mail: axis-c-user-help@ws.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: axis-c-dev-unsubscribe@ws.apache.org
For additional commands, e-mail: axis-c-dev-help@ws.apache.org
Re: Core dumps in Axis libraries
Posted by Samisa Abeysinghe <sa...@wso2.com>.
Raghavendra SM wrote:
> As expected, when I put a delay of about 50 milliseconds between the
> requests there is no dump.
>
> But we would still like to know the following,
> + Isn't Axis capable of handling such huge bombardment of requests (of
> the order of 1000's) one after another without any delay between the
> requests? Has anyone tried such client before?
>
Yes it is supposed to. We have done performance testing that proved that
Axis2/C can serve more than 300 rps.
> + Can we make out anything out of the valgrind report or the axis logs
> that were attached?
>
Unfortunately not. When valgrind runs, it makes the system slow, that
prevents the system crashing in situations like that you are
experiencing. To solve this kind of a problem, it is a must that you
provide us with some sample code that can be used to reproduce the
problem. Without having a way to re-produce the situation, there is no
way to fix this problem - it could well be a problem in Axis2/C as well
as it could also be in the code that you use.
Samisa...
> Regards,
> ~raghav
>
>
> -----Original Message-----
> From: Senaka Fernando [mailto:senaka@wso2.com]
> Sent: Thursday, March 06, 2008 4:22 PM
> To: Apache AXIS C User List
> Subject: RE: Core dumps in Axis libraries
>
> Hi Raghav,
>
> Can you please try this with --enable-guthilla=yes, and send the
> dump.log,
> and also the appropriate log_files generated by the server/client.
>
> There also can be issues with your "condition variable" being used in
> multiple threads.
>
> It also seems that if you pause between your requests you might get this
> solved. According to what you say, when you increase the log_level,
> obviously you waste some time there which might be the solution rather.
> You can use the AXIS2_SLEEP() and AXIS2_USLEEP() inside Axis2/C to pause
> between requests. You can get an idea on how to use these functions by
> taking a look at the Axis2/C user_guide/client samples.
>
> Regards,
> Senaka
>
>
>> Thanks,
>>
>> As suggested, I tried using axis2c-1.3.0 both with and without
>> --enable-guthilla=yes. But I still see different dumps consistently.
>> Please find the attached file with back-traces.
>>
>> Are you using the same service client instance from multiple threads?
>>
>>>>> We have a axis thread pool with default number of threads and a set
>>>>>
>> of our own threads(about 10, lets call them "database threads").
>> + On receiving a HTTP soap request, axis thread invokes the
>>
> appropriate
>
>> function that is bound to our logic.
>> + Now, the axis thread requests a "database thread" to perform the
>> appropriate query and waits on a condition variable.
>> + our "db thread" performs the necessary query and signals on that
>> condition variable so that axis thread shall take the control over.
>> + axis thread will create the om output and dispatch as usual.
>>
>> My doubts:
>> + does this design has any apparent drawback when there is a flurry of
>> back-to-back soap requests?
>> + Is there a need to increase the axis thread pool size? If yes, how
>>
> to
>
>> do it?
>> + As told before, the dump is seen during a particular scenario only.
>> i.e: while trying to add a list of already existing users to my
>> database. For each user there is a SOAP request generated and the
>> requests are sequential, i.e the second request is sent only after the
>> response for previous request is received.
>> + if I increase the log level of my own module that uses axis2c, there
>> is no dump. On increasing the log level, my module writes quite a bit
>>
> of
>
>> information to my log file.
>>
>> Your help is appreciated.
>>
>> Thanks & Regards,
>> ~raghav
>>
>>
>> -----Original Message-----
>> From: Samisa Abeysinghe [mailto:samisa@wso2.com]
>> Sent: Wednesday, March 05, 2008 7:25 PM
>> To: Apache AXIS C User List
>> Subject: Re: Core dumps in Axis libraries
>>
>> Raghavendra SM wrote:
>>
>>> Thanks Manjula,
>>>
>>> But thats not a viable option for us. We would still want to continue
>>> using axis2C-1.0 for some more time.
>>>
>>> Are these dumps well known?
>>>
>> No.
>>
>> Are you using the same service client instance from multiple threads?
>>
>> Samisa...
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: axis-c-user-unsubscribe@ws.apache.org
>> For additional commands, e-mail: axis-c-user-help@ws.apache.org
>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: axis-c-user-unsubscribe@ws.apache.org
>> For additional commands, e-mail: axis-c-user-help@ws.apache.org
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: axis-c-user-unsubscribe@ws.apache.org
> For additional commands, e-mail: axis-c-user-help@ws.apache.org
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: axis-c-user-unsubscribe@ws.apache.org
> For additional commands, e-mail: axis-c-user-help@ws.apache.org
>
>
>
>
--
Samisa Abeysinghe
Software Architect; WSO2 Inc.
http://www.wso2.com/ - "Oxygenating the Web Service Platform."
---------------------------------------------------------------------
To unsubscribe, e-mail: axis-c-user-unsubscribe@ws.apache.org
For additional commands, e-mail: axis-c-user-help@ws.apache.org
RE: Core dumps in Axis libraries
Posted by Raghavendra SM <ra...@aylus.com>.
As expected, when I put a delay of about 50 milliseconds between the
requests there is no dump.
But we would still like to know the following,
+ Isn't Axis capable of handling such huge bombardment of requests (of
the order of 1000's) one after another without any delay between the
requests? Has anyone tried such client before?
+ Can we make out anything out of the valgrind report or the axis logs
that were attached?
Regards,
~raghav
-----Original Message-----
From: Senaka Fernando [mailto:senaka@wso2.com]
Sent: Thursday, March 06, 2008 4:22 PM
To: Apache AXIS C User List
Subject: RE: Core dumps in Axis libraries
Hi Raghav,
Can you please try this with --enable-guthilla=yes, and send the
dump.log,
and also the appropriate log_files generated by the server/client.
There also can be issues with your "condition variable" being used in
multiple threads.
It also seems that if you pause between your requests you might get this
solved. According to what you say, when you increase the log_level,
obviously you waste some time there which might be the solution rather.
You can use the AXIS2_SLEEP() and AXIS2_USLEEP() inside Axis2/C to pause
between requests. You can get an idea on how to use these functions by
taking a look at the Axis2/C user_guide/client samples.
Regards,
Senaka
> Thanks,
>
> As suggested, I tried using axis2c-1.3.0 both with and without
> --enable-guthilla=yes. But I still see different dumps consistently.
> Please find the attached file with back-traces.
>
> Are you using the same service client instance from multiple threads?
>>>> We have a axis thread pool with default number of threads and a set
> of our own threads(about 10, lets call them "database threads").
> + On receiving a HTTP soap request, axis thread invokes the
appropriate
> function that is bound to our logic.
> + Now, the axis thread requests a "database thread" to perform the
> appropriate query and waits on a condition variable.
> + our "db thread" performs the necessary query and signals on that
> condition variable so that axis thread shall take the control over.
> + axis thread will create the om output and dispatch as usual.
>
> My doubts:
> + does this design has any apparent drawback when there is a flurry of
> back-to-back soap requests?
> + Is there a need to increase the axis thread pool size? If yes, how
to
> do it?
> + As told before, the dump is seen during a particular scenario only.
> i.e: while trying to add a list of already existing users to my
> database. For each user there is a SOAP request generated and the
> requests are sequential, i.e the second request is sent only after the
> response for previous request is received.
> + if I increase the log level of my own module that uses axis2c, there
> is no dump. On increasing the log level, my module writes quite a bit
of
> information to my log file.
>
> Your help is appreciated.
>
> Thanks & Regards,
> ~raghav
>
>
> -----Original Message-----
> From: Samisa Abeysinghe [mailto:samisa@wso2.com]
> Sent: Wednesday, March 05, 2008 7:25 PM
> To: Apache AXIS C User List
> Subject: Re: Core dumps in Axis libraries
>
> Raghavendra SM wrote:
>> Thanks Manjula,
>>
>> But thats not a viable option for us. We would still want to continue
>> using axis2C-1.0 for some more time.
>>
>> Are these dumps well known?
>
> No.
>
> Are you using the same service client instance from multiple threads?
>
> Samisa...
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: axis-c-user-unsubscribe@ws.apache.org
> For additional commands, e-mail: axis-c-user-help@ws.apache.org
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: axis-c-user-unsubscribe@ws.apache.org
> For additional commands, e-mail: axis-c-user-help@ws.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: axis-c-user-unsubscribe@ws.apache.org
For additional commands, e-mail: axis-c-user-help@ws.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: axis-c-user-unsubscribe@ws.apache.org
For additional commands, e-mail: axis-c-user-help@ws.apache.org
Re: Core dumps in Axis libraries
Posted by Samisa Abeysinghe <sa...@wso2.com>.
I too think that it is not the place that it is crashing. We have to
locate the place where it is corrupting memory.
Samisa...
Raghavendra SM wrote:
> Isn't that just reflecting that the process has been killed?
> I can't any programming error being detected in that. Please correct me
> if wrong.
>
> Regards,
> ~raghav
>
>
> -----Original Message-----
> From: Senaka Fernando [mailto:senaka@wso2.com]
> Sent: Monday, March 10, 2008 1:02 PM
> To: Apache AXIS C User List
> Subject: RE: Core dumps in Axis libraries
>
> Found it!
>
> ==22786== Process terminating with default action of signal 15 (SIGTERM)
> ==22786== at 0x403760E: accept (in /lib/tls/libpthread-2.3.6.so)
> ==22786== by 0x44C2CF2: axis2_http_svr_thread_run
> (http_svr_thread.c:128)
> ==22786== by 0x44C28D4: axis2_http_server_start (http_receiver.c:227)
> ==22786== by 0x44DA92E: axis2_transport_receiver_start
> (transport_receiver.c:41)
> ==22786== by 0x8084D3C: axis_init(int, char**)
> (http_server_main.cpp:155)
> ==22786== by 0x8059896: main (sps_server.h:159)
>
> Regards,
> Senaka
>
>
>> Manjula,
>>
>> This was with out my logs enabled.
>>
>> Regards,
>> ~raghav
>>
>>
>> -----Original Message-----
>> From: Manjula Peiris [mailto:manjula@wso2.com]
>> Sent: Monday, March 10, 2008 12:58 PM
>> To: Apache AXIS C User List
>> Subject: RE: Core dumps in Axis libraries
>>
>> Is this valgrind report is when your log level increased or not ? If
>>
> yes
>
>> then can you try valgrind without increasing the log level.
>>
>> -Manjula.
>>
>>
>> On Mon, 2008-03-10 at 12:39 +0530, Raghavendra SM wrote:
>>
>>> When I tried the Valgrind, it doesn't point to any obvious clues
>>> atleast. It doesn't show any "definitely lost" or "indirectly lost"
>>> blocks. Though, there are some "still reachable" and "possibly lost"
>>> blocks.
>>>
>>> Please find the attached valgrind report. Please let me know if the
>>> valgrind is revealing something here?
>>>
>>> Thanks & Regards,
>>> ~raghav
>>>
>>> -----Original Message-----
>>> From: Samisa Abeysinghe [mailto:samisa@wso2.com]
>>> Sent: Thursday, March 06, 2008 7:07 PM
>>> To: Apache AXIS C User List
>>> Subject: Re: Core dumps in Axis libraries
>>>
>>> Raghavendra SM wrote:
>>>
>>>> Thanks Folks,
>>>>
>>>> I'm attaching the server log during the crash. Also, the first
>>>> back-trace in the dump.log points to dump with guthilla=yes.
>>>>
>>> The attached dump does not lead to a clue on what is going wrong. It
>>> looks to me as if there is some or the other memory corruption
>>> somewhere. I hope valgrind would lead to a clue.
>>>
>>> Thanks,
>>> Samisa...
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: axis-c-user-unsubscribe@ws.apache.org
>>> For additional commands, e-mail: axis-c-user-help@ws.apache.org
>>>
>>>
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: axis-c-user-unsubscribe@ws.apache.org
>>> For additional commands, e-mail: axis-c-user-help@ws.apache.org
>>>
>>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: axis-c-user-unsubscribe@ws.apache.org
>> For additional commands, e-mail: axis-c-user-help@ws.apache.org
>>
>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: axis-c-user-unsubscribe@ws.apache.org
>> For additional commands, e-mail: axis-c-user-help@ws.apache.org
>>
>>
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: axis-c-user-unsubscribe@ws.apache.org
> For additional commands, e-mail: axis-c-user-help@ws.apache.org
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: axis-c-user-unsubscribe@ws.apache.org
> For additional commands, e-mail: axis-c-user-help@ws.apache.org
>
>
>
>
--
Samisa Abeysinghe
Software Architect; WSO2 Inc.
http://www.wso2.com/ - "Oxygenating the Web Service Platform."
---------------------------------------------------------------------
To unsubscribe, e-mail: axis-c-user-unsubscribe@ws.apache.org
For additional commands, e-mail: axis-c-user-help@ws.apache.org
RE: Core dumps in Axis libraries
Posted by Raghavendra SM <ra...@aylus.com>.
Isn't that just reflecting that the process has been killed?
I can't any programming error being detected in that. Please correct me
if wrong.
Regards,
~raghav
-----Original Message-----
From: Senaka Fernando [mailto:senaka@wso2.com]
Sent: Monday, March 10, 2008 1:02 PM
To: Apache AXIS C User List
Subject: RE: Core dumps in Axis libraries
Found it!
==22786== Process terminating with default action of signal 15 (SIGTERM)
==22786== at 0x403760E: accept (in /lib/tls/libpthread-2.3.6.so)
==22786== by 0x44C2CF2: axis2_http_svr_thread_run
(http_svr_thread.c:128)
==22786== by 0x44C28D4: axis2_http_server_start (http_receiver.c:227)
==22786== by 0x44DA92E: axis2_transport_receiver_start
(transport_receiver.c:41)
==22786== by 0x8084D3C: axis_init(int, char**)
(http_server_main.cpp:155)
==22786== by 0x8059896: main (sps_server.h:159)
Regards,
Senaka
> Manjula,
>
> This was with out my logs enabled.
>
> Regards,
> ~raghav
>
>
> -----Original Message-----
> From: Manjula Peiris [mailto:manjula@wso2.com]
> Sent: Monday, March 10, 2008 12:58 PM
> To: Apache AXIS C User List
> Subject: RE: Core dumps in Axis libraries
>
> Is this valgrind report is when your log level increased or not ? If
yes
> then can you try valgrind without increasing the log level.
>
> -Manjula.
>
>
> On Mon, 2008-03-10 at 12:39 +0530, Raghavendra SM wrote:
>> When I tried the Valgrind, it doesn't point to any obvious clues
>> atleast. It doesn't show any "definitely lost" or "indirectly lost"
>> blocks. Though, there are some "still reachable" and "possibly lost"
>> blocks.
>>
>> Please find the attached valgrind report. Please let me know if the
>> valgrind is revealing something here?
>>
>> Thanks & Regards,
>> ~raghav
>>
>> -----Original Message-----
>> From: Samisa Abeysinghe [mailto:samisa@wso2.com]
>> Sent: Thursday, March 06, 2008 7:07 PM
>> To: Apache AXIS C User List
>> Subject: Re: Core dumps in Axis libraries
>>
>> Raghavendra SM wrote:
>> > Thanks Folks,
>> >
>> > I'm attaching the server log during the crash. Also, the first
>> > back-trace in the dump.log points to dump with guthilla=yes.
>>
>> The attached dump does not lead to a clue on what is going wrong. It
>> looks to me as if there is some or the other memory corruption
>> somewhere. I hope valgrind would lead to a clue.
>>
>> Thanks,
>> Samisa...
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: axis-c-user-unsubscribe@ws.apache.org
>> For additional commands, e-mail: axis-c-user-help@ws.apache.org
>>
>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: axis-c-user-unsubscribe@ws.apache.org
>> For additional commands, e-mail: axis-c-user-help@ws.apache.org
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: axis-c-user-unsubscribe@ws.apache.org
> For additional commands, e-mail: axis-c-user-help@ws.apache.org
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: axis-c-user-unsubscribe@ws.apache.org
> For additional commands, e-mail: axis-c-user-help@ws.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: axis-c-user-unsubscribe@ws.apache.org
For additional commands, e-mail: axis-c-user-help@ws.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: axis-c-user-unsubscribe@ws.apache.org
For additional commands, e-mail: axis-c-user-help@ws.apache.org
RE: Core dumps in Axis libraries
Posted by Senaka Fernando <se...@wso2.com>.
Found it!
==22786== Process terminating with default action of signal 15 (SIGTERM)
==22786== at 0x403760E: accept (in /lib/tls/libpthread-2.3.6.so)
==22786== by 0x44C2CF2: axis2_http_svr_thread_run (http_svr_thread.c:128)
==22786== by 0x44C28D4: axis2_http_server_start (http_receiver.c:227)
==22786== by 0x44DA92E: axis2_transport_receiver_start
(transport_receiver.c:41)
==22786== by 0x8084D3C: axis_init(int, char**) (http_server_main.cpp:155)
==22786== by 0x8059896: main (sps_server.h:159)
Regards,
Senaka
> Manjula,
>
> This was with out my logs enabled.
>
> Regards,
> ~raghav
>
>
> -----Original Message-----
> From: Manjula Peiris [mailto:manjula@wso2.com]
> Sent: Monday, March 10, 2008 12:58 PM
> To: Apache AXIS C User List
> Subject: RE: Core dumps in Axis libraries
>
> Is this valgrind report is when your log level increased or not ? If yes
> then can you try valgrind without increasing the log level.
>
> -Manjula.
>
>
> On Mon, 2008-03-10 at 12:39 +0530, Raghavendra SM wrote:
>> When I tried the Valgrind, it doesn't point to any obvious clues
>> atleast. It doesn't show any "definitely lost" or "indirectly lost"
>> blocks. Though, there are some "still reachable" and "possibly lost"
>> blocks.
>>
>> Please find the attached valgrind report. Please let me know if the
>> valgrind is revealing something here?
>>
>> Thanks & Regards,
>> ~raghav
>>
>> -----Original Message-----
>> From: Samisa Abeysinghe [mailto:samisa@wso2.com]
>> Sent: Thursday, March 06, 2008 7:07 PM
>> To: Apache AXIS C User List
>> Subject: Re: Core dumps in Axis libraries
>>
>> Raghavendra SM wrote:
>> > Thanks Folks,
>> >
>> > I'm attaching the server log during the crash. Also, the first
>> > back-trace in the dump.log points to dump with guthilla=yes.
>>
>> The attached dump does not lead to a clue on what is going wrong. It
>> looks to me as if there is some or the other memory corruption
>> somewhere. I hope valgrind would lead to a clue.
>>
>> Thanks,
>> Samisa...
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: axis-c-user-unsubscribe@ws.apache.org
>> For additional commands, e-mail: axis-c-user-help@ws.apache.org
>>
>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: axis-c-user-unsubscribe@ws.apache.org
>> For additional commands, e-mail: axis-c-user-help@ws.apache.org
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: axis-c-user-unsubscribe@ws.apache.org
> For additional commands, e-mail: axis-c-user-help@ws.apache.org
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: axis-c-user-unsubscribe@ws.apache.org
> For additional commands, e-mail: axis-c-user-help@ws.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: axis-c-user-unsubscribe@ws.apache.org
For additional commands, e-mail: axis-c-user-help@ws.apache.org
RE: Core dumps in Axis libraries
Posted by Raghavendra SM <ra...@aylus.com>.
Manjula,
This was with out my logs enabled.
Regards,
~raghav
-----Original Message-----
From: Manjula Peiris [mailto:manjula@wso2.com]
Sent: Monday, March 10, 2008 12:58 PM
To: Apache AXIS C User List
Subject: RE: Core dumps in Axis libraries
Is this valgrind report is when your log level increased or not ? If yes
then can you try valgrind without increasing the log level.
-Manjula.
On Mon, 2008-03-10 at 12:39 +0530, Raghavendra SM wrote:
> When I tried the Valgrind, it doesn't point to any obvious clues
> atleast. It doesn't show any "definitely lost" or "indirectly lost"
> blocks. Though, there are some "still reachable" and "possibly lost"
> blocks.
>
> Please find the attached valgrind report. Please let me know if the
> valgrind is revealing something here?
>
> Thanks & Regards,
> ~raghav
>
> -----Original Message-----
> From: Samisa Abeysinghe [mailto:samisa@wso2.com]
> Sent: Thursday, March 06, 2008 7:07 PM
> To: Apache AXIS C User List
> Subject: Re: Core dumps in Axis libraries
>
> Raghavendra SM wrote:
> > Thanks Folks,
> >
> > I'm attaching the server log during the crash. Also, the first
> > back-trace in the dump.log points to dump with guthilla=yes.
>
> The attached dump does not lead to a clue on what is going wrong. It
> looks to me as if there is some or the other memory corruption
> somewhere. I hope valgrind would lead to a clue.
>
> Thanks,
> Samisa...
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: axis-c-user-unsubscribe@ws.apache.org
> For additional commands, e-mail: axis-c-user-help@ws.apache.org
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: axis-c-user-unsubscribe@ws.apache.org
> For additional commands, e-mail: axis-c-user-help@ws.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: axis-c-user-unsubscribe@ws.apache.org
For additional commands, e-mail: axis-c-user-help@ws.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: axis-c-user-unsubscribe@ws.apache.org
For additional commands, e-mail: axis-c-user-help@ws.apache.org
RE: Core dumps in Axis libraries
Posted by Manjula Peiris <ma...@wso2.com>.
Is this valgrind report is when your log level increased or not ? If yes
then can you try valgrind without increasing the log level.
-Manjula.
On Mon, 2008-03-10 at 12:39 +0530, Raghavendra SM wrote:
> When I tried the Valgrind, it doesn't point to any obvious clues
> atleast. It doesn't show any "definitely lost" or "indirectly lost"
> blocks. Though, there are some "still reachable" and "possibly lost"
> blocks.
>
> Please find the attached valgrind report. Please let me know if the
> valgrind is revealing something here?
>
> Thanks & Regards,
> ~raghav
>
> -----Original Message-----
> From: Samisa Abeysinghe [mailto:samisa@wso2.com]
> Sent: Thursday, March 06, 2008 7:07 PM
> To: Apache AXIS C User List
> Subject: Re: Core dumps in Axis libraries
>
> Raghavendra SM wrote:
> > Thanks Folks,
> >
> > I'm attaching the server log during the crash. Also, the first
> > back-trace in the dump.log points to dump with guthilla=yes.
>
> The attached dump does not lead to a clue on what is going wrong. It
> looks to me as if there is some or the other memory corruption
> somewhere. I hope valgrind would lead to a clue.
>
> Thanks,
> Samisa...
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: axis-c-user-unsubscribe@ws.apache.org
> For additional commands, e-mail: axis-c-user-help@ws.apache.org
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: axis-c-user-unsubscribe@ws.apache.org
> For additional commands, e-mail: axis-c-user-help@ws.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: axis-c-user-unsubscribe@ws.apache.org
For additional commands, e-mail: axis-c-user-help@ws.apache.org
RE: Core dumps in Axis libraries
Posted by Raghavendra SM <ra...@aylus.com>.
Sorry, I forgot to attach the report.
Regards,
~raghav
-----Original Message-----
From: Raghavendra SM
Sent: Monday, March 10, 2008 12:40 PM
To: axis-c-user
Subject: RE: Core dumps in Axis libraries
When I tried the Valgrind, it doesn't point to any obvious clues
atleast. It doesn't show any "definitely lost" or "indirectly lost"
blocks. Though, there are some "still reachable" and "possibly lost"
blocks.
Please find the attached valgrind report. Please let me know if the
valgrind is revealing something here?
Thanks & Regards,
~raghav
-----Original Message-----
From: Samisa Abeysinghe [mailto:samisa@wso2.com]
Sent: Thursday, March 06, 2008 7:07 PM
To: Apache AXIS C User List
Subject: Re: Core dumps in Axis libraries
Raghavendra SM wrote:
> Thanks Folks,
>
> I'm attaching the server log during the crash. Also, the first
> back-trace in the dump.log points to dump with guthilla=yes.
The attached dump does not lead to a clue on what is going wrong. It
looks to me as if there is some or the other memory corruption
somewhere. I hope valgrind would lead to a clue.
Thanks,
Samisa...
---------------------------------------------------------------------
To unsubscribe, e-mail: axis-c-user-unsubscribe@ws.apache.org
For additional commands, e-mail: axis-c-user-help@ws.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: axis-c-user-unsubscribe@ws.apache.org
For additional commands, e-mail: axis-c-user-help@ws.apache.org
RE: Core dumps in Axis libraries
Posted by Raghavendra SM <ra...@aylus.com>.
When I tried the Valgrind, it doesn't point to any obvious clues
atleast. It doesn't show any "definitely lost" or "indirectly lost"
blocks. Though, there are some "still reachable" and "possibly lost"
blocks.
Please find the attached valgrind report. Please let me know if the
valgrind is revealing something here?
Thanks & Regards,
~raghav
-----Original Message-----
From: Samisa Abeysinghe [mailto:samisa@wso2.com]
Sent: Thursday, March 06, 2008 7:07 PM
To: Apache AXIS C User List
Subject: Re: Core dumps in Axis libraries
Raghavendra SM wrote:
> Thanks Folks,
>
> I'm attaching the server log during the crash. Also, the first
> back-trace in the dump.log points to dump with guthilla=yes.
The attached dump does not lead to a clue on what is going wrong. It
looks to me as if there is some or the other memory corruption
somewhere. I hope valgrind would lead to a clue.
Thanks,
Samisa...
---------------------------------------------------------------------
To unsubscribe, e-mail: axis-c-user-unsubscribe@ws.apache.org
For additional commands, e-mail: axis-c-user-help@ws.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: axis-c-user-unsubscribe@ws.apache.org
For additional commands, e-mail: axis-c-user-help@ws.apache.org
Re: Core dumps in Axis libraries
Posted by Samisa Abeysinghe <sa...@wso2.com>.
Raghavendra SM wrote:
> Thanks Folks,
>
> I'm attaching the server log during the crash. Also, the first
> back-trace in the dump.log points to dump with guthilla=yes.
The attached dump does not lead to a clue on what is going wrong. It
looks to me as if there is some or the other memory corruption
somewhere. I hope valgrind would lead to a clue.
Thanks,
Samisa...
---------------------------------------------------------------------
To unsubscribe, e-mail: axis-c-user-unsubscribe@ws.apache.org
For additional commands, e-mail: axis-c-user-help@ws.apache.org
RE: Core dumps in Axis libraries
Posted by Raghavendra SM <ra...@aylus.com>.
Thanks Folks,
I'm attaching the server log during the crash. Also, the first
back-trace in the dump.log points to dump with guthilla=yes.
Yes, I'll try with Valgrind too.
Regards,
~raghav
-----Original Message-----
From: Senaka Fernando [mailto:senaka@wso2.com]
Sent: Thursday, March 06, 2008 4:22 PM
To: Apache AXIS C User List
Subject: RE: Core dumps in Axis libraries
Hi Raghav,
Can you please try this with --enable-guthilla=yes, and send the
dump.log,
and also the appropriate log_files generated by the server/client.
There also can be issues with your "condition variable" being used in
multiple threads.
It also seems that if you pause between your requests you might get this
solved. According to what you say, when you increase the log_level,
obviously you waste some time there which might be the solution rather.
You can use the AXIS2_SLEEP() and AXIS2_USLEEP() inside Axis2/C to pause
between requests. You can get an idea on how to use these functions by
taking a look at the Axis2/C user_guide/client samples.
Regards,
Senaka
> Thanks,
>
> As suggested, I tried using axis2c-1.3.0 both with and without
> --enable-guthilla=yes. But I still see different dumps consistently.
> Please find the attached file with back-traces.
>
> Are you using the same service client instance from multiple threads?
>>>> We have a axis thread pool with default number of threads and a set
> of our own threads(about 10, lets call them "database threads").
> + On receiving a HTTP soap request, axis thread invokes the
appropriate
> function that is bound to our logic.
> + Now, the axis thread requests a "database thread" to perform the
> appropriate query and waits on a condition variable.
> + our "db thread" performs the necessary query and signals on that
> condition variable so that axis thread shall take the control over.
> + axis thread will create the om output and dispatch as usual.
>
> My doubts:
> + does this design has any apparent drawback when there is a flurry of
> back-to-back soap requests?
> + Is there a need to increase the axis thread pool size? If yes, how
to
> do it?
> + As told before, the dump is seen during a particular scenario only.
> i.e: while trying to add a list of already existing users to my
> database. For each user there is a SOAP request generated and the
> requests are sequential, i.e the second request is sent only after the
> response for previous request is received.
> + if I increase the log level of my own module that uses axis2c, there
> is no dump. On increasing the log level, my module writes quite a bit
of
> information to my log file.
>
> Your help is appreciated.
>
> Thanks & Regards,
> ~raghav
>
>
> -----Original Message-----
> From: Samisa Abeysinghe [mailto:samisa@wso2.com]
> Sent: Wednesday, March 05, 2008 7:25 PM
> To: Apache AXIS C User List
> Subject: Re: Core dumps in Axis libraries
>
> Raghavendra SM wrote:
>> Thanks Manjula,
>>
>> But thats not a viable option for us. We would still want to continue
>> using axis2C-1.0 for some more time.
>>
>> Are these dumps well known?
>
> No.
>
> Are you using the same service client instance from multiple threads?
>
> Samisa...
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: axis-c-user-unsubscribe@ws.apache.org
> For additional commands, e-mail: axis-c-user-help@ws.apache.org
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: axis-c-user-unsubscribe@ws.apache.org
> For additional commands, e-mail: axis-c-user-help@ws.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: axis-c-user-unsubscribe@ws.apache.org
For additional commands, e-mail: axis-c-user-help@ws.apache.org
RE: Core dumps in Axis libraries
Posted by Senaka Fernando <se...@wso2.com>.
Hi Raghav,
Can you please try this with --enable-guthilla=yes, and send the dump.log,
and also the appropriate log_files generated by the server/client.
There also can be issues with your "condition variable" being used in
multiple threads.
It also seems that if you pause between your requests you might get this
solved. According to what you say, when you increase the log_level,
obviously you waste some time there which might be the solution rather.
You can use the AXIS2_SLEEP() and AXIS2_USLEEP() inside Axis2/C to pause
between requests. You can get an idea on how to use these functions by
taking a look at the Axis2/C user_guide/client samples.
Regards,
Senaka
> Thanks,
>
> As suggested, I tried using axis2c-1.3.0 both with and without
> --enable-guthilla=yes. But I still see different dumps consistently.
> Please find the attached file with back-traces.
>
> Are you using the same service client instance from multiple threads?
>>>> We have a axis thread pool with default number of threads and a set
> of our own threads(about 10, lets call them "database threads").
> + On receiving a HTTP soap request, axis thread invokes the appropriate
> function that is bound to our logic.
> + Now, the axis thread requests a "database thread" to perform the
> appropriate query and waits on a condition variable.
> + our "db thread" performs the necessary query and signals on that
> condition variable so that axis thread shall take the control over.
> + axis thread will create the om output and dispatch as usual.
>
> My doubts:
> + does this design has any apparent drawback when there is a flurry of
> back-to-back soap requests?
> + Is there a need to increase the axis thread pool size? If yes, how to
> do it?
> + As told before, the dump is seen during a particular scenario only.
> i.e: while trying to add a list of already existing users to my
> database. For each user there is a SOAP request generated and the
> requests are sequential, i.e the second request is sent only after the
> response for previous request is received.
> + if I increase the log level of my own module that uses axis2c, there
> is no dump. On increasing the log level, my module writes quite a bit of
> information to my log file.
>
> Your help is appreciated.
>
> Thanks & Regards,
> ~raghav
>
>
> -----Original Message-----
> From: Samisa Abeysinghe [mailto:samisa@wso2.com]
> Sent: Wednesday, March 05, 2008 7:25 PM
> To: Apache AXIS C User List
> Subject: Re: Core dumps in Axis libraries
>
> Raghavendra SM wrote:
>> Thanks Manjula,
>>
>> But thats not a viable option for us. We would still want to continue
>> using axis2C-1.0 for some more time.
>>
>> Are these dumps well known?
>
> No.
>
> Are you using the same service client instance from multiple threads?
>
> Samisa...
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: axis-c-user-unsubscribe@ws.apache.org
> For additional commands, e-mail: axis-c-user-help@ws.apache.org
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: axis-c-user-unsubscribe@ws.apache.org
> For additional commands, e-mail: axis-c-user-help@ws.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: axis-c-user-unsubscribe@ws.apache.org
For additional commands, e-mail: axis-c-user-help@ws.apache.org
Re: Core dumps in Axis libraries
Posted by Samisa Abeysinghe <sa...@wso2.com>.
Raghavendra SM wrote:
> + As told before, the dump is seen during a particular scenario only.
> i.e: while trying to add a list of already existing users to my
> database. For each user there is a SOAP request generated and the
> requests are sequential, i.e the second request is sent only after the
> response for previous request is received.
> + if I increase the log level of my own module that uses axis2c, there
> is no dump. On increasing the log level, my module writes quite a bit of
> information to my log file.
>
If it does not crash when more logging is happening, that indicates that
this is a thread related timing issue. When you have more logs verbose
logs written, the threads get slowed down, and the chances of race
conditions drop. Hence, this gives a clue to the cause of the problem,
in that, there is a some situation in the code where one thread is
trying to use some memory while the other thread running or something
along those lines.
It is hard to track down such a problem with the log alone. Either you
have to run it with gdb and analyze the back trace when it crashes, or
you have to use a profiler such as valgrind.
If you can provide the code that helps to recreate the problems, the
devs could be able to help.
Thanks,
Samisa...
> Your help is appreciated.
>
> Thanks & Regards,
> ~raghav
>
>
> -----Original Message-----
> From: Samisa Abeysinghe [mailto:samisa@wso2.com]
> Sent: Wednesday, March 05, 2008 7:25 PM
> To: Apache AXIS C User List
> Subject: Re: Core dumps in Axis libraries
>
> Raghavendra SM wrote:
>
>> Thanks Manjula,
>>
>> But thats not a viable option for us. We would still want to continue
>> using axis2C-1.0 for some more time.
>>
>> Are these dumps well known?
>>
>
> No.
>
> Are you using the same service client instance from multiple threads?
>
> Samisa...
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: axis-c-user-unsubscribe@ws.apache.org
> For additional commands, e-mail: axis-c-user-help@ws.apache.org
>
>
>
>
> ------------------------------------------------------------------------
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: axis-c-user-unsubscribe@ws.apache.org
> For additional commands, e-mail: axis-c-user-help@ws.apache.org
> ------------------------------------------------------------------------
>
> No virus found in this incoming message.
> Checked by AVG Free Edition.
> Version: 7.5.516 / Virus Database: 269.21.5/1314 - Release Date: 3/5/2008 6:38 PM
>
--
Samisa Abeysinghe
Software Architect; WSO2 Inc.
http://www.wso2.com/ - "Oxygenating the Web Service Platform."
---------------------------------------------------------------------
To unsubscribe, e-mail: axis-c-user-unsubscribe@ws.apache.org
For additional commands, e-mail: axis-c-user-help@ws.apache.org
RE: Core dumps in Axis libraries
Posted by Senaka Fernando <se...@wso2.com>.
Hi Simon, and Raghav,
Yes, valgrind can help you track the exact cause of the core dump.
Regards,
Senaka
> Hi,
> running valgrin tool on (httpd or axis_http_server) may help you to find
> the
> reason on the core (valgrind.org).
>
> Regards
> Simon
>
> Quoting Raghavendra SM <ra...@aylus.com>:
>
>> Thanks,
>>
>> As suggested, I tried using axis2c-1.3.0 both with and without
>> --enable-guthilla=yes. But I still see different dumps consistently.
>> Please find the attached file with back-traces.
>>
>> Are you using the same service client instance from multiple threads?
>> >>> We have a axis thread pool with default number of threads and a set
>> of our own threads(about 10, lets call them "database threads").
>> + On receiving a HTTP soap request, axis thread invokes the appropriate
>> function that is bound to our logic.
>> + Now, the axis thread requests a "database thread" to perform the
>> appropriate query and waits on a condition variable.
>> + our "db thread" performs the necessary query and signals on that
>> condition variable so that axis thread shall take the control over.
>> + axis thread will create the om output and dispatch as usual.
>>
>> My doubts:
>> + does this design has any apparent drawback when there is a flurry of
>> back-to-back soap requests?
>> + Is there a need to increase the axis thread pool size? If yes, how to
>> do it?
>> + As told before, the dump is seen during a particular scenario only.
>> i.e: while trying to add a list of already existing users to my
>> database. For each user there is a SOAP request generated and the
>> requests are sequential, i.e the second request is sent only after the
>> response for previous request is received.
>> + if I increase the log level of my own module that uses axis2c, there
>> is no dump. On increasing the log level, my module writes quite a bit of
>> information to my log file.
>>
>> Your help is appreciated.
>>
>> Thanks & Regards,
>> ~raghav
>>
>>
>> -----Original Message-----
>> From: Samisa Abeysinghe [mailto:samisa@wso2.com]
>> Sent: Wednesday, March 05, 2008 7:25 PM
>> To: Apache AXIS C User List
>> Subject: Re: Core dumps in Axis libraries
>>
>> Raghavendra SM wrote:
>> > Thanks Manjula,
>> >
>> > But thats not a viable option for us. We would still want to continue
>> > using axis2C-1.0 for some more time.
>> >
>> > Are these dumps well known?
>>
>> No.
>>
>> Are you using the same service client instance from multiple threads?
>>
>> Samisa...
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: axis-c-user-unsubscribe@ws.apache.org
>> For additional commands, e-mail: axis-c-user-help@ws.apache.org
>>
>>
>>
>>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: axis-c-user-unsubscribe@ws.apache.org
> For additional commands, e-mail: axis-c-user-help@ws.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: axis-c-user-unsubscribe@ws.apache.org
For additional commands, e-mail: axis-c-user-help@ws.apache.org
RE: Core dumps in Axis libraries
Posted by si...@free.fr.
Hi,
running valgrin tool on (httpd or axis_http_server) may help you to find the
reason on the core (valgrind.org).
Regards
Simon
Quoting Raghavendra SM <ra...@aylus.com>:
> Thanks,
>
> As suggested, I tried using axis2c-1.3.0 both with and without
> --enable-guthilla=yes. But I still see different dumps consistently.
> Please find the attached file with back-traces.
>
> Are you using the same service client instance from multiple threads?
> >>> We have a axis thread pool with default number of threads and a set
> of our own threads(about 10, lets call them "database threads").
> + On receiving a HTTP soap request, axis thread invokes the appropriate
> function that is bound to our logic.
> + Now, the axis thread requests a "database thread" to perform the
> appropriate query and waits on a condition variable.
> + our "db thread" performs the necessary query and signals on that
> condition variable so that axis thread shall take the control over.
> + axis thread will create the om output and dispatch as usual.
>
> My doubts:
> + does this design has any apparent drawback when there is a flurry of
> back-to-back soap requests?
> + Is there a need to increase the axis thread pool size? If yes, how to
> do it?
> + As told before, the dump is seen during a particular scenario only.
> i.e: while trying to add a list of already existing users to my
> database. For each user there is a SOAP request generated and the
> requests are sequential, i.e the second request is sent only after the
> response for previous request is received.
> + if I increase the log level of my own module that uses axis2c, there
> is no dump. On increasing the log level, my module writes quite a bit of
> information to my log file.
>
> Your help is appreciated.
>
> Thanks & Regards,
> ~raghav
>
>
> -----Original Message-----
> From: Samisa Abeysinghe [mailto:samisa@wso2.com]
> Sent: Wednesday, March 05, 2008 7:25 PM
> To: Apache AXIS C User List
> Subject: Re: Core dumps in Axis libraries
>
> Raghavendra SM wrote:
> > Thanks Manjula,
> >
> > But thats not a viable option for us. We would still want to continue
> > using axis2C-1.0 for some more time.
> >
> > Are these dumps well known?
>
> No.
>
> Are you using the same service client instance from multiple threads?
>
> Samisa...
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: axis-c-user-unsubscribe@ws.apache.org
> For additional commands, e-mail: axis-c-user-help@ws.apache.org
>
>
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: axis-c-user-unsubscribe@ws.apache.org
For additional commands, e-mail: axis-c-user-help@ws.apache.org
RE: Core dumps in Axis libraries
Posted by Raghavendra SM <ra...@aylus.com>.
Thanks,
As suggested, I tried using axis2c-1.3.0 both with and without
--enable-guthilla=yes. But I still see different dumps consistently.
Please find the attached file with back-traces.
Are you using the same service client instance from multiple threads?
>>> We have a axis thread pool with default number of threads and a set
of our own threads(about 10, lets call them "database threads").
+ On receiving a HTTP soap request, axis thread invokes the appropriate
function that is bound to our logic.
+ Now, the axis thread requests a "database thread" to perform the
appropriate query and waits on a condition variable.
+ our "db thread" performs the necessary query and signals on that
condition variable so that axis thread shall take the control over.
+ axis thread will create the om output and dispatch as usual.
My doubts:
+ does this design has any apparent drawback when there is a flurry of
back-to-back soap requests?
+ Is there a need to increase the axis thread pool size? If yes, how to
do it?
+ As told before, the dump is seen during a particular scenario only.
i.e: while trying to add a list of already existing users to my
database. For each user there is a SOAP request generated and the
requests are sequential, i.e the second request is sent only after the
response for previous request is received.
+ if I increase the log level of my own module that uses axis2c, there
is no dump. On increasing the log level, my module writes quite a bit of
information to my log file.
Your help is appreciated.
Thanks & Regards,
~raghav
-----Original Message-----
From: Samisa Abeysinghe [mailto:samisa@wso2.com]
Sent: Wednesday, March 05, 2008 7:25 PM
To: Apache AXIS C User List
Subject: Re: Core dumps in Axis libraries
Raghavendra SM wrote:
> Thanks Manjula,
>
> But thats not a viable option for us. We would still want to continue
> using axis2C-1.0 for some more time.
>
> Are these dumps well known?
No.
Are you using the same service client instance from multiple threads?
Samisa...
---------------------------------------------------------------------
To unsubscribe, e-mail: axis-c-user-unsubscribe@ws.apache.org
For additional commands, e-mail: axis-c-user-help@ws.apache.org
Re: Core dumps in Axis libraries
Posted by Samisa Abeysinghe <sa...@wso2.com>.
Raghavendra SM wrote:
> Thanks Manjula,
>
> But thats not a viable option for us. We would still want to continue
> using axis2C-1.0 for some more time.
>
> Are these dumps well known?
No.
Are you using the same service client instance from multiple threads?
Samisa...
---------------------------------------------------------------------
To unsubscribe, e-mail: axis-c-user-unsubscribe@ws.apache.org
For additional commands, e-mail: axis-c-user-help@ws.apache.org
Re: Core dumps in Axis libraries
Posted by Samisa Abeysinghe <sa...@wso2.com>.
Raghavendra SM wrote:
> Thanks Manjula,
>
> But thats not a viable option for us. We would still want to continue
> using axis2C-1.0 for some more time.
>
Well, the community in general is unable to support this scenario.
Almost all of the devs are working with the latest release. And over
time, since the release of 1.0, we have done a great deal of bug fixes.
It is always advisable to switch to the latest release.
Samisa...
---------------------------------------------------------------------
To unsubscribe, e-mail: axis-c-user-unsubscribe@ws.apache.org
For additional commands, e-mail: axis-c-user-help@ws.apache.org
Re: Core dumps in Axis libraries
Posted by Dinesh Premalal <xy...@gmail.com>.
Hi Raghavendra,
"Raghavendra SM" <ra...@aylus.com> writes:
> Thanks Manjula,
>
> But thats not a viable option for us. We would still want to continue
> using axis2C-1.0 for some more time.
>
> Are these dumps well known? And have they been specifically fixed in
> axis2c-1.3?
Hmm..For me it seems most of dumps occurred from libxml2, could you
please try with Guththila parser.
to enable guththila , you need to use --enable-guththila option. i.e
./configure --enable-guththila=yes [other options]
thanks,
Dinesh
--
http://nethu.org
---------------------------------------------------------------------
To unsubscribe, e-mail: axis-c-user-unsubscribe@ws.apache.org
For additional commands, e-mail: axis-c-user-help@ws.apache.org
RE: Core dumps in Axis libraries
Posted by Raghavendra SM <ra...@aylus.com>.
Thanks Manjula,
But thats not a viable option for us. We would still want to continue
using axis2C-1.0 for some more time.
Are these dumps well known? And have they been specifically fixed in
axis2c-1.3?
Regards,
~raghav
-----Original Message-----
From: Manjula Peiris [mailto:manjula@wso2.com]
Sent: Wednesday, March 05, 2008 6:05 PM
To: Apache AXIS C User List
Subject: Re: Core dumps in Axis libraries
Hi raghav,
Axis2/C 1.3 was recently released. There are lots of new features and
bug fixes in that release. Please try that.
http://ws.apache.org/axis2/c/download.cgi
-Thanks,
Manjula.
On Wed, 2008-03-05 at 17:34 +0530, Raghavendra SM wrote:
> All,
>
> For a particular scenario, our module core dumps in some axis library
> functions consistently but with different back traces. I'm attaching
the
> dumps.log file that has as many as 4 different back-traces produced
for
> the same scenario.
> Also, please find attached the log that has axis generated log for
both
> successful and erroneous times, axis.log.
>
> To give you all a brief background,
> + we are using axis2c-1.0.0
> + the core dump appears during a particular scenario only. Eg: while
> trying to add a list of already existing users to my database. For
each
> user there is a SOAP request generated and the requests are
sequential,
> i.e the second request is sent only after the response for previous
> request is received.
> + if I increase the log level of my own module that uses axis2c, there
> is no dump. On increasing the log level, my module writes quite a bit
of
> information to my log file.
>
> Since the back trace is pointing to the libraries(for which there is
no
> source code given), how can this be debugged and fixed?
>
> Any inputs will be of great help. Please reply.
>
>
> Regards,
> ~raghav
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: axis-c-user-unsubscribe@ws.apache.org
> For additional commands, e-mail: axis-c-user-help@ws.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: axis-c-user-unsubscribe@ws.apache.org
For additional commands, e-mail: axis-c-user-help@ws.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: axis-c-user-unsubscribe@ws.apache.org
For additional commands, e-mail: axis-c-user-help@ws.apache.org
Re: Core dumps in Axis libraries
Posted by Manjula Peiris <ma...@wso2.com>.
Hi raghav,
Axis2/C 1.3 was recently released. There are lots of new features and
bug fixes in that release. Please try that.
http://ws.apache.org/axis2/c/download.cgi
-Thanks,
Manjula.
On Wed, 2008-03-05 at 17:34 +0530, Raghavendra SM wrote:
> All,
>
> For a particular scenario, our module core dumps in some axis library
> functions consistently but with different back traces. I'm attaching the
> dumps.log file that has as many as 4 different back-traces produced for
> the same scenario.
> Also, please find attached the log that has axis generated log for both
> successful and erroneous times, axis.log.
>
> To give you all a brief background,
> + we are using axis2c-1.0.0
> + the core dump appears during a particular scenario only. Eg: while
> trying to add a list of already existing users to my database. For each
> user there is a SOAP request generated and the requests are sequential,
> i.e the second request is sent only after the response for previous
> request is received.
> + if I increase the log level of my own module that uses axis2c, there
> is no dump. On increasing the log level, my module writes quite a bit of
> information to my log file.
>
> Since the back trace is pointing to the libraries(for which there is no
> source code given), how can this be debugged and fixed?
>
> Any inputs will be of great help. Please reply.
>
>
> Regards,
> ~raghav
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: axis-c-user-unsubscribe@ws.apache.org
> For additional commands, e-mail: axis-c-user-help@ws.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: axis-c-user-unsubscribe@ws.apache.org
For additional commands, e-mail: axis-c-user-help@ws.apache.org