You are viewing a plain text version of this content. The canonical link for it is here.
Posted to c-dev@axis.apache.org by Senaka Fernando <se...@wso2.com> on 2008/03/10 12:39:17 UTC
Re: Solution to Core Dump Issue Raised in User List [was: Re: Core
dumps in Axis libraries]
> Senaka Fernando wrote:
>> Hi Samisa,
>>
>> I fixed two bugs regarding stream handling inside
>> simple_http_svr_conn.c.
>> This fixes an issue of this nature.
>>
>> Say, we have two server threads that try to serve requests offered to
>> the
>> axis2 server. Now, according to what is intended, one must serve and the
>> other must wait until it can serve. The previous implementation caused
>> this to fail with a segfault. But, with the fix, it will rather not
>> crash
>> but, loop until it gets a chance to serve the request.
>>
>
> Did you test before the fix and did it seg fault for you? Can you please
> explain the scenario under which it seg faulted? If it is a problem, I
> too should be able to re-create it.
>
> And how did you solve the problem? By adding a sleep?
I made the accept(2) call fail. By passing a -1 to it. Once that failed,
there were two places that segfaulted which I already fixed, few minutes
ago. No, I didn't add a sleep(). I simply prevented the segfault by adding
necessary is_null checks before calling methods. Then, the server just
kept on spawning threads and freeing them as it can't serve the request.
This is the expected behaviour in http_svr_thread.c I believe.
In a real world scenario, a situation similar to a -1 passed to accept(2)
will be temporal. So a respawned thread down the line should server the
request without any problem.
Passing -1 to accept was done by,
Index: util/src/network_handler.c
===================================================================
--- util/src/network_handler.c (revision 634578)
+++ util/src/network_handler.c (working copy)
@@ -221,7 +221,7 @@
AXIS2_ENV_CHECK(env, AXIS2_CRITICAL_FAILURE);
cli_len = sizeof(cli_addr);
- cli_socket = accept(svr_socket, (struct sockaddr *) &cli_addr,
&cli_len);
+ cli_socket = accept(-1, (struct sockaddr *) &cli_addr, &cli_len);
if (cli_socket < 0)
AXIS2_LOG_ERROR(env->log, AXIS2_LOG_SI,
"[Axis2][network_handler] Socket accept \
---
I'm not sure whether this is what really lead to the core dump, or was it
something else.
This was all that I could try on based on the less informative
valgrind.log that was sent. :(...
Also i did test 1000 back-to-back requests and they all were served
without any issues.
Regards,
Senaka
>
> Samisa...
>
>> Could this solve the core dump issue?
>>
>> Regards,
>> Senaka
>>
>>
>>> Raghavendra SM wrote:
>>>
>>>> As expected, when I put a delay of about 50 milliseconds between the
>>>> requests there is no dump.
>>>>
>>>> But we would still like to know the following,
>>>> + Isn't Axis capable of handling such huge bombardment of requests (of
>>>> the order of 1000's) one after another without any delay between the
>>>> requests? Has anyone tried such client before?
>>>>
>>>>
>>> Yes it is supposed to. We have done performance testing that proved
>>> that
>>> Axis2/C can serve more than 300 rps.
>>>
>>>
>>>> + Can we make out anything out of the valgrind report or the axis logs
>>>> that were attached?
>>>>
>>>>
>>> Unfortunately not. When valgrind runs, it makes the system slow, that
>>> prevents the system crashing in situations like that you are
>>> experiencing. To solve this kind of a problem, it is a must that you
>>> provide us with some sample code that can be used to reproduce the
>>> problem. Without having a way to re-produce the situation, there is no
>>> way to fix this problem - it could well be a problem in Axis2/C as well
>>> as it could also be in the code that you use.
>>>
>>> Samisa...
>>>
>>>
>>>
>>>> Regards,
>>>> ~raghav
>>>>
>>>>
>>>> -----Original Message-----
>>>> From: Senaka Fernando [mailto:senaka@wso2.com]
>>>> Sent: Thursday, March 06, 2008 4:22 PM
>>>> To: Apache AXIS C User List
>>>> Subject: RE: Core dumps in Axis libraries
>>>>
>>>> Hi Raghav,
>>>>
>>>> Can you please try this with --enable-guthilla=yes, and send the
>>>> dump.log,
>>>> and also the appropriate log_files generated by the server/client.
>>>>
>>>> There also can be issues with your "condition variable" being used in
>>>> multiple threads.
>>>>
>>>> It also seems that if you pause between your requests you might get
>>>> this
>>>> solved. According to what you say, when you increase the log_level,
>>>> obviously you waste some time there which might be the solution
>>>> rather.
>>>> You can use the AXIS2_SLEEP() and AXIS2_USLEEP() inside Axis2/C to
>>>> pause
>>>> between requests. You can get an idea on how to use these functions by
>>>> taking a look at the Axis2/C user_guide/client samples.
>>>>
>>>> Regards,
>>>> Senaka
>>>>
>>>>
>>>>
>>>>> Thanks,
>>>>>
>>>>> As suggested, I tried using axis2c-1.3.0 both with and without
>>>>> --enable-guthilla=yes. But I still see different dumps consistently.
>>>>> Please find the attached file with back-traces.
>>>>>
>>>>> Are you using the same service client instance from multiple threads?
>>>>>
>>>>>
>>>>>>>> We have a axis thread pool with default number of threads and a
>>>>>>>> set
>>>>>>>>
>>>>>>>>
>>>>> of our own threads(about 10, lets call them "database threads").
>>>>> + On receiving a HTTP soap request, axis thread invokes the
>>>>>
>>>>>
>>>> appropriate
>>>>
>>>>
>>>>> function that is bound to our logic.
>>>>> + Now, the axis thread requests a "database thread" to perform the
>>>>> appropriate query and waits on a condition variable.
>>>>> + our "db thread" performs the necessary query and signals on that
>>>>> condition variable so that axis thread shall take the control over.
>>>>> + axis thread will create the om output and dispatch as usual.
>>>>>
>>>>> My doubts:
>>>>> + does this design has any apparent drawback when there is a flurry
>>>>> of
>>>>> back-to-back soap requests?
>>>>> + Is there a need to increase the axis thread pool size? If yes, how
>>>>>
>>>>>
>>>> to
>>>>
>>>>
>>>>> do it?
>>>>> + As told before, the dump is seen during a particular scenario only.
>>>>> i.e: while trying to add a list of already existing users to my
>>>>> database. For each user there is a SOAP request generated and the
>>>>> requests are sequential, i.e the second request is sent only after
>>>>> the
>>>>> response for previous request is received.
>>>>> + if I increase the log level of my own module that uses axis2c,
>>>>> there
>>>>> is no dump. On increasing the log level, my module writes quite a bit
>>>>>
>>>>>
>>>> of
>>>>
>>>>
>>>>> information to my log file.
>>>>>
>>>>> Your help is appreciated.
>>>>>
>>>>> Thanks & Regards,
>>>>> ~raghav
>>>>>
>>>>>
>>>>> -----Original Message-----
>>>>> From: Samisa Abeysinghe [mailto:samisa@wso2.com]
>>>>> Sent: Wednesday, March 05, 2008 7:25 PM
>>>>> To: Apache AXIS C User List
>>>>> Subject: Re: Core dumps in Axis libraries
>>>>>
>>>>> Raghavendra SM wrote:
>>>>>
>>>>>
>>>>>> Thanks Manjula,
>>>>>>
>>>>>> But thats not a viable option for us. We would still want to
>>>>>> continue
>>>>>> using axis2C-1.0 for some more time.
>>>>>>
>>>>>> Are these dumps well known?
>>>>>>
>>>>>>
>>>>> No.
>>>>>
>>>>> Are you using the same service client instance from multiple threads?
>>>>>
>>>>> Samisa...
>>>>>
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: axis-c-user-unsubscribe@ws.apache.org
>>>>> For additional commands, e-mail: axis-c-user-help@ws.apache.org
>>>>>
>>>>>
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: axis-c-user-unsubscribe@ws.apache.org
>>>>> For additional commands, e-mail: axis-c-user-help@ws.apache.org
>>>>>
>>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: axis-c-user-unsubscribe@ws.apache.org
>>>> For additional commands, e-mail: axis-c-user-help@ws.apache.org
>>>>
>>>>
>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: axis-c-user-unsubscribe@ws.apache.org
>>>> For additional commands, e-mail: axis-c-user-help@ws.apache.org
>>>>
>>>>
>>>>
>>>>
>>>>
>>> --
>>> Samisa Abeysinghe
>>> Software Architect; WSO2 Inc.
>>>
>>> http://www.wso2.com/ - "Oxygenating the Web Service Platform."
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: axis-c-user-unsubscribe@ws.apache.org
>>> For additional commands, e-mail: axis-c-user-help@ws.apache.org
>>>
>>>
>>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: axis-c-dev-unsubscribe@ws.apache.org
>> For additional commands, e-mail: axis-c-dev-help@ws.apache.org
>>
>>
>>
>>
>
>
> --
> Samisa Abeysinghe
> Software Architect; WSO2 Inc.
>
> http://www.wso2.com/ - "Oxygenating the Web Service Platform."
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: axis-c-dev-unsubscribe@ws.apache.org
> For additional commands, e-mail: axis-c-dev-help@ws.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: axis-c-dev-unsubscribe@ws.apache.org
For additional commands, e-mail: axis-c-dev-help@ws.apache.org
Re: Solution to Core Dump Issue Raised in User List [was: Re: Core
dumps in Axis libraries]
Posted by Senaka Fernando <se...@wso2.com>.
> There are two sides to this problem. One is that, if it is failing for
> some scenario, we should be able to re-produce it. So without some code
> to re-produce the fault, we cannot come to a conclusion that it fails.
> If accepts fails, and it crashes, then there needs to be someuser
> scenario that makes this happen. If we cannot find a user scenario that
> makes accept to fail, then even though the bug is there, it will never
> be exposed.
>
> Anyway we can ask the user to try with your fixes for accept failure
> case and test to see if the problem persists.
Yes, that can be done. Actually as I said before, this is just a wild
guess that an issue in accept(2) could've created a segfault. Reason is
that there is no way to figure out what could be the reason from the
valgrind log.
Of course if this is a SIGTERM inside accept, he will have to use a signal
handler, there was an example in the Fast CGI Issue fix that I sent, and
return the appropriate error to the rest of the source following
accept(2). Then our code will pick it up correctly from there onwards with
this fix in place.
Also, adding a signal handler is a user-specific thing which we need not
attend to because, AFAIK, Axis2/C implementation will not cause a SIGTERM
inside accept(2) and that should be through the intervention of some
external source.
Regards,
Senaka
>
> Samisa...
>
> Senaka Fernando wrote:
>> Hi again Samisa,
>>
>> And, also, I don't have any similar database accessing logic etc.
>> Therefore, it is hard to conclude without seeing the actual code. Also,
>> the user seems to be using a modified version of "http_server_main.c",
>> it
>> is actually a cpp file if you go through that portion which I
>> highlighted
>> in his log. May be some issues due to not handling concurrency properly
>> could also lead him into trouble.
>>
>> However, what I can say is that there was a bug if accept(2) was to
>> fail,
>> according to the tests I did. And, according to literature available
>> online that can fail. Plus, we do have a mechanism of retrying, if it
>> does
>> fail which did not work due to the segfault.
>>
>> Regards,
>> Senaka
>>
>>
>>>> Were you able to re-create the problem without passing -1 to accept?
>>>>
>>> No I was not able to make accept fail without passing -1. Added a
>>> printf
>>> there to print any failure in accept and it didn't. However, ran into
>>> the
>>> segfault issue when accept actually failed, which I had to do manually.
>>>
>>> I did test with 1000 back-2-back mtom requests in 5 terminals, and also
>>> 1000 repeated calls to an echo client that sends 1000 back-2-back
>>> requests, and in 4 terminals.
>>>
>>> None of these cases failed. There were occasional hangs but still it
>>> didn't result in any failure on either client or server side.
>>>
>>> However, I didn't have any way to test multiple requests made to a
>>> single
>>> port from multiple machines. That may be a different scenario.
>>>
>>> Regards,
>>> Senaka
>>>
>>>
>>>> Samisa...
>>>>
>>>> Senaka Fernando wrote:
>>>>
>>>>>> Senaka Fernando wrote:
>>>>>>
>>>>>>
>>>>>>> Hi Samisa,
>>>>>>>
>>>>>>> I fixed two bugs regarding stream handling inside
>>>>>>> simple_http_svr_conn.c.
>>>>>>> This fixes an issue of this nature.
>>>>>>>
>>>>>>> Say, we have two server threads that try to serve requests offered
>>>>>>> to
>>>>>>> the
>>>>>>> axis2 server. Now, according to what is intended, one must serve
>>>>>>> and
>>>>>>> the
>>>>>>> other must wait until it can serve. The previous implementation
>>>>>>> caused
>>>>>>> this to fail with a segfault. But, with the fix, it will rather not
>>>>>>> crash
>>>>>>> but, loop until it gets a chance to serve the request.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>> Did you test before the fix and did it seg fault for you? Can you
>>>>>> please
>>>>>> explain the scenario under which it seg faulted? If it is a problem,
>>>>>> I
>>>>>> too should be able to re-create it.
>>>>>>
>>>>>> And how did you solve the problem? By adding a sleep?
>>>>>>
>>>>>>
>>>>> I made the accept(2) call fail. By passing a -1 to it. Once that
>>>>> failed,
>>>>> there were two places that segfaulted which I already fixed, few
>>>>> minutes
>>>>> ago. No, I didn't add a sleep(). I simply prevented the segfault by
>>>>> adding
>>>>> necessary is_null checks before calling methods. Then, the server
>>>>> just
>>>>> kept on spawning threads and freeing them as it can't serve the
>>>>> request.
>>>>> This is the expected behaviour in http_svr_thread.c I believe.
>>>>>
>>>>> In a real world scenario, a situation similar to a -1 passed to
>>>>> accept(2)
>>>>> will be temporal. So a respawned thread down the line should server
>>>>> the
>>>>> request without any problem.
>>>>>
>>>>> Passing -1 to accept was done by,
>>>>>
>>>>> Index: util/src/network_handler.c
>>>>> ===================================================================
>>>>> --- util/src/network_handler.c (revision 634578)
>>>>> +++ util/src/network_handler.c (working copy)
>>>>> @@ -221,7 +221,7 @@
>>>>> AXIS2_ENV_CHECK(env, AXIS2_CRITICAL_FAILURE);
>>>>>
>>>>> cli_len = sizeof(cli_addr);
>>>>> - cli_socket = accept(svr_socket, (struct sockaddr *) &cli_addr,
>>>>> &cli_len);
>>>>> + cli_socket = accept(-1, (struct sockaddr *) &cli_addr,
>>>>> &cli_len);
>>>>> if (cli_socket < 0)
>>>>> AXIS2_LOG_ERROR(env->log, AXIS2_LOG_SI,
>>>>> "[Axis2][network_handler] Socket accept \
>>>>> ---
>>>>>
>>>>> I'm not sure whether this is what really lead to the core dump, or
>>>>> was
>>>>> it
>>>>> something else.
>>>>>
>>>>> This was all that I could try on based on the less informative
>>>>> valgrind.log that was sent. :(...
>>>>>
>>>>> Also i did test 1000 back-to-back requests and they all were served
>>>>> without any issues.
>>>>>
>>>>> Regards,
>>>>> Senaka
>>>>>
>>>>>
>>>>>
>>>>>> Samisa...
>>>>>>
>>>>>>
>>>>>>
>>>>>>> Could this solve the core dump issue?
>>>>>>>
>>>>>>> Regards,
>>>>>>> Senaka
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>> Raghavendra SM wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>> As expected, when I put a delay of about 50 milliseconds between
>>>>>>>>> the
>>>>>>>>> requests there is no dump.
>>>>>>>>>
>>>>>>>>> But we would still like to know the following,
>>>>>>>>> + Isn't Axis capable of handling such huge bombardment of
>>>>>>>>> requests
>>>>>>>>> (of
>>>>>>>>> the order of 1000's) one after another without any delay between
>>>>>>>>> the
>>>>>>>>> requests? Has anyone tried such client before?
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>> Yes it is supposed to. We have done performance testing that
>>>>>>>> proved
>>>>>>>> that
>>>>>>>> Axis2/C can serve more than 300 rps.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>> + Can we make out anything out of the valgrind report or the axis
>>>>>>>>> logs
>>>>>>>>> that were attached?
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>> Unfortunately not. When valgrind runs, it makes the system slow,
>>>>>>>> that
>>>>>>>> prevents the system crashing in situations like that you are
>>>>>>>> experiencing. To solve this kind of a problem, it is a must that
>>>>>>>> you
>>>>>>>> provide us with some sample code that can be used to reproduce the
>>>>>>>> problem. Without having a way to re-produce the situation, there
>>>>>>>> is
>>>>>>>> no
>>>>>>>> way to fix this problem - it could well be a problem in Axis2/C as
>>>>>>>> well
>>>>>>>> as it could also be in the code that you use.
>>>>>>>>
>>>>>>>> Samisa...
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>> Regards,
>>>>>>>>> ~raghav
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> -----Original Message-----
>>>>>>>>> From: Senaka Fernando [mailto:senaka@wso2.com]
>>>>>>>>> Sent: Thursday, March 06, 2008 4:22 PM
>>>>>>>>> To: Apache AXIS C User List
>>>>>>>>> Subject: RE: Core dumps in Axis libraries
>>>>>>>>>
>>>>>>>>> Hi Raghav,
>>>>>>>>>
>>>>>>>>> Can you please try this with --enable-guthilla=yes, and send the
>>>>>>>>> dump.log,
>>>>>>>>> and also the appropriate log_files generated by the
>>>>>>>>> server/client.
>>>>>>>>>
>>>>>>>>> There also can be issues with your "condition variable" being
>>>>>>>>> used
>>>>>>>>> in
>>>>>>>>> multiple threads.
>>>>>>>>>
>>>>>>>>> It also seems that if you pause between your requests you might
>>>>>>>>> get
>>>>>>>>> this
>>>>>>>>> solved. According to what you say, when you increase the
>>>>>>>>> log_level,
>>>>>>>>> obviously you waste some time there which might be the solution
>>>>>>>>> rather.
>>>>>>>>> You can use the AXIS2_SLEEP() and AXIS2_USLEEP() inside Axis2/C
>>>>>>>>> to
>>>>>>>>> pause
>>>>>>>>> between requests. You can get an idea on how to use these
>>>>>>>>> functions
>>>>>>>>> by
>>>>>>>>> taking a look at the Axis2/C user_guide/client samples.
>>>>>>>>>
>>>>>>>>> Regards,
>>>>>>>>> Senaka
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>>
>>>>>>>>>> As suggested, I tried using axis2c-1.3.0 both with and without
>>>>>>>>>> --enable-guthilla=yes. But I still see different dumps
>>>>>>>>>> consistently.
>>>>>>>>>> Please find the attached file with back-traces.
>>>>>>>>>>
>>>>>>>>>> Are you using the same service client instance from multiple
>>>>>>>>>> threads?
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>>>> We have a axis thread pool with default number of threads and
>>>>>>>>>>>>> a
>>>>>>>>>>>>> set
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>> of our own threads(about 10, lets call them "database threads").
>>>>>>>>>> + On receiving a HTTP soap request, axis thread invokes the
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>> appropriate
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> function that is bound to our logic.
>>>>>>>>>> + Now, the axis thread requests a "database thread" to perform
>>>>>>>>>> the
>>>>>>>>>> appropriate query and waits on a condition variable.
>>>>>>>>>> + our "db thread" performs the necessary query and signals on
>>>>>>>>>> that
>>>>>>>>>> condition variable so that axis thread shall take the control
>>>>>>>>>> over.
>>>>>>>>>> + axis thread will create the om output and dispatch as usual.
>>>>>>>>>>
>>>>>>>>>> My doubts:
>>>>>>>>>> + does this design has any apparent drawback when there is a
>>>>>>>>>> flurry
>>>>>>>>>> of
>>>>>>>>>> back-to-back soap requests?
>>>>>>>>>> + Is there a need to increase the axis thread pool size? If yes,
>>>>>>>>>> how
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>> to
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> do it?
>>>>>>>>>> + As told before, the dump is seen during a particular scenario
>>>>>>>>>> only.
>>>>>>>>>> i.e: while trying to add a list of already existing users to my
>>>>>>>>>> database. For each user there is a SOAP request generated and
>>>>>>>>>> the
>>>>>>>>>> requests are sequential, i.e the second request is sent only
>>>>>>>>>> after
>>>>>>>>>> the
>>>>>>>>>> response for previous request is received.
>>>>>>>>>> + if I increase the log level of my own module that uses axis2c,
>>>>>>>>>> there
>>>>>>>>>> is no dump. On increasing the log level, my module writes quite
>>>>>>>>>> a
>>>>>>>>>> bit
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>> of
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> information to my log file.
>>>>>>>>>>
>>>>>>>>>> Your help is appreciated.
>>>>>>>>>>
>>>>>>>>>> Thanks & Regards,
>>>>>>>>>> ~raghav
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> -----Original Message-----
>>>>>>>>>> From: Samisa Abeysinghe [mailto:samisa@wso2.com]
>>>>>>>>>> Sent: Wednesday, March 05, 2008 7:25 PM
>>>>>>>>>> To: Apache AXIS C User List
>>>>>>>>>> Subject: Re: Core dumps in Axis libraries
>>>>>>>>>>
>>>>>>>>>> Raghavendra SM wrote:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> Thanks Manjula,
>>>>>>>>>>>
>>>>>>>>>>> But thats not a viable option for us. We would still want to
>>>>>>>>>>> continue
>>>>>>>>>>> using axis2C-1.0 for some more time.
>>>>>>>>>>>
>>>>>>>>>>> Are these dumps well known?
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>> No.
>>>>>>>>>>
>>>>>>>>>> Are you using the same service client instance from multiple
>>>>>>>>>> threads?
>>>>>>>>>>
>>>>>>>>>> Samisa...
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>> To unsubscribe, e-mail: axis-c-user-unsubscribe@ws.apache.org
>>>>>>>>>> For additional commands, e-mail: axis-c-user-help@ws.apache.org
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>> To unsubscribe, e-mail: axis-c-user-unsubscribe@ws.apache.org
>>>>>>>>>> For additional commands, e-mail: axis-c-user-help@ws.apache.org
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>> To unsubscribe, e-mail: axis-c-user-unsubscribe@ws.apache.org
>>>>>>>>> For additional commands, e-mail: axis-c-user-help@ws.apache.org
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>> To unsubscribe, e-mail: axis-c-user-unsubscribe@ws.apache.org
>>>>>>>>> For additional commands, e-mail: axis-c-user-help@ws.apache.org
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>> --
>>>>>>>> Samisa Abeysinghe
>>>>>>>> Software Architect; WSO2 Inc.
>>>>>>>>
>>>>>>>> http://www.wso2.com/ - "Oxygenating the Web Service Platform."
>>>>>>>>
>>>>>>>>
>>>>>>>> ---------------------------------------------------------------------
>>>>>>>> To unsubscribe, e-mail: axis-c-user-unsubscribe@ws.apache.org
>>>>>>>> For additional commands, e-mail: axis-c-user-help@ws.apache.org
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>> ---------------------------------------------------------------------
>>>>>>> To unsubscribe, e-mail: axis-c-dev-unsubscribe@ws.apache.org
>>>>>>> For additional commands, e-mail: axis-c-dev-help@ws.apache.org
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>> --
>>>>>> Samisa Abeysinghe
>>>>>> Software Architect; WSO2 Inc.
>>>>>>
>>>>>> http://www.wso2.com/ - "Oxygenating the Web Service Platform."
>>>>>>
>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: axis-c-dev-unsubscribe@ws.apache.org
>>>>>> For additional commands, e-mail: axis-c-dev-help@ws.apache.org
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: axis-c-dev-unsubscribe@ws.apache.org
>>>>> For additional commands, e-mail: axis-c-dev-help@ws.apache.org
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>> --
>>>> Samisa Abeysinghe
>>>> Software Architect; WSO2 Inc.
>>>>
>>>> http://www.wso2.com/ - "Oxygenating the Web Service Platform."
>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: axis-c-dev-unsubscribe@ws.apache.org
>>>> For additional commands, e-mail: axis-c-dev-help@ws.apache.org
>>>>
>>>>
>>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: axis-c-dev-unsubscribe@ws.apache.org
>>> For additional commands, e-mail: axis-c-dev-help@ws.apache.org
>>>
>>>
>>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: axis-c-dev-unsubscribe@ws.apache.org
>> For additional commands, e-mail: axis-c-dev-help@ws.apache.org
>>
>>
>>
>>
>
>
> --
> Samisa Abeysinghe
> Software Architect; WSO2 Inc.
>
> http://www.wso2.com/ - "Oxygenating the Web Service Platform."
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: axis-c-dev-unsubscribe@ws.apache.org
> For additional commands, e-mail: axis-c-dev-help@ws.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: axis-c-dev-unsubscribe@ws.apache.org
For additional commands, e-mail: axis-c-dev-help@ws.apache.org
Re: Solution to Core Dump Issue Raised in User List [was: Re: Core
dumps in Axis libraries]
Posted by Samisa Abeysinghe <sa...@wso2.com>.
There are two sides to this problem. One is that, if it is failing for
some scenario, we should be able to re-produce it. So without some code
to re-produce the fault, we cannot come to a conclusion that it fails.
If accepts fails, and it crashes, then there needs to be someuser
scenario that makes this happen. If we cannot find a user scenario that
makes accept to fail, then even though the bug is there, it will never
be exposed.
Anyway we can ask the user to try with your fixes for accept failure
case and test to see if the problem persists.
Samisa...
Senaka Fernando wrote:
> Hi again Samisa,
>
> And, also, I don't have any similar database accessing logic etc.
> Therefore, it is hard to conclude without seeing the actual code. Also,
> the user seems to be using a modified version of "http_server_main.c", it
> is actually a cpp file if you go through that portion which I highlighted
> in his log. May be some issues due to not handling concurrency properly
> could also lead him into trouble.
>
> However, what I can say is that there was a bug if accept(2) was to fail,
> according to the tests I did. And, according to literature available
> online that can fail. Plus, we do have a mechanism of retrying, if it does
> fail which did not work due to the segfault.
>
> Regards,
> Senaka
>
>
>>> Were you able to re-create the problem without passing -1 to accept?
>>>
>> No I was not able to make accept fail without passing -1. Added a printf
>> there to print any failure in accept and it didn't. However, ran into the
>> segfault issue when accept actually failed, which I had to do manually.
>>
>> I did test with 1000 back-2-back mtom requests in 5 terminals, and also
>> 1000 repeated calls to an echo client that sends 1000 back-2-back
>> requests, and in 4 terminals.
>>
>> None of these cases failed. There were occasional hangs but still it
>> didn't result in any failure on either client or server side.
>>
>> However, I didn't have any way to test multiple requests made to a single
>> port from multiple machines. That may be a different scenario.
>>
>> Regards,
>> Senaka
>>
>>
>>> Samisa...
>>>
>>> Senaka Fernando wrote:
>>>
>>>>> Senaka Fernando wrote:
>>>>>
>>>>>
>>>>>> Hi Samisa,
>>>>>>
>>>>>> I fixed two bugs regarding stream handling inside
>>>>>> simple_http_svr_conn.c.
>>>>>> This fixes an issue of this nature.
>>>>>>
>>>>>> Say, we have two server threads that try to serve requests offered to
>>>>>> the
>>>>>> axis2 server. Now, according to what is intended, one must serve and
>>>>>> the
>>>>>> other must wait until it can serve. The previous implementation
>>>>>> caused
>>>>>> this to fail with a segfault. But, with the fix, it will rather not
>>>>>> crash
>>>>>> but, loop until it gets a chance to serve the request.
>>>>>>
>>>>>>
>>>>>>
>>>>> Did you test before the fix and did it seg fault for you? Can you
>>>>> please
>>>>> explain the scenario under which it seg faulted? If it is a problem, I
>>>>> too should be able to re-create it.
>>>>>
>>>>> And how did you solve the problem? By adding a sleep?
>>>>>
>>>>>
>>>> I made the accept(2) call fail. By passing a -1 to it. Once that
>>>> failed,
>>>> there were two places that segfaulted which I already fixed, few
>>>> minutes
>>>> ago. No, I didn't add a sleep(). I simply prevented the segfault by
>>>> adding
>>>> necessary is_null checks before calling methods. Then, the server just
>>>> kept on spawning threads and freeing them as it can't serve the
>>>> request.
>>>> This is the expected behaviour in http_svr_thread.c I believe.
>>>>
>>>> In a real world scenario, a situation similar to a -1 passed to
>>>> accept(2)
>>>> will be temporal. So a respawned thread down the line should server the
>>>> request without any problem.
>>>>
>>>> Passing -1 to accept was done by,
>>>>
>>>> Index: util/src/network_handler.c
>>>> ===================================================================
>>>> --- util/src/network_handler.c (revision 634578)
>>>> +++ util/src/network_handler.c (working copy)
>>>> @@ -221,7 +221,7 @@
>>>> AXIS2_ENV_CHECK(env, AXIS2_CRITICAL_FAILURE);
>>>>
>>>> cli_len = sizeof(cli_addr);
>>>> - cli_socket = accept(svr_socket, (struct sockaddr *) &cli_addr,
>>>> &cli_len);
>>>> + cli_socket = accept(-1, (struct sockaddr *) &cli_addr, &cli_len);
>>>> if (cli_socket < 0)
>>>> AXIS2_LOG_ERROR(env->log, AXIS2_LOG_SI,
>>>> "[Axis2][network_handler] Socket accept \
>>>> ---
>>>>
>>>> I'm not sure whether this is what really lead to the core dump, or was
>>>> it
>>>> something else.
>>>>
>>>> This was all that I could try on based on the less informative
>>>> valgrind.log that was sent. :(...
>>>>
>>>> Also i did test 1000 back-to-back requests and they all were served
>>>> without any issues.
>>>>
>>>> Regards,
>>>> Senaka
>>>>
>>>>
>>>>
>>>>> Samisa...
>>>>>
>>>>>
>>>>>
>>>>>> Could this solve the core dump issue?
>>>>>>
>>>>>> Regards,
>>>>>> Senaka
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>> Raghavendra SM wrote:
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>> As expected, when I put a delay of about 50 milliseconds between
>>>>>>>> the
>>>>>>>> requests there is no dump.
>>>>>>>>
>>>>>>>> But we would still like to know the following,
>>>>>>>> + Isn't Axis capable of handling such huge bombardment of requests
>>>>>>>> (of
>>>>>>>> the order of 1000's) one after another without any delay between
>>>>>>>> the
>>>>>>>> requests? Has anyone tried such client before?
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>> Yes it is supposed to. We have done performance testing that proved
>>>>>>> that
>>>>>>> Axis2/C can serve more than 300 rps.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>> + Can we make out anything out of the valgrind report or the axis
>>>>>>>> logs
>>>>>>>> that were attached?
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>> Unfortunately not. When valgrind runs, it makes the system slow,
>>>>>>> that
>>>>>>> prevents the system crashing in situations like that you are
>>>>>>> experiencing. To solve this kind of a problem, it is a must that you
>>>>>>> provide us with some sample code that can be used to reproduce the
>>>>>>> problem. Without having a way to re-produce the situation, there is
>>>>>>> no
>>>>>>> way to fix this problem - it could well be a problem in Axis2/C as
>>>>>>> well
>>>>>>> as it could also be in the code that you use.
>>>>>>>
>>>>>>> Samisa...
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>> Regards,
>>>>>>>> ~raghav
>>>>>>>>
>>>>>>>>
>>>>>>>> -----Original Message-----
>>>>>>>> From: Senaka Fernando [mailto:senaka@wso2.com]
>>>>>>>> Sent: Thursday, March 06, 2008 4:22 PM
>>>>>>>> To: Apache AXIS C User List
>>>>>>>> Subject: RE: Core dumps in Axis libraries
>>>>>>>>
>>>>>>>> Hi Raghav,
>>>>>>>>
>>>>>>>> Can you please try this with --enable-guthilla=yes, and send the
>>>>>>>> dump.log,
>>>>>>>> and also the appropriate log_files generated by the server/client.
>>>>>>>>
>>>>>>>> There also can be issues with your "condition variable" being used
>>>>>>>> in
>>>>>>>> multiple threads.
>>>>>>>>
>>>>>>>> It also seems that if you pause between your requests you might get
>>>>>>>> this
>>>>>>>> solved. According to what you say, when you increase the log_level,
>>>>>>>> obviously you waste some time there which might be the solution
>>>>>>>> rather.
>>>>>>>> You can use the AXIS2_SLEEP() and AXIS2_USLEEP() inside Axis2/C to
>>>>>>>> pause
>>>>>>>> between requests. You can get an idea on how to use these functions
>>>>>>>> by
>>>>>>>> taking a look at the Axis2/C user_guide/client samples.
>>>>>>>>
>>>>>>>> Regards,
>>>>>>>> Senaka
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>>
>>>>>>>>> As suggested, I tried using axis2c-1.3.0 both with and without
>>>>>>>>> --enable-guthilla=yes. But I still see different dumps
>>>>>>>>> consistently.
>>>>>>>>> Please find the attached file with back-traces.
>>>>>>>>>
>>>>>>>>> Are you using the same service client instance from multiple
>>>>>>>>> threads?
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>>>> We have a axis thread pool with default number of threads and a
>>>>>>>>>>>> set
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>> of our own threads(about 10, lets call them "database threads").
>>>>>>>>> + On receiving a HTTP soap request, axis thread invokes the
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>> appropriate
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>> function that is bound to our logic.
>>>>>>>>> + Now, the axis thread requests a "database thread" to perform the
>>>>>>>>> appropriate query and waits on a condition variable.
>>>>>>>>> + our "db thread" performs the necessary query and signals on that
>>>>>>>>> condition variable so that axis thread shall take the control
>>>>>>>>> over.
>>>>>>>>> + axis thread will create the om output and dispatch as usual.
>>>>>>>>>
>>>>>>>>> My doubts:
>>>>>>>>> + does this design has any apparent drawback when there is a
>>>>>>>>> flurry
>>>>>>>>> of
>>>>>>>>> back-to-back soap requests?
>>>>>>>>> + Is there a need to increase the axis thread pool size? If yes,
>>>>>>>>> how
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>> to
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>> do it?
>>>>>>>>> + As told before, the dump is seen during a particular scenario
>>>>>>>>> only.
>>>>>>>>> i.e: while trying to add a list of already existing users to my
>>>>>>>>> database. For each user there is a SOAP request generated and the
>>>>>>>>> requests are sequential, i.e the second request is sent only after
>>>>>>>>> the
>>>>>>>>> response for previous request is received.
>>>>>>>>> + if I increase the log level of my own module that uses axis2c,
>>>>>>>>> there
>>>>>>>>> is no dump. On increasing the log level, my module writes quite a
>>>>>>>>> bit
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>> of
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>> information to my log file.
>>>>>>>>>
>>>>>>>>> Your help is appreciated.
>>>>>>>>>
>>>>>>>>> Thanks & Regards,
>>>>>>>>> ~raghav
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> -----Original Message-----
>>>>>>>>> From: Samisa Abeysinghe [mailto:samisa@wso2.com]
>>>>>>>>> Sent: Wednesday, March 05, 2008 7:25 PM
>>>>>>>>> To: Apache AXIS C User List
>>>>>>>>> Subject: Re: Core dumps in Axis libraries
>>>>>>>>>
>>>>>>>>> Raghavendra SM wrote:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> Thanks Manjula,
>>>>>>>>>>
>>>>>>>>>> But thats not a viable option for us. We would still want to
>>>>>>>>>> continue
>>>>>>>>>> using axis2C-1.0 for some more time.
>>>>>>>>>>
>>>>>>>>>> Are these dumps well known?
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>> No.
>>>>>>>>>
>>>>>>>>> Are you using the same service client instance from multiple
>>>>>>>>> threads?
>>>>>>>>>
>>>>>>>>> Samisa...
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>> To unsubscribe, e-mail: axis-c-user-unsubscribe@ws.apache.org
>>>>>>>>> For additional commands, e-mail: axis-c-user-help@ws.apache.org
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>> To unsubscribe, e-mail: axis-c-user-unsubscribe@ws.apache.org
>>>>>>>>> For additional commands, e-mail: axis-c-user-help@ws.apache.org
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>> ---------------------------------------------------------------------
>>>>>>>> To unsubscribe, e-mail: axis-c-user-unsubscribe@ws.apache.org
>>>>>>>> For additional commands, e-mail: axis-c-user-help@ws.apache.org
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> ---------------------------------------------------------------------
>>>>>>>> To unsubscribe, e-mail: axis-c-user-unsubscribe@ws.apache.org
>>>>>>>> For additional commands, e-mail: axis-c-user-help@ws.apache.org
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>> --
>>>>>>> Samisa Abeysinghe
>>>>>>> Software Architect; WSO2 Inc.
>>>>>>>
>>>>>>> http://www.wso2.com/ - "Oxygenating the Web Service Platform."
>>>>>>>
>>>>>>>
>>>>>>> ---------------------------------------------------------------------
>>>>>>> To unsubscribe, e-mail: axis-c-user-unsubscribe@ws.apache.org
>>>>>>> For additional commands, e-mail: axis-c-user-help@ws.apache.org
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: axis-c-dev-unsubscribe@ws.apache.org
>>>>>> For additional commands, e-mail: axis-c-dev-help@ws.apache.org
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>> --
>>>>> Samisa Abeysinghe
>>>>> Software Architect; WSO2 Inc.
>>>>>
>>>>> http://www.wso2.com/ - "Oxygenating the Web Service Platform."
>>>>>
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: axis-c-dev-unsubscribe@ws.apache.org
>>>>> For additional commands, e-mail: axis-c-dev-help@ws.apache.org
>>>>>
>>>>>
>>>>>
>>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: axis-c-dev-unsubscribe@ws.apache.org
>>>> For additional commands, e-mail: axis-c-dev-help@ws.apache.org
>>>>
>>>>
>>>>
>>>>
>>>>
>>> --
>>> Samisa Abeysinghe
>>> Software Architect; WSO2 Inc.
>>>
>>> http://www.wso2.com/ - "Oxygenating the Web Service Platform."
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: axis-c-dev-unsubscribe@ws.apache.org
>>> For additional commands, e-mail: axis-c-dev-help@ws.apache.org
>>>
>>>
>>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: axis-c-dev-unsubscribe@ws.apache.org
>> For additional commands, e-mail: axis-c-dev-help@ws.apache.org
>>
>>
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: axis-c-dev-unsubscribe@ws.apache.org
> For additional commands, e-mail: axis-c-dev-help@ws.apache.org
>
>
>
>
--
Samisa Abeysinghe
Software Architect; WSO2 Inc.
http://www.wso2.com/ - "Oxygenating the Web Service Platform."
---------------------------------------------------------------------
To unsubscribe, e-mail: axis-c-dev-unsubscribe@ws.apache.org
For additional commands, e-mail: axis-c-dev-help@ws.apache.org
Re: Solution to Core Dump Issue Raised in User List [was: Re: Core
dumps in Axis libraries]
Posted by Senaka Fernando <se...@wso2.com>.
Hi again Samisa,
And, also, I don't have any similar database accessing logic etc.
Therefore, it is hard to conclude without seeing the actual code. Also,
the user seems to be using a modified version of "http_server_main.c", it
is actually a cpp file if you go through that portion which I highlighted
in his log. May be some issues due to not handling concurrency properly
could also lead him into trouble.
However, what I can say is that there was a bug if accept(2) was to fail,
according to the tests I did. And, according to literature available
online that can fail. Plus, we do have a mechanism of retrying, if it does
fail which did not work due to the segfault.
Regards,
Senaka
>> Were you able to re-create the problem without passing -1 to accept?
>
> No I was not able to make accept fail without passing -1. Added a printf
> there to print any failure in accept and it didn't. However, ran into the
> segfault issue when accept actually failed, which I had to do manually.
>
> I did test with 1000 back-2-back mtom requests in 5 terminals, and also
> 1000 repeated calls to an echo client that sends 1000 back-2-back
> requests, and in 4 terminals.
>
> None of these cases failed. There were occasional hangs but still it
> didn't result in any failure on either client or server side.
>
> However, I didn't have any way to test multiple requests made to a single
> port from multiple machines. That may be a different scenario.
>
> Regards,
> Senaka
>
>>
>> Samisa...
>>
>> Senaka Fernando wrote:
>>>> Senaka Fernando wrote:
>>>>
>>>>> Hi Samisa,
>>>>>
>>>>> I fixed two bugs regarding stream handling inside
>>>>> simple_http_svr_conn.c.
>>>>> This fixes an issue of this nature.
>>>>>
>>>>> Say, we have two server threads that try to serve requests offered to
>>>>> the
>>>>> axis2 server. Now, according to what is intended, one must serve and
>>>>> the
>>>>> other must wait until it can serve. The previous implementation
>>>>> caused
>>>>> this to fail with a segfault. But, with the fix, it will rather not
>>>>> crash
>>>>> but, loop until it gets a chance to serve the request.
>>>>>
>>>>>
>>>> Did you test before the fix and did it seg fault for you? Can you
>>>> please
>>>> explain the scenario under which it seg faulted? If it is a problem, I
>>>> too should be able to re-create it.
>>>>
>>>> And how did you solve the problem? By adding a sleep?
>>>>
>>>
>>> I made the accept(2) call fail. By passing a -1 to it. Once that
>>> failed,
>>> there were two places that segfaulted which I already fixed, few
>>> minutes
>>> ago. No, I didn't add a sleep(). I simply prevented the segfault by
>>> adding
>>> necessary is_null checks before calling methods. Then, the server just
>>> kept on spawning threads and freeing them as it can't serve the
>>> request.
>>> This is the expected behaviour in http_svr_thread.c I believe.
>>>
>>> In a real world scenario, a situation similar to a -1 passed to
>>> accept(2)
>>> will be temporal. So a respawned thread down the line should server the
>>> request without any problem.
>>>
>>> Passing -1 to accept was done by,
>>>
>>> Index: util/src/network_handler.c
>>> ===================================================================
>>> --- util/src/network_handler.c (revision 634578)
>>> +++ util/src/network_handler.c (working copy)
>>> @@ -221,7 +221,7 @@
>>> AXIS2_ENV_CHECK(env, AXIS2_CRITICAL_FAILURE);
>>>
>>> cli_len = sizeof(cli_addr);
>>> - cli_socket = accept(svr_socket, (struct sockaddr *) &cli_addr,
>>> &cli_len);
>>> + cli_socket = accept(-1, (struct sockaddr *) &cli_addr, &cli_len);
>>> if (cli_socket < 0)
>>> AXIS2_LOG_ERROR(env->log, AXIS2_LOG_SI,
>>> "[Axis2][network_handler] Socket accept \
>>> ---
>>>
>>> I'm not sure whether this is what really lead to the core dump, or was
>>> it
>>> something else.
>>>
>>> This was all that I could try on based on the less informative
>>> valgrind.log that was sent. :(...
>>>
>>> Also i did test 1000 back-to-back requests and they all were served
>>> without any issues.
>>>
>>> Regards,
>>> Senaka
>>>
>>>
>>>> Samisa...
>>>>
>>>>
>>>>> Could this solve the core dump issue?
>>>>>
>>>>> Regards,
>>>>> Senaka
>>>>>
>>>>>
>>>>>
>>>>>> Raghavendra SM wrote:
>>>>>>
>>>>>>
>>>>>>> As expected, when I put a delay of about 50 milliseconds between
>>>>>>> the
>>>>>>> requests there is no dump.
>>>>>>>
>>>>>>> But we would still like to know the following,
>>>>>>> + Isn't Axis capable of handling such huge bombardment of requests
>>>>>>> (of
>>>>>>> the order of 1000's) one after another without any delay between
>>>>>>> the
>>>>>>> requests? Has anyone tried such client before?
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>> Yes it is supposed to. We have done performance testing that proved
>>>>>> that
>>>>>> Axis2/C can serve more than 300 rps.
>>>>>>
>>>>>>
>>>>>>
>>>>>>> + Can we make out anything out of the valgrind report or the axis
>>>>>>> logs
>>>>>>> that were attached?
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>> Unfortunately not. When valgrind runs, it makes the system slow,
>>>>>> that
>>>>>> prevents the system crashing in situations like that you are
>>>>>> experiencing. To solve this kind of a problem, it is a must that you
>>>>>> provide us with some sample code that can be used to reproduce the
>>>>>> problem. Without having a way to re-produce the situation, there is
>>>>>> no
>>>>>> way to fix this problem - it could well be a problem in Axis2/C as
>>>>>> well
>>>>>> as it could also be in the code that you use.
>>>>>>
>>>>>> Samisa...
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>> Regards,
>>>>>>> ~raghav
>>>>>>>
>>>>>>>
>>>>>>> -----Original Message-----
>>>>>>> From: Senaka Fernando [mailto:senaka@wso2.com]
>>>>>>> Sent: Thursday, March 06, 2008 4:22 PM
>>>>>>> To: Apache AXIS C User List
>>>>>>> Subject: RE: Core dumps in Axis libraries
>>>>>>>
>>>>>>> Hi Raghav,
>>>>>>>
>>>>>>> Can you please try this with --enable-guthilla=yes, and send the
>>>>>>> dump.log,
>>>>>>> and also the appropriate log_files generated by the server/client.
>>>>>>>
>>>>>>> There also can be issues with your "condition variable" being used
>>>>>>> in
>>>>>>> multiple threads.
>>>>>>>
>>>>>>> It also seems that if you pause between your requests you might get
>>>>>>> this
>>>>>>> solved. According to what you say, when you increase the log_level,
>>>>>>> obviously you waste some time there which might be the solution
>>>>>>> rather.
>>>>>>> You can use the AXIS2_SLEEP() and AXIS2_USLEEP() inside Axis2/C to
>>>>>>> pause
>>>>>>> between requests. You can get an idea on how to use these functions
>>>>>>> by
>>>>>>> taking a look at the Axis2/C user_guide/client samples.
>>>>>>>
>>>>>>> Regards,
>>>>>>> Senaka
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>> Thanks,
>>>>>>>>
>>>>>>>> As suggested, I tried using axis2c-1.3.0 both with and without
>>>>>>>> --enable-guthilla=yes. But I still see different dumps
>>>>>>>> consistently.
>>>>>>>> Please find the attached file with back-traces.
>>>>>>>>
>>>>>>>> Are you using the same service client instance from multiple
>>>>>>>> threads?
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>>>> We have a axis thread pool with default number of threads and a
>>>>>>>>>>> set
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>> of our own threads(about 10, lets call them "database threads").
>>>>>>>> + On receiving a HTTP soap request, axis thread invokes the
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>> appropriate
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>> function that is bound to our logic.
>>>>>>>> + Now, the axis thread requests a "database thread" to perform the
>>>>>>>> appropriate query and waits on a condition variable.
>>>>>>>> + our "db thread" performs the necessary query and signals on that
>>>>>>>> condition variable so that axis thread shall take the control
>>>>>>>> over.
>>>>>>>> + axis thread will create the om output and dispatch as usual.
>>>>>>>>
>>>>>>>> My doubts:
>>>>>>>> + does this design has any apparent drawback when there is a
>>>>>>>> flurry
>>>>>>>> of
>>>>>>>> back-to-back soap requests?
>>>>>>>> + Is there a need to increase the axis thread pool size? If yes,
>>>>>>>> how
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>> to
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>> do it?
>>>>>>>> + As told before, the dump is seen during a particular scenario
>>>>>>>> only.
>>>>>>>> i.e: while trying to add a list of already existing users to my
>>>>>>>> database. For each user there is a SOAP request generated and the
>>>>>>>> requests are sequential, i.e the second request is sent only after
>>>>>>>> the
>>>>>>>> response for previous request is received.
>>>>>>>> + if I increase the log level of my own module that uses axis2c,
>>>>>>>> there
>>>>>>>> is no dump. On increasing the log level, my module writes quite a
>>>>>>>> bit
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>> of
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>> information to my log file.
>>>>>>>>
>>>>>>>> Your help is appreciated.
>>>>>>>>
>>>>>>>> Thanks & Regards,
>>>>>>>> ~raghav
>>>>>>>>
>>>>>>>>
>>>>>>>> -----Original Message-----
>>>>>>>> From: Samisa Abeysinghe [mailto:samisa@wso2.com]
>>>>>>>> Sent: Wednesday, March 05, 2008 7:25 PM
>>>>>>>> To: Apache AXIS C User List
>>>>>>>> Subject: Re: Core dumps in Axis libraries
>>>>>>>>
>>>>>>>> Raghavendra SM wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>> Thanks Manjula,
>>>>>>>>>
>>>>>>>>> But thats not a viable option for us. We would still want to
>>>>>>>>> continue
>>>>>>>>> using axis2C-1.0 for some more time.
>>>>>>>>>
>>>>>>>>> Are these dumps well known?
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>> No.
>>>>>>>>
>>>>>>>> Are you using the same service client instance from multiple
>>>>>>>> threads?
>>>>>>>>
>>>>>>>> Samisa...
>>>>>>>>
>>>>>>>>
>>>>>>>> ---------------------------------------------------------------------
>>>>>>>> To unsubscribe, e-mail: axis-c-user-unsubscribe@ws.apache.org
>>>>>>>> For additional commands, e-mail: axis-c-user-help@ws.apache.org
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> ---------------------------------------------------------------------
>>>>>>>> To unsubscribe, e-mail: axis-c-user-unsubscribe@ws.apache.org
>>>>>>>> For additional commands, e-mail: axis-c-user-help@ws.apache.org
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>> ---------------------------------------------------------------------
>>>>>>> To unsubscribe, e-mail: axis-c-user-unsubscribe@ws.apache.org
>>>>>>> For additional commands, e-mail: axis-c-user-help@ws.apache.org
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> ---------------------------------------------------------------------
>>>>>>> To unsubscribe, e-mail: axis-c-user-unsubscribe@ws.apache.org
>>>>>>> For additional commands, e-mail: axis-c-user-help@ws.apache.org
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>> --
>>>>>> Samisa Abeysinghe
>>>>>> Software Architect; WSO2 Inc.
>>>>>>
>>>>>> http://www.wso2.com/ - "Oxygenating the Web Service Platform."
>>>>>>
>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: axis-c-user-unsubscribe@ws.apache.org
>>>>>> For additional commands, e-mail: axis-c-user-help@ws.apache.org
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: axis-c-dev-unsubscribe@ws.apache.org
>>>>> For additional commands, e-mail: axis-c-dev-help@ws.apache.org
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>> --
>>>> Samisa Abeysinghe
>>>> Software Architect; WSO2 Inc.
>>>>
>>>> http://www.wso2.com/ - "Oxygenating the Web Service Platform."
>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: axis-c-dev-unsubscribe@ws.apache.org
>>>> For additional commands, e-mail: axis-c-dev-help@ws.apache.org
>>>>
>>>>
>>>>
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: axis-c-dev-unsubscribe@ws.apache.org
>>> For additional commands, e-mail: axis-c-dev-help@ws.apache.org
>>>
>>>
>>>
>>>
>>
>>
>> --
>> Samisa Abeysinghe
>> Software Architect; WSO2 Inc.
>>
>> http://www.wso2.com/ - "Oxygenating the Web Service Platform."
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: axis-c-dev-unsubscribe@ws.apache.org
>> For additional commands, e-mail: axis-c-dev-help@ws.apache.org
>>
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: axis-c-dev-unsubscribe@ws.apache.org
> For additional commands, e-mail: axis-c-dev-help@ws.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: axis-c-dev-unsubscribe@ws.apache.org
For additional commands, e-mail: axis-c-dev-help@ws.apache.org
Re: Solution to Core Dump Issue Raised in User List [was: Re: Core
dumps in Axis libraries]
Posted by Senaka Fernando <se...@wso2.com>.
> Were you able to re-create the problem without passing -1 to accept?
No I was not able to make accept fail without passing -1. Added a printf
there to print any failure in accept and it didn't. However, ran into the
segfault issue when accept actually failed, which I had to do manually.
I did test with 1000 back-2-back mtom requests in 5 terminals, and also
1000 repeated calls to an echo client that sends 1000 back-2-back
requests, and in 4 terminals.
None of these cases failed. There were occasional hangs but still it
didn't result in any failure on either client or server side.
However, I didn't have any way to test multiple requests made to a single
port from multiple machines. That may be a different scenario.
Regards,
Senaka
>
> Samisa...
>
> Senaka Fernando wrote:
>>> Senaka Fernando wrote:
>>>
>>>> Hi Samisa,
>>>>
>>>> I fixed two bugs regarding stream handling inside
>>>> simple_http_svr_conn.c.
>>>> This fixes an issue of this nature.
>>>>
>>>> Say, we have two server threads that try to serve requests offered to
>>>> the
>>>> axis2 server. Now, according to what is intended, one must serve and
>>>> the
>>>> other must wait until it can serve. The previous implementation caused
>>>> this to fail with a segfault. But, with the fix, it will rather not
>>>> crash
>>>> but, loop until it gets a chance to serve the request.
>>>>
>>>>
>>> Did you test before the fix and did it seg fault for you? Can you
>>> please
>>> explain the scenario under which it seg faulted? If it is a problem, I
>>> too should be able to re-create it.
>>>
>>> And how did you solve the problem? By adding a sleep?
>>>
>>
>> I made the accept(2) call fail. By passing a -1 to it. Once that failed,
>> there were two places that segfaulted which I already fixed, few minutes
>> ago. No, I didn't add a sleep(). I simply prevented the segfault by
>> adding
>> necessary is_null checks before calling methods. Then, the server just
>> kept on spawning threads and freeing them as it can't serve the request.
>> This is the expected behaviour in http_svr_thread.c I believe.
>>
>> In a real world scenario, a situation similar to a -1 passed to
>> accept(2)
>> will be temporal. So a respawned thread down the line should server the
>> request without any problem.
>>
>> Passing -1 to accept was done by,
>>
>> Index: util/src/network_handler.c
>> ===================================================================
>> --- util/src/network_handler.c (revision 634578)
>> +++ util/src/network_handler.c (working copy)
>> @@ -221,7 +221,7 @@
>> AXIS2_ENV_CHECK(env, AXIS2_CRITICAL_FAILURE);
>>
>> cli_len = sizeof(cli_addr);
>> - cli_socket = accept(svr_socket, (struct sockaddr *) &cli_addr,
>> &cli_len);
>> + cli_socket = accept(-1, (struct sockaddr *) &cli_addr, &cli_len);
>> if (cli_socket < 0)
>> AXIS2_LOG_ERROR(env->log, AXIS2_LOG_SI,
>> "[Axis2][network_handler] Socket accept \
>> ---
>>
>> I'm not sure whether this is what really lead to the core dump, or was
>> it
>> something else.
>>
>> This was all that I could try on based on the less informative
>> valgrind.log that was sent. :(...
>>
>> Also i did test 1000 back-to-back requests and they all were served
>> without any issues.
>>
>> Regards,
>> Senaka
>>
>>
>>> Samisa...
>>>
>>>
>>>> Could this solve the core dump issue?
>>>>
>>>> Regards,
>>>> Senaka
>>>>
>>>>
>>>>
>>>>> Raghavendra SM wrote:
>>>>>
>>>>>
>>>>>> As expected, when I put a delay of about 50 milliseconds between the
>>>>>> requests there is no dump.
>>>>>>
>>>>>> But we would still like to know the following,
>>>>>> + Isn't Axis capable of handling such huge bombardment of requests
>>>>>> (of
>>>>>> the order of 1000's) one after another without any delay between the
>>>>>> requests? Has anyone tried such client before?
>>>>>>
>>>>>>
>>>>>>
>>>>> Yes it is supposed to. We have done performance testing that proved
>>>>> that
>>>>> Axis2/C can serve more than 300 rps.
>>>>>
>>>>>
>>>>>
>>>>>> + Can we make out anything out of the valgrind report or the axis
>>>>>> logs
>>>>>> that were attached?
>>>>>>
>>>>>>
>>>>>>
>>>>> Unfortunately not. When valgrind runs, it makes the system slow, that
>>>>> prevents the system crashing in situations like that you are
>>>>> experiencing. To solve this kind of a problem, it is a must that you
>>>>> provide us with some sample code that can be used to reproduce the
>>>>> problem. Without having a way to re-produce the situation, there is
>>>>> no
>>>>> way to fix this problem - it could well be a problem in Axis2/C as
>>>>> well
>>>>> as it could also be in the code that you use.
>>>>>
>>>>> Samisa...
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>> Regards,
>>>>>> ~raghav
>>>>>>
>>>>>>
>>>>>> -----Original Message-----
>>>>>> From: Senaka Fernando [mailto:senaka@wso2.com]
>>>>>> Sent: Thursday, March 06, 2008 4:22 PM
>>>>>> To: Apache AXIS C User List
>>>>>> Subject: RE: Core dumps in Axis libraries
>>>>>>
>>>>>> Hi Raghav,
>>>>>>
>>>>>> Can you please try this with --enable-guthilla=yes, and send the
>>>>>> dump.log,
>>>>>> and also the appropriate log_files generated by the server/client.
>>>>>>
>>>>>> There also can be issues with your "condition variable" being used
>>>>>> in
>>>>>> multiple threads.
>>>>>>
>>>>>> It also seems that if you pause between your requests you might get
>>>>>> this
>>>>>> solved. According to what you say, when you increase the log_level,
>>>>>> obviously you waste some time there which might be the solution
>>>>>> rather.
>>>>>> You can use the AXIS2_SLEEP() and AXIS2_USLEEP() inside Axis2/C to
>>>>>> pause
>>>>>> between requests. You can get an idea on how to use these functions
>>>>>> by
>>>>>> taking a look at the Axis2/C user_guide/client samples.
>>>>>>
>>>>>> Regards,
>>>>>> Senaka
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>> Thanks,
>>>>>>>
>>>>>>> As suggested, I tried using axis2c-1.3.0 both with and without
>>>>>>> --enable-guthilla=yes. But I still see different dumps
>>>>>>> consistently.
>>>>>>> Please find the attached file with back-traces.
>>>>>>>
>>>>>>> Are you using the same service client instance from multiple
>>>>>>> threads?
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>>>> We have a axis thread pool with default number of threads and a
>>>>>>>>>> set
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>> of our own threads(about 10, lets call them "database threads").
>>>>>>> + On receiving a HTTP soap request, axis thread invokes the
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>> appropriate
>>>>>>
>>>>>>
>>>>>>
>>>>>>> function that is bound to our logic.
>>>>>>> + Now, the axis thread requests a "database thread" to perform the
>>>>>>> appropriate query and waits on a condition variable.
>>>>>>> + our "db thread" performs the necessary query and signals on that
>>>>>>> condition variable so that axis thread shall take the control over.
>>>>>>> + axis thread will create the om output and dispatch as usual.
>>>>>>>
>>>>>>> My doubts:
>>>>>>> + does this design has any apparent drawback when there is a flurry
>>>>>>> of
>>>>>>> back-to-back soap requests?
>>>>>>> + Is there a need to increase the axis thread pool size? If yes,
>>>>>>> how
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>> to
>>>>>>
>>>>>>
>>>>>>
>>>>>>> do it?
>>>>>>> + As told before, the dump is seen during a particular scenario
>>>>>>> only.
>>>>>>> i.e: while trying to add a list of already existing users to my
>>>>>>> database. For each user there is a SOAP request generated and the
>>>>>>> requests are sequential, i.e the second request is sent only after
>>>>>>> the
>>>>>>> response for previous request is received.
>>>>>>> + if I increase the log level of my own module that uses axis2c,
>>>>>>> there
>>>>>>> is no dump. On increasing the log level, my module writes quite a
>>>>>>> bit
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>> of
>>>>>>
>>>>>>
>>>>>>
>>>>>>> information to my log file.
>>>>>>>
>>>>>>> Your help is appreciated.
>>>>>>>
>>>>>>> Thanks & Regards,
>>>>>>> ~raghav
>>>>>>>
>>>>>>>
>>>>>>> -----Original Message-----
>>>>>>> From: Samisa Abeysinghe [mailto:samisa@wso2.com]
>>>>>>> Sent: Wednesday, March 05, 2008 7:25 PM
>>>>>>> To: Apache AXIS C User List
>>>>>>> Subject: Re: Core dumps in Axis libraries
>>>>>>>
>>>>>>> Raghavendra SM wrote:
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>> Thanks Manjula,
>>>>>>>>
>>>>>>>> But thats not a viable option for us. We would still want to
>>>>>>>> continue
>>>>>>>> using axis2C-1.0 for some more time.
>>>>>>>>
>>>>>>>> Are these dumps well known?
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>> No.
>>>>>>>
>>>>>>> Are you using the same service client instance from multiple
>>>>>>> threads?
>>>>>>>
>>>>>>> Samisa...
>>>>>>>
>>>>>>>
>>>>>>> ---------------------------------------------------------------------
>>>>>>> To unsubscribe, e-mail: axis-c-user-unsubscribe@ws.apache.org
>>>>>>> For additional commands, e-mail: axis-c-user-help@ws.apache.org
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> ---------------------------------------------------------------------
>>>>>>> To unsubscribe, e-mail: axis-c-user-unsubscribe@ws.apache.org
>>>>>>> For additional commands, e-mail: axis-c-user-help@ws.apache.org
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: axis-c-user-unsubscribe@ws.apache.org
>>>>>> For additional commands, e-mail: axis-c-user-help@ws.apache.org
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: axis-c-user-unsubscribe@ws.apache.org
>>>>>> For additional commands, e-mail: axis-c-user-help@ws.apache.org
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>> --
>>>>> Samisa Abeysinghe
>>>>> Software Architect; WSO2 Inc.
>>>>>
>>>>> http://www.wso2.com/ - "Oxygenating the Web Service Platform."
>>>>>
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: axis-c-user-unsubscribe@ws.apache.org
>>>>> For additional commands, e-mail: axis-c-user-help@ws.apache.org
>>>>>
>>>>>
>>>>>
>>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: axis-c-dev-unsubscribe@ws.apache.org
>>>> For additional commands, e-mail: axis-c-dev-help@ws.apache.org
>>>>
>>>>
>>>>
>>>>
>>>>
>>> --
>>> Samisa Abeysinghe
>>> Software Architect; WSO2 Inc.
>>>
>>> http://www.wso2.com/ - "Oxygenating the Web Service Platform."
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: axis-c-dev-unsubscribe@ws.apache.org
>>> For additional commands, e-mail: axis-c-dev-help@ws.apache.org
>>>
>>>
>>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: axis-c-dev-unsubscribe@ws.apache.org
>> For additional commands, e-mail: axis-c-dev-help@ws.apache.org
>>
>>
>>
>>
>
>
> --
> Samisa Abeysinghe
> Software Architect; WSO2 Inc.
>
> http://www.wso2.com/ - "Oxygenating the Web Service Platform."
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: axis-c-dev-unsubscribe@ws.apache.org
> For additional commands, e-mail: axis-c-dev-help@ws.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: axis-c-dev-unsubscribe@ws.apache.org
For additional commands, e-mail: axis-c-dev-help@ws.apache.org
Re: Solution to Core Dump Issue Raised in User List [was: Re: Core
dumps in Axis libraries]
Posted by Samisa Abeysinghe <sa...@wso2.com>.
Were you able to re-create the problem without passing -1 to accept?
Samisa...
Senaka Fernando wrote:
>> Senaka Fernando wrote:
>>
>>> Hi Samisa,
>>>
>>> I fixed two bugs regarding stream handling inside
>>> simple_http_svr_conn.c.
>>> This fixes an issue of this nature.
>>>
>>> Say, we have two server threads that try to serve requests offered to
>>> the
>>> axis2 server. Now, according to what is intended, one must serve and the
>>> other must wait until it can serve. The previous implementation caused
>>> this to fail with a segfault. But, with the fix, it will rather not
>>> crash
>>> but, loop until it gets a chance to serve the request.
>>>
>>>
>> Did you test before the fix and did it seg fault for you? Can you please
>> explain the scenario under which it seg faulted? If it is a problem, I
>> too should be able to re-create it.
>>
>> And how did you solve the problem? By adding a sleep?
>>
>
> I made the accept(2) call fail. By passing a -1 to it. Once that failed,
> there were two places that segfaulted which I already fixed, few minutes
> ago. No, I didn't add a sleep(). I simply prevented the segfault by adding
> necessary is_null checks before calling methods. Then, the server just
> kept on spawning threads and freeing them as it can't serve the request.
> This is the expected behaviour in http_svr_thread.c I believe.
>
> In a real world scenario, a situation similar to a -1 passed to accept(2)
> will be temporal. So a respawned thread down the line should server the
> request without any problem.
>
> Passing -1 to accept was done by,
>
> Index: util/src/network_handler.c
> ===================================================================
> --- util/src/network_handler.c (revision 634578)
> +++ util/src/network_handler.c (working copy)
> @@ -221,7 +221,7 @@
> AXIS2_ENV_CHECK(env, AXIS2_CRITICAL_FAILURE);
>
> cli_len = sizeof(cli_addr);
> - cli_socket = accept(svr_socket, (struct sockaddr *) &cli_addr,
> &cli_len);
> + cli_socket = accept(-1, (struct sockaddr *) &cli_addr, &cli_len);
> if (cli_socket < 0)
> AXIS2_LOG_ERROR(env->log, AXIS2_LOG_SI,
> "[Axis2][network_handler] Socket accept \
> ---
>
> I'm not sure whether this is what really lead to the core dump, or was it
> something else.
>
> This was all that I could try on based on the less informative
> valgrind.log that was sent. :(...
>
> Also i did test 1000 back-to-back requests and they all were served
> without any issues.
>
> Regards,
> Senaka
>
>
>> Samisa...
>>
>>
>>> Could this solve the core dump issue?
>>>
>>> Regards,
>>> Senaka
>>>
>>>
>>>
>>>> Raghavendra SM wrote:
>>>>
>>>>
>>>>> As expected, when I put a delay of about 50 milliseconds between the
>>>>> requests there is no dump.
>>>>>
>>>>> But we would still like to know the following,
>>>>> + Isn't Axis capable of handling such huge bombardment of requests (of
>>>>> the order of 1000's) one after another without any delay between the
>>>>> requests? Has anyone tried such client before?
>>>>>
>>>>>
>>>>>
>>>> Yes it is supposed to. We have done performance testing that proved
>>>> that
>>>> Axis2/C can serve more than 300 rps.
>>>>
>>>>
>>>>
>>>>> + Can we make out anything out of the valgrind report or the axis logs
>>>>> that were attached?
>>>>>
>>>>>
>>>>>
>>>> Unfortunately not. When valgrind runs, it makes the system slow, that
>>>> prevents the system crashing in situations like that you are
>>>> experiencing. To solve this kind of a problem, it is a must that you
>>>> provide us with some sample code that can be used to reproduce the
>>>> problem. Without having a way to re-produce the situation, there is no
>>>> way to fix this problem - it could well be a problem in Axis2/C as well
>>>> as it could also be in the code that you use.
>>>>
>>>> Samisa...
>>>>
>>>>
>>>>
>>>>
>>>>> Regards,
>>>>> ~raghav
>>>>>
>>>>>
>>>>> -----Original Message-----
>>>>> From: Senaka Fernando [mailto:senaka@wso2.com]
>>>>> Sent: Thursday, March 06, 2008 4:22 PM
>>>>> To: Apache AXIS C User List
>>>>> Subject: RE: Core dumps in Axis libraries
>>>>>
>>>>> Hi Raghav,
>>>>>
>>>>> Can you please try this with --enable-guthilla=yes, and send the
>>>>> dump.log,
>>>>> and also the appropriate log_files generated by the server/client.
>>>>>
>>>>> There also can be issues with your "condition variable" being used in
>>>>> multiple threads.
>>>>>
>>>>> It also seems that if you pause between your requests you might get
>>>>> this
>>>>> solved. According to what you say, when you increase the log_level,
>>>>> obviously you waste some time there which might be the solution
>>>>> rather.
>>>>> You can use the AXIS2_SLEEP() and AXIS2_USLEEP() inside Axis2/C to
>>>>> pause
>>>>> between requests. You can get an idea on how to use these functions by
>>>>> taking a look at the Axis2/C user_guide/client samples.
>>>>>
>>>>> Regards,
>>>>> Senaka
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>> Thanks,
>>>>>>
>>>>>> As suggested, I tried using axis2c-1.3.0 both with and without
>>>>>> --enable-guthilla=yes. But I still see different dumps consistently.
>>>>>> Please find the attached file with back-traces.
>>>>>>
>>>>>> Are you using the same service client instance from multiple threads?
>>>>>>
>>>>>>
>>>>>>
>>>>>>>>> We have a axis thread pool with default number of threads and a
>>>>>>>>> set
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>> of our own threads(about 10, lets call them "database threads").
>>>>>> + On receiving a HTTP soap request, axis thread invokes the
>>>>>>
>>>>>>
>>>>>>
>>>>> appropriate
>>>>>
>>>>>
>>>>>
>>>>>> function that is bound to our logic.
>>>>>> + Now, the axis thread requests a "database thread" to perform the
>>>>>> appropriate query and waits on a condition variable.
>>>>>> + our "db thread" performs the necessary query and signals on that
>>>>>> condition variable so that axis thread shall take the control over.
>>>>>> + axis thread will create the om output and dispatch as usual.
>>>>>>
>>>>>> My doubts:
>>>>>> + does this design has any apparent drawback when there is a flurry
>>>>>> of
>>>>>> back-to-back soap requests?
>>>>>> + Is there a need to increase the axis thread pool size? If yes, how
>>>>>>
>>>>>>
>>>>>>
>>>>> to
>>>>>
>>>>>
>>>>>
>>>>>> do it?
>>>>>> + As told before, the dump is seen during a particular scenario only.
>>>>>> i.e: while trying to add a list of already existing users to my
>>>>>> database. For each user there is a SOAP request generated and the
>>>>>> requests are sequential, i.e the second request is sent only after
>>>>>> the
>>>>>> response for previous request is received.
>>>>>> + if I increase the log level of my own module that uses axis2c,
>>>>>> there
>>>>>> is no dump. On increasing the log level, my module writes quite a bit
>>>>>>
>>>>>>
>>>>>>
>>>>> of
>>>>>
>>>>>
>>>>>
>>>>>> information to my log file.
>>>>>>
>>>>>> Your help is appreciated.
>>>>>>
>>>>>> Thanks & Regards,
>>>>>> ~raghav
>>>>>>
>>>>>>
>>>>>> -----Original Message-----
>>>>>> From: Samisa Abeysinghe [mailto:samisa@wso2.com]
>>>>>> Sent: Wednesday, March 05, 2008 7:25 PM
>>>>>> To: Apache AXIS C User List
>>>>>> Subject: Re: Core dumps in Axis libraries
>>>>>>
>>>>>> Raghavendra SM wrote:
>>>>>>
>>>>>>
>>>>>>
>>>>>>> Thanks Manjula,
>>>>>>>
>>>>>>> But thats not a viable option for us. We would still want to
>>>>>>> continue
>>>>>>> using axis2C-1.0 for some more time.
>>>>>>>
>>>>>>> Are these dumps well known?
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>> No.
>>>>>>
>>>>>> Are you using the same service client instance from multiple threads?
>>>>>>
>>>>>> Samisa...
>>>>>>
>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: axis-c-user-unsubscribe@ws.apache.org
>>>>>> For additional commands, e-mail: axis-c-user-help@ws.apache.org
>>>>>>
>>>>>>
>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: axis-c-user-unsubscribe@ws.apache.org
>>>>>> For additional commands, e-mail: axis-c-user-help@ws.apache.org
>>>>>>
>>>>>>
>>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: axis-c-user-unsubscribe@ws.apache.org
>>>>> For additional commands, e-mail: axis-c-user-help@ws.apache.org
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: axis-c-user-unsubscribe@ws.apache.org
>>>>> For additional commands, e-mail: axis-c-user-help@ws.apache.org
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>> --
>>>> Samisa Abeysinghe
>>>> Software Architect; WSO2 Inc.
>>>>
>>>> http://www.wso2.com/ - "Oxygenating the Web Service Platform."
>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: axis-c-user-unsubscribe@ws.apache.org
>>>> For additional commands, e-mail: axis-c-user-help@ws.apache.org
>>>>
>>>>
>>>>
>>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: axis-c-dev-unsubscribe@ws.apache.org
>>> For additional commands, e-mail: axis-c-dev-help@ws.apache.org
>>>
>>>
>>>
>>>
>>>
>> --
>> Samisa Abeysinghe
>> Software Architect; WSO2 Inc.
>>
>> http://www.wso2.com/ - "Oxygenating the Web Service Platform."
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: axis-c-dev-unsubscribe@ws.apache.org
>> For additional commands, e-mail: axis-c-dev-help@ws.apache.org
>>
>>
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: axis-c-dev-unsubscribe@ws.apache.org
> For additional commands, e-mail: axis-c-dev-help@ws.apache.org
>
>
>
>
--
Samisa Abeysinghe
Software Architect; WSO2 Inc.
http://www.wso2.com/ - "Oxygenating the Web Service Platform."
---------------------------------------------------------------------
To unsubscribe, e-mail: axis-c-dev-unsubscribe@ws.apache.org
For additional commands, e-mail: axis-c-dev-help@ws.apache.org