How to avoid timeouts during long processing?

[Georgios Petasis <pe...@iit.demokritos.gr>]

Hi all,

I have a quite lengthy computation exposed as a service through rivet. 
What I have observed, is that apache after some time,
thinks that the process running my computation has stuck, and starts the 
same computation in a second process.

Is there a way to avoid this?

Regards,

George

---------------------------------------------------------------------
To unsubscribe, e-mail: rivet-dev-unsubscribe@tcl.apache.org
For additional commands, e-mail: rivet-dev-help@tcl.apache.org

Re: How to avoid timeouts during long processing?

[Georgios Petasis <pe...@iit.demokritos.gr>]

On 25/11/2014 01:33, Massimo Manghi wrote:
> On 11/24/2014 06:58 PM, Georgios Petasis wrote:
>> Well it is a badly designed system. I have a lengthy calculation (like 8
> yes, definitely
>
>> hours). So the caller (which is an apache tomcat, written in java) calls
>> my service (apache2 + rivet), and waits until I have finished. But for
>> some reason (I don't know), apache starts a second instance after about
>> 2 hours, so I end up running multiple instances of the computation in
>> parallel.
>>
>
>> I don't think it is an issue of connection timeout (both processes run
>> on the same server). Something internally in apache 2 times out, and
>> thinks the process serving the request has stopped, and tries to run
>> again the same request on another child.
>>
> TCP connections through loopback intefaces have timeouts like any other
> TCP connections, don't they? The sharp 2 hours limit seems to be more
> like something like a resource consumption limit imposed by the Apache
> framework, maybe a CPU time consumed by the child process. From what you
> say the data consumer (client) is not an ordinary web browser, so the
> client itself could follow any policy and maybe restart the computation
> when the server fails and drops the connection....just guessing....
I am guessing also. For sure, they don't have such a problem with 
tomcat, where connections do not time out after 12 or more hours. It is 
happening only under apache...

Best,

George

---------------------------------------------------------------------
To unsubscribe, e-mail: rivet-dev-unsubscribe@tcl.apache.org
For additional commands, e-mail: rivet-dev-help@tcl.apache.org

Re: How to avoid timeouts during long processing?

[Massimo Manghi <mx...@apache.org>]

On 11/24/2014 06:58 PM, Georgios Petasis wrote:
> Well it is a badly designed system. I have a lengthy calculation (like 8

yes, definitely

> hours). So the caller (which is an apache tomcat, written in java) calls
> my service (apache2 + rivet), and waits until I have finished. But for
> some reason (I don't know), apache starts a second instance after about
> 2 hours, so I end up running multiple instances of the computation in
> parallel.
> 


> I don't think it is an issue of connection timeout (both processes run
> on the same server). Something internally in apache 2 times out, and
> thinks the process serving the request has stopped, and tries to run
> again the same request on another child.
> 

TCP connections through loopback intefaces have timeouts like any other
TCP connections, don't they? The sharp 2 hours limit seems to be more
like something like a resource consumption limit imposed by the Apache
framework, maybe a CPU time consumed by the child process. From what you
say the data consumer (client) is not an ordinary web browser, so the
client itself could follow any policy and maybe restart the computation
when the server fails and drops the connection....just guessing....

 -- Massimo

---------------------------------------------------------------------
To unsubscribe, e-mail: rivet-dev-unsubscribe@tcl.apache.org
For additional commands, e-mail: rivet-dev-help@tcl.apache.org

Re: How to avoid timeouts during long processing?

[Georgios Petasis <pe...@iit.demokritos.gr>]

On 24/11/2014 20:28, Jeff Lawson wrote:
> It's probably best to perform such long processing as a separate 
> crontab or daemon that runs your Tcl code, rather than doing it all as 
> a single Rivet page. Your Rivet page should instead initiate the 
> processing by using a database or a file to store new "jobs" to be 
> performed by the backend crontab/daemon, and then return. You can make 
> a different Rivet page that checks for the results of the processing 
> that had been stored in a file or in a database updated by your 
> processing script.
>
>
Dear Jeff,

I know about this alternative, but it is not an option to apply it, the 
decision has been taken in the past and cannot be changed.

Regards,

George

---------------------------------------------------------------------
To unsubscribe, e-mail: rivet-dev-unsubscribe@tcl.apache.org
For additional commands, e-mail: rivet-dev-help@tcl.apache.org

Re: How to avoid timeouts during long processing?

[Jeff Lawson <je...@bovine.net>]

It's probably best to perform such long processing as a separate crontab or
daemon that runs your Tcl code, rather than doing it all as a single Rivet
page. Your Rivet page should instead initiate the processing by using a
database or a file to store new "jobs" to be performed by the backend
crontab/daemon, and then return. You can make a different Rivet page that
checks for the results of the processing that had been stored in a file or
in a database updated by your processing script.


On Mon, Nov 24, 2014 at 11:58 AM, Georgios Petasis <
petasis@iit.demokritos.gr> wrote:

> On 22/11/2014 17:15, Massimo Manghi wrote:
>
>> I can understand something like a timeout occurs with the connection, but
>> I don't understand why apache starts it over in another process. From what
>> you say I can only gather that the new computation is started from the
>> beginning and the state of the former lost. What do you read from the logs?
>> If it's just a problem of connection timeout I think you can keep it alive
>> just sending to the client some character from time to time during the
>> computation flushing the channel after calling puts...
>>
> Well it is a badly designed system. I have a lengthy calculation (like 8
> hours). So the caller (which is an apache tomcat, written in java) calls my
> service (apache2 + rivet), and waits until I have finished. But for some
> reason (I don't know), apache starts a second instance after about 2 hours,
> so I end up running multiple instances of the computation in parallel.
>
> I don't think it is an issue of connection timeout (both processes run on
> the same server). Something internally in apache 2 times out, and thinks
> the process serving the request has stopped, and tries to run again the
> same request on another child.
>
> Searching the documentation, Timeout=0 is suggested, but I don't know the
> implications.
>
>
>> apt-cache-ng is repository cache for Debian/Ubuntu and has a web
>> interface for the database diagnostics and maintenance. Some of the
>> maintenance tasks take several minutes and the process never stops, but it
>> sends some feedback to the client
>>
> (I know about starting a process and check the status with subsequent
> calls, but unfortunately the system was not designed this way)
>
> Best regards,
>
> George
>
>>
>> let me know
>>
>>  -- Massimo
>>
>>
>> On 11/22/2014 12:26 PM, Georgios Petasis wrote:
>>
>>> Hi all,
>>>
>>> I have a quite lengthy computation exposed as a service through rivet.
>>> What I have observed, is that apache after some time,
>>> thinks that the process running my computation has stuck, and starts the
>>> same computation in a second process.
>>>
>>> Is there a way to avoid this?
>>>
>>> Regards,
>>>
>>> George
>>>
>>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: rivet-dev-unsubscribe@tcl.apache.org
> For additional commands, e-mail: rivet-dev-help@tcl.apache.org
>
>

Re: How to avoid timeouts during long processing?

[Georgios Petasis <pe...@iit.demokritos.gr>]

On 22/11/2014 17:15, Massimo Manghi wrote:
> I can understand something like a timeout occurs with the connection, 
> but I don't understand why apache starts it over in another process. 
> From what you say I can only gather that the new computation is 
> started from the beginning and the state of the former lost. What do 
> you read from the logs? If it's just a problem of connection timeout I 
> think you can keep it alive just sending to the client some character 
> from time to time during the computation flushing the channel after 
> calling puts...
Well it is a badly designed system. I have a lengthy calculation (like 8 
hours). So the caller (which is an apache tomcat, written in java) calls 
my service (apache2 + rivet), and waits until I have finished. But for 
some reason (I don't know), apache starts a second instance after about 
2 hours, so I end up running multiple instances of the computation in 
parallel.

I don't think it is an issue of connection timeout (both processes run 
on the same server). Something internally in apache 2 times out, and 
thinks the process serving the request has stopped, and tries to run 
again the same request on another child.

Searching the documentation, Timeout=0 is suggested, but I don't know 
the implications.

>
> apt-cache-ng is repository cache for Debian/Ubuntu and has a web 
> interface for the database diagnostics and maintenance. Some of the 
> maintenance tasks take several minutes and the process never stops, 
> but it sends some feedback to the client
(I know about starting a process and check the status with subsequent 
calls, but unfortunately the system was not designed this way)

Best regards,

George
>
> let me know
>
>  -- Massimo
>
>
> On 11/22/2014 12:26 PM, Georgios Petasis wrote:
>> Hi all,
>>
>> I have a quite lengthy computation exposed as a service through rivet.
>> What I have observed, is that apache after some time,
>> thinks that the process running my computation has stuck, and starts the
>> same computation in a second process.
>>
>> Is there a way to avoid this?
>>
>> Regards,
>>
>> George
>>


---------------------------------------------------------------------
To unsubscribe, e-mail: rivet-dev-unsubscribe@tcl.apache.org
For additional commands, e-mail: rivet-dev-help@tcl.apache.org

Re: How to avoid timeouts during long processing?

[Massimo Manghi <ma...@unipr.it>]

I can understand something like a timeout occurs with the connection, 
but I don't understand why apache starts it over in another process. 
 From what you say I can only gather that the new computation is started 
from the beginning and the state of the former lost. What do you read 
from the logs? If it's just a problem of connection timeout I think you 
can keep it alive just sending to the client some character from time to 
time during the computation flushing the channel after calling puts...

apt-cache-ng is repository cache for Debian/Ubuntu and has a web 
interface for the database diagnostics and maintenance. Some of the 
maintenance tasks take several minutes and the process never stops, but 
it sends some feedback to the client

let me know

  -- Massimo


On 11/22/2014 12:26 PM, Georgios Petasis wrote:
> Hi all,
>
> I have a quite lengthy computation exposed as a service through rivet.
> What I have observed, is that apache after some time,
> thinks that the process running my computation has stuck, and starts the
> same computation in a second process.
>
> Is there a way to avoid this?
>
> Regards,
>
> George
>

---------------------------------------------------------------------
To unsubscribe, e-mail: rivet-dev-unsubscribe@tcl.apache.org
For additional commands, e-mail: rivet-dev-help@tcl.apache.org