You are viewing a plain text version of this content. The canonical link for it is here.
Posted to modperl@perl.apache.org by Vincent Veyron <vv...@wanadoo.fr> on 2025/05/13 19:31:04 UTC
Return a 200 0K status and send the content of the request later
On Wed, 23 Apr 2025 15:06:44 -0500
Mithun Bhattacharya <mi...@gmail.com> wrote:
Hi Mithun,
In a recent thread, you wrote :
>
> mod_perl code is where you are sending the http return status to make sure
> the client doesn't timeout waiting for the server to respond.
>
Could you explain how this is done, I thought one had to send the content with the headers?
I have a function in my web application which allows users to download the entirety of their data in the form of a postgresql dump, wrapped in a zip file. The files are getting bigger, and take more time to generate and download, so I'm going to hit the timeout one day.
--
Bien à vous, Vincent Veyron
https://marica.fr/
Logiciel de gestion des sinistres assurances, des dossiers contentieux et des contrats pour le service juridique
Re: Return a 200 0K status and send the content of the request later
Posted by Vincent Veyron <vv...@wanadoo.fr>.
On Fri, 16 May 2025 01:11:48 +0200
Vincent Veyron <vv...@wanadoo.fr> wrote:
>
> I have a working prototype
>
Well, scratch that, it does work on a slow spinning disk, not on SSD :-\
Also, I'm getting something strange when testing for the presence of a file before removing it, as in :
if ( -e $r->document_root() . '/base/listing/' . $r->pnotes('session')->{_session_id} . '.tar.gz' ) {
warn 'tar file detected' ;
my @args = ( 'rm', $r->document_root() . '/base/listing/' . $r->pnotes('session')->{_session_id} . '.tar.gz' ) ;
system(@args) == 0 or warn "system @args failed: $?" ;
} else {
warn 'tar file NOT detected' ;
}
error_log shows :
tar file detected at /home/lib/Marica/Base/Rapports/export_raw_data.pm line 67.
tar file NOT detected at /home/lib/Marica/Base/Rapports/export_raw_data.pm line 75.
?
Not sure what's wrong, but I'll have to work on this some more.
--
vv.lists <vv...@wanadoo.fr>
Re: Return a 200 0K status and send the content of the request later
Posted by Vincent Veyron <vv...@wanadoo.fr>.
On Tue, 13 May 2025 22:57:05 -0400
Ed Sabol <ed...@gmail.com> wrote:
Hi Ed,
>
> Well, the standard way of doing that is you fork a child process to run the system command while the parent process loops and prints a period (or whatever you prefer) every 1 second (make sure you call flush() after printing!) while waiting for the child process to finish.
Indeed. As it happens, the Paris Perl Mongueurs (http://paris.mongueurs.net/) held their monthly meeting last Wednesday night, and I requested advice from the fine members of the club. They rapidly concluded that's what I should do.
I have a working prototype, it's modeled after https://stackoverflow.com/questions/471681/how-do-i-fork-properly-with-mod-perl2/. The page prints the asterisks while the fork builds the tar file, and adds the summary with file_size, interval, etc... when it detects the file.
It probably needs some polishing, critics welcome. Excuse my French.
https://pastebin.com/wn2Cw5KF
(not sure what line 114 'warn "started\n" ;' is for, since all STDs are redirected to /dev/null?)
>Lots of good examples of this can be found here:
>
> https://stackoverflow.com/questions/3193091/showing-progress-whilst-running-a-system-command-in-perl/
>
Thank you for this. Interesting answer with waitpid, but have not tested yet
> I also recommend that you still put an upper time limit on your subprocess and not just loop infinitely. 30 minutes to an hour seems like a reasonable choice. Depends on how big your tar files can be, of course.
Gee, just when I thought I was done with this ;-) But you're right
--
Bien à vous, Vincent Veyron
https://legalcase.libremen.com
Open source legal case, contract and insurance claims management software
Re: Return a 200 0K status and send the content of the request later
Posted by Vincent Veyron <vv...@wanadoo.fr>.
Ha! it turns out I had forgotten to deactivate a Filter. It works great.
I put it online, you can see it in action if you're curious.
1 - Use the login form in the demo section of the site in my sig
2 - navigate to https://legalcase.libremen.com/base/export_raw_data
Thanks to Ed and Mithun for the guidance.
--
Bien à vous, Vincent Veyron
https://legalcase.libremen.com
Open source legal case, contract and insurance claims management software
Re: Return a 200 0K status and send the content of the request later
Posted by Ed Sabol <ed...@gmail.com>.
On May 13, 2025, at 8:23 PM, Vincent Veyron <vv...@wanadoo.fr> wrote:
> Is there a way I could do :
>
> $r->print('<h3>Building tar file</h3>') ;
>
> every second or other until the tar file is built? I can't seem to think of one.
>
> My handler calls an sql script that dumps the database with system() :
>
> @args = ('psql', '-f', '/path/to/script/to/export_raw_data.sql', '-v', 'id_client=' . $r->pnotes('session')->{id_client}, '-v', 'database=' . $database, 'postgres') ;
>
> system(@args) == 0 or warn "system @args failed: $!" ;
>
> then waits for the resulting file before proceeding to compress and tar it, also with a system() call.
>
> How can I wrap this in a loop that re-sends the 'Building tar file' message until the file exists?
Well, the standard way of doing that is you fork a child process to run the system command while the parent process loops and prints a period (or whatever you prefer) every 1 second (make sure you call flush() after printing!) while waiting for the child process to finish. Lots of good examples of this can be found here:
https://stackoverflow.com/questions/3193091/showing-progress-whilst-running-a-system-command-in-perl/
The problem with this is that forking from a mod_perl process is more complicated than it is in a normal Perl process. The usual advice is to use a module called Apache2::SubProcess instead:
https://metacpan.org/pod/Apache2::SubProcess
But I don't know that that will work for what you want to do. I've never actually used that module, and I can't find an example in my Internet searches that shows the parent process doing something while the subprocess executes. Maybe try the example using detach_script.pl and then modify that detach_script.pl to build the tar file or execute psql or do whatever you want.
If that doesn't work, you can still fork from mod_perl, but you just need to know how to do it properly. Make sure you read this:
https://stackoverflow.com/questions/471681/how-do-i-fork-properly-with-mod-perl2/
Since I'm not convinced Apache2::SubProcess would actually work here, I would probably go with the "fork properly with mod_perl2" method, personally.
I also recommend that you still put an upper time limit on your subprocess and not just loop infinitely. 30 minutes to an hour seems like a reasonable choice. Depends on how big your tar files can be, of course.
When/if you get something along these lines working the way you want, please share your solution here on the mailing list for others to learn from.
Hope this helps,
Ed
Re: Return a 200 0K status and send the content of the request later
Posted by Vincent Veyron <vv...@wanadoo.fr>.
On Tue, 13 May 2025 21:45:35 -0500
Mithun Bhattacharya <mi...@gmail.com> wrote:
Hi Mithun,
> You should do the flush after the ContentType - content_type is header the
> H3 is body of the response or at least it should be.
>
I can do that, but then I need to flush again after $r->print('<h3>Building tar file</h3>') ;
Also, in mod_perl Developer's Cookbook (by Young, Lindner & Kobes), see 6.9 Flushing the Print Buffers, page 202. They give an example where they flush after $r->print, not before.
> - once rflush has happened the client side will wait forever without a
> timeout.
>
Thanks for the tip. That lead me to a solution without javascript, nice.
--
Bien à vous, Vincent Veyron
https://legalcase.libremen.com
Open source legal case, contract and insurance claims management software
Re: Return a 200 0K status and send the content of the request later
Posted by Mithun Bhattacharya <mi...@gmail.com>.
You should do the flush after the ContentType - content_type is header the
H3 is body of the response or at least it should be.
As for your second question, you are looking at a javascript based solution
- this isn't something a server side code can do. Look into a sleep
followed by a window.location.href. This wont help your long running
process though since the refresh will trigger another dump and tar. What
you need to do is send the header and flush followed by the system commands
- once rflush has happened the client side will wait forever without a
timeout.
On Tue, May 13, 2025 at 7:24 PM Vincent Veyron <vv...@wanadoo.fr> wrote:
> On Tue, 13 May 2025 14:57:15 -0500
> Mithun Bhattacharya <mi...@gmail.com> wrote:
>
> Hi Mithun,
>
> > Hence if you are writing a mod_perl handler for such an endpoint you
> return
> > success immediately and then start processing the request. In most cases
> > this is simply a matter of doing $r->print on the headers followed by
> print
> > of two new lines.
> >
>
> Nice!
>
> Your comment made me revisit mod_perl's documentation, and I found this :
>
>
> https://perl.apache.org/docs/2.0/user/coding/coding.html#Forcing_HTTP_Response_Headers_Out
>
> The following works, where exportation($r) is the long running sub that
> builds a tar file and returns a page with a link to the file, for the user
> to click. '<h3>Building tar file</h3>' does get sent first, then the rest
> of the response after completion of the sub.
>
>
> $r->content_type('text/html; charset=utf-8') ;
>
> $r->print('<h3>Building tar file</h3>') ;
>
> $r->rflush; # send the headers out
>
> $r->print( exportation($r) ) ;
>
> return Apache2::Const::OK ;
>
>
> This is great, I can thus send a message to the user, maybe with an
> estimated time of arrival. Thanks for the help.
>
> I have an another question, if I may.
>
> Is there a way I could do :
>
> $r->print('<h3>Building tar file</h3>') ;
>
> every second or other until the tar file is built? I can't seem to think
> of one.
>
> My handler calls an sql script that dumps the database with system() :
>
> @args = ('psql', '-f', '/path/to/script/to/export_raw_data.sql', '-v',
> 'id_client=' . $r->pnotes('session')->{id_client}, '-v', 'database=' .
> $database, 'postgres') ;
>
> system(@args) == 0 or warn "system @args failed: $!" ;
>
> then waits for the resulting file before proceeding to compress and tar
> it, also with a system() call.
>
> How can I wrap this in a loop that re-sends the 'Building tar file'
> message until the file exists?
>
>
>
> --
> Bien à vous, Vincent Veyron
>
> https://marica.fr/
> Logiciel de suivi des contentieux juridiques, des sinistres d'assurance et
> des contrats
>
Re: Return a 200 0K status and send the content of the request later
Posted by Vincent Veyron <vv...@wanadoo.fr>.
On Tue, 13 May 2025 14:57:15 -0500
Mithun Bhattacharya <mi...@gmail.com> wrote:
Hi Mithun,
> Hence if you are writing a mod_perl handler for such an endpoint you return
> success immediately and then start processing the request. In most cases
> this is simply a matter of doing $r->print on the headers followed by print
> of two new lines.
>
Nice!
Your comment made me revisit mod_perl's documentation, and I found this :
https://perl.apache.org/docs/2.0/user/coding/coding.html#Forcing_HTTP_Response_Headers_Out
The following works, where exportation($r) is the long running sub that builds a tar file and returns a page with a link to the file, for the user to click. '<h3>Building tar file</h3>' does get sent first, then the rest of the response after completion of the sub.
$r->content_type('text/html; charset=utf-8') ;
$r->print('<h3>Building tar file</h3>') ;
$r->rflush; # send the headers out
$r->print( exportation($r) ) ;
return Apache2::Const::OK ;
This is great, I can thus send a message to the user, maybe with an estimated time of arrival. Thanks for the help.
I have an another question, if I may.
Is there a way I could do :
$r->print('<h3>Building tar file</h3>') ;
every second or other until the tar file is built? I can't seem to think of one.
My handler calls an sql script that dumps the database with system() :
@args = ('psql', '-f', '/path/to/script/to/export_raw_data.sql', '-v', 'id_client=' . $r->pnotes('session')->{id_client}, '-v', 'database=' . $database, 'postgres') ;
system(@args) == 0 or warn "system @args failed: $!" ;
then waits for the resulting file before proceeding to compress and tar it, also with a system() call.
How can I wrap this in a loop that re-sends the 'Building tar file' message until the file exists?
--
Bien à vous, Vincent Veyron
https://marica.fr/
Logiciel de suivi des contentieux juridiques, des sinistres d'assurance et des contrats
Re: Return a 200 0K status and send the content of the request later
Posted by Mithun Bhattacharya <mi...@gmail.com>.
Http return codes are relevant only for request with fast response time. It
helps the requestor get a quick status 200 is all good, 3xx something
temporary, 4xx client has made some mistakes and 5xx server faulted.
Unfortunately this design doesn't map well to requests which take
significant execution time.
The easiest way to solve this is respond with 200 OK and then the body
tells you whether it succeeded or not. This is not something which is
defined in any place nor is it considered bad.
Hence if you are writing a mod_perl handler for such an endpoint you return
success immediately and then start processing the request. In most cases
this is simply a matter of doing $r->print on the headers followed by print
of two new lines.
On Tue, May 13, 2025, 2:33 PM Vincent Veyron <vv...@wanadoo.fr> wrote:
> On Wed, 23 Apr 2025 15:06:44 -0500
> Mithun Bhattacharya <mi...@gmail.com> wrote:
>
> Hi Mithun,
>
> In a recent thread, you wrote :
>
> >
> > mod_perl code is where you are sending the http return status to make
> sure
> > the client doesn't timeout waiting for the server to respond.
> >
>
> Could you explain how this is done, I thought one had to send the content
> with the headers?
>
> I have a function in my web application which allows users to download the
> entirety of their data in the form of a postgresql dump, wrapped in a zip
> file. The files are getting bigger, and take more time to generate and
> download, so I'm going to hit the timeout one day.
>
>
> --
> Bien à vous, Vincent Veyron
>
> https://marica.fr/
> Logiciel de gestion des sinistres assurances, des dossiers contentieux et
> des contrats pour le service juridique
>