You are viewing a plain text version of this content. The canonical link for it is here.
Posted to rivet-dev@tcl.apache.org by Harald Oehlmann <ha...@elmicron.de> on 2013/05/10 15:30:18 UTC
Verify Link using http package
I am trying to verify some links of a form entered by a user by:
package require http
set requestHandle [::http::geturl $urlIn -validate 1 -timeout 5000]
Outside of a rivet script, this works well.
Inside a rivet script, I always run into the timeout.
Is the tclsh interpreter inside rivet not allowed to use sockets to call
out of the machine ?
This is CentOS 6.2 64 bit with recent Rivet and tcl8.6.0.
I have activated SE Linux but there are no warnings within the log, so
there should be no issue with that.
Thank you for any idea,
Harald
---------------------------------------------------------------------
To unsubscribe, e-mail: rivet-dev-unsubscribe@tcl.apache.org
For additional commands, e-mail: rivet-dev-help@tcl.apache.org
Re: Verify Link using http package
Posted by David Welton <da...@dedasys.com>.
> I just noticed, that there is also a user mailing list.
> I just subscribed, as this is more a user-type question.
I think that it would be realistic to shut that list down and direct
everything here. It's not like there's *that* much traffic.
--
David N. Welton
http://www.dedasys.com/
---------------------------------------------------------------------
To unsubscribe, e-mail: rivet-dev-unsubscribe@tcl.apache.org
For additional commands, e-mail: rivet-dev-help@tcl.apache.org
Re: Verify Link using http package
Posted by Harald Oehlmann <ha...@elmicron.de>.
Hi list,
Damon and Massimo already answered, but I still have the issue:
Page service stalls until apache restart on:
package require http
set requestHandle [::http::geturl $urlIn -validate 1 -timeout 5000]
Damon proposed to try without -timeout or -validate, which did not
change anything.
Stupid question:
Do I have to start the event queue with this command ?
I just noticed, that there is also a user mailing list.
I just subscribed, as this is more a user-type question.
Thank you,
Harald
Am 10.05.2013 15:30, schrieb Harald Oehlmann:
> I am trying to verify some links of a form entered by a user by:
>
> package require http
> set requestHandle [::http::geturl $urlIn -validate 1 -timeout 5000]
>
> Outside of a rivet script, this works well.
> Inside a rivet script, I always run into the timeout.
>
> Is the tclsh interpreter inside rivet not allowed to use sockets to call
> out of the machine ?
>
> This is CentOS 6.2 64 bit with recent Rivet and tcl8.6.0.
>
> I have activated SE Linux but there are no warnings within the log, so
> there should be no issue with that.
>
> Thank you for any idea,
> Harald
>
---------------------------------------------------------------------
To unsubscribe, e-mail: rivet-dev-unsubscribe@tcl.apache.org
For additional commands, e-mail: rivet-dev-help@tcl.apache.org
Re: Verify Link using http package
Posted by Harald Oehlmann <ha...@elmicron.de>.
Thank you, Damon and Massimo, for the answers.
At least it should work in principle and the idea is not stupid in
principle.
I do not want to do that in JavaScript on Client side as I want to be
shure that I get only valid Links in a data base and it might be sent by
a web service client and not by an interactive page.
I know that SE Linux may presevent this, but I see no SE-Linux errors...
The comments about -timeout and -head are correct.
When I remove them, the page with the verification never returns.
So it blocks and times out.
Normally, this looks like DNS issues, as this may take a long time...
So I tried to put an IP-Address instead of a server name, same result.
The relevant page is accessed via https and protected by a password via
html auth.
I have tried the same request on the same server outside of rivet in a
tcl shell and there it worked...
Thank you,
Harald
Am 10.05.2013 16:10, schrieb Damon Courtney:
> I use the HTTP package within Rivet to make outside requests all the time. Not sure what could be causing the problem. Try without the timeout and just see how long it takes to come back. The problem with the timeout feature of the http package is that it times out even if the request comes back and is delivering data.
>
> Also, -validate sends a HEAD request, which can actually be disabled on the web server. Not sure what kind of response you would get back in that case, but you might try just doing a regular request with no timeout from within Rivet to see what you get back.
>
> Damon
>
>
> On May 10, 2013, at 8:30 AM, Harald Oehlmann <ha...@elmicron.de> wrote:
>
>> I am trying to verify some links of a form entered by a user by:
>>
>> package require http
>> set requestHandle [::http::geturl $urlIn -validate 1 -timeout 5000]
>>
>> Outside of a rivet script, this works well.
>> Inside a rivet script, I always run into the timeout.
>>
>> Is the tclsh interpreter inside rivet not allowed to use sockets to call
>> out of the machine ?
>>
>> This is CentOS 6.2 64 bit with recent Rivet and tcl8.6.0.
>>
>> I have activated SE Linux but there are no warnings within the log, so
>> there should be no issue with that.
>>
>> Thank you for any idea,
>> Harald
---------------------------------------------------------------------
To unsubscribe, e-mail: rivet-dev-unsubscribe@tcl.apache.org
For additional commands, e-mail: rivet-dev-help@tcl.apache.org
Re: Verify Link using http package
Posted by Damon Courtney <da...@tclhome.com>.
I use the HTTP package within Rivet to make outside requests all the time. Not sure what could be causing the problem. Try without the timeout and just see how long it takes to come back. The problem with the timeout feature of the http package is that it times out even if the request comes back and is delivering data.
Also, -validate sends a HEAD request, which can actually be disabled on the web server. Not sure what kind of response you would get back in that case, but you might try just doing a regular request with no timeout from within Rivet to see what you get back.
Damon
On May 10, 2013, at 8:30 AM, Harald Oehlmann <ha...@elmicron.de> wrote:
> I am trying to verify some links of a form entered by a user by:
>
> package require http
> set requestHandle [::http::geturl $urlIn -validate 1 -timeout 5000]
>
> Outside of a rivet script, this works well.
> Inside a rivet script, I always run into the timeout.
>
> Is the tclsh interpreter inside rivet not allowed to use sockets to call
> out of the machine ?
>
> This is CentOS 6.2 64 bit with recent Rivet and tcl8.6.0.
>
> I have activated SE Linux but there are no warnings within the log, so
> there should be no issue with that.
>
> Thank you for any idea,
> Harald
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: rivet-dev-unsubscribe@tcl.apache.org
> For additional commands, e-mail: rivet-dev-help@tcl.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: rivet-dev-unsubscribe@tcl.apache.org
For additional commands, e-mail: rivet-dev-help@tcl.apache.org
Re: Verify Link using http package
Posted by Massimo Manghi <ma...@unipr.it>.
On 05/10/2013 03:30 PM, Harald Oehlmann wrote:
> I am trying to verify some links of a form entered by a user by:
>
> package require http
> set requestHandle [::http::geturl $urlIn -validate 1 -timeout 5000]
>
> Outside of a rivet script, this works well.
> Inside a rivet script, I always run into the timeout.
>
Is the server logging some request in the access log file? I haven't
tested sockets within Rivet, I did test threads (which might be useful
in this case) but not sockets. We don't remove commands from the slave
intepreters and if SeparateVirtualInterps is off we don't even run slave
intepreters.
Why are you testing a form from within a Rivet script? Maybe JavaScript
might work out a better solution?
> Is the tclsh interpreter inside rivet not allowed to use sockets to call
> out of the machine ?
>
> This is CentOS 6.2 64 bit with recent Rivet and tcl8.6.0.
>
> I have activated SE Linux but there are no warnings within the log, so
> there should be no issue with that.
>
> Thank you for any idea,
> Harald
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: rivet-dev-unsubscribe@tcl.apache.org
> For additional commands, e-mail: rivet-dev-help@tcl.apache.org
>
--
-- Massimo Manghi
Dipartimento di Neuroscienze
Unità di Biofisica e Fisica Sanitaria
via Volturno 39
43125 Parma
---------------------------------------------------------------------
To unsubscribe, e-mail: rivet-dev-unsubscribe@tcl.apache.org
For additional commands, e-mail: rivet-dev-help@tcl.apache.org
Re: Verify Link using http package
Posted by Massimo Manghi <mx...@apache.org>.
Hi Harald
I read the thread you started on clt and it's definitely interesting.
Thanks for involving Alexandre on the problem, I'd like to do some
experiment on it but I won't be able to do it soon for various reasons,
including a busy time at work and children management (real human
children I mean, not a product of 'fork'....)
It's puzzling the notifier works when apache runs as a single process.
Unfortunately we don't have a working Windows port, I would like to know
if it's a problem arising from some Linux/Tcl interaction in the process
management. I will take a look at the prefork MPM code to look for a
clue. Combining our efforts with Alex experience might work out
something....
-- Massimo
On 07/02/2013 10:54 AM, Harald Oehlmann wrote:
> Dear folks,
>
> IMHO, the missing fileevent is quite an issue in Rivet.
>
> Alexandre Ferrieux looked to the strace log and gave good pointers where
> to look.
>
> I would anybody having more understanding of the Rivet C level to look
> into his comments and maybee find the issue.
>
> I would appreciate this.
>
> Thank you all,
> Harald
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: rivet-dev-unsubscribe@tcl.apache.org
> For additional commands, e-mail: rivet-dev-help@tcl.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: rivet-dev-unsubscribe@tcl.apache.org
For additional commands, e-mail: rivet-dev-help@tcl.apache.org
Re: Verify Link using http package
Posted by Harald Oehlmann <ha...@elmicron.de>.
Dear folks,
IMHO, the missing fileevent is quite an issue in Rivet.
Alexandre Ferrieux looked to the strace log and gave good pointers where
to look.
I would anybody having more understanding of the Rivet C level to look
into his comments and maybee find the issue.
I would appreciate this.
Thank you all,
Harald
---------------------------------------------------------------------
To unsubscribe, e-mail: rivet-dev-unsubscribe@tcl.apache.org
For additional commands, e-mail: rivet-dev-help@tcl.apache.org
Re: Verify Link using http package
Posted by Harald Oehlmann <ha...@elmicron.de>.
Hi Massimo,
thank you for the message.
At least, I have got the information, that it is not a new bug in
tcl8.6.0 or of my setup. I suppose, you use tcl 8.5.10-14.
thank you for the idea.
Unfortunately, the time the process takes should be limited, and the
operations should be aborded aber a given time.
This is not possible with the linear program.
And it is a pity that such a central function does not work...
I may try an strace, but I am not at all a wizard there....
Regards,
Harald
Am 28.06.2013 03:03, schrieb Massimo Manghi:
> Like in your case it's not working for me also. I'm afraid the answer
> should be searched in the docs or even the code in the way the event
> queue and other subsystems are prepared by the shell. After all tclsh
> (and likewise wish) are more the programs wrapping the interpreter for
> the command line in the way we do within mod_rivet.
>
> I have a workaround that uses Tcl threads and it works for me and I'm
> confident it could work for you provided your rivet build was done with
> a threaded Tcl library.
>
> Here is my proof of concept: a pool of 10 threads (you probably don't
> need that many) is started and then sits waiting for some task to be
> signaled through the ::thread::send call. Then the 'signal_threads'
> procedure is called and each thread is requested to run the procedure
> passed as second argument (my example simply sent a procedure that let a
> thread wait for a random interval thus emulating different tasks taking
> different times for their execution). I posted this procedure months ago
> to clarify that in principle mpm_prefork doesn't imply you can't use Tcl
> threads.
>
> I added your synchronous code and it worked. Notice that with this
> approach you may be fetching data from multiple data sources at the same
> time.
>
> caveats: you can call Rivet commands within a Tcl thread. Rivet commands
> are not recognized. Perhaps a package require should be issued I'm not
> keen to mix Tcl threads and the Apache framework: I suspect they
> wouldn't get along...
>
> -- Massimo
>
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: rivet-dev-unsubscribe@tcl.apache.org
> For additional commands, e-mail: rivet-dev-help@tcl.apache.org
>
--
ELMICRON Dr. Harald Oehlmann GmbH
Koesener Str. 85
06618 Naumburg
Germany
Phone: +49 (0)3445 78112-0
Fax: +49 (0)3445 78112-19
www.Elmicron.de
German legal references:
Geschaeftsfuehrer: Dr. Harald Oehlmann, Jens Oehlmann
UST Nr. / VAT ID No.: DE206105272
HRB 212803 Stendal
---------------------------------------------------------------------
To unsubscribe, e-mail: rivet-dev-unsubscribe@tcl.apache.org
For additional commands, e-mail: rivet-dev-help@tcl.apache.org
Re: Verify Link using http package
Posted by Massimo Manghi <mx...@apache.org>.
Like in your case it's not working for me also. I'm afraid the answer
should be searched in the docs or even the code in the way the event
queue and other subsystems are prepared by the shell. After all tclsh
(and likewise wish) are more the programs wrapping the interpreter for
the command line in the way we do within mod_rivet.
I have a workaround that uses Tcl threads and it works for me and I'm
confident it could work for you provided your rivet build was done with
a threaded Tcl library.
Here is my proof of concept: a pool of 10 threads (you probably don't
need that many) is started and then sits waiting for some task to be
signaled through the ::thread::send call. Then the 'signal_threads'
procedure is called and each thread is requested to run the procedure
passed as second argument (my example simply sent a procedure that let a
thread wait for a random interval thus emulating different tasks taking
different times for their execution). I posted this procedure months ago
to clarify that in principle mpm_prefork doesn't imply you can't use Tcl
threads.
I added your synchronous code and it worked. Notice that with this
approach you may be fetching data from multiple data sources at the same
time.
caveats: you can call Rivet commands within a Tcl thread. Rivet commands
are not recognized. Perhaps a package require should be issued I'm not
keen to mix Tcl threads and the Apache framework: I suspect they
wouldn't get along...
-- Massimo
Re: Verify Link using http package
Posted by Harald Oehlmann <ha...@elmicron.de>.
Just to inform you, that the not fireing fileevent continuous on clt:
https://groups.google.com/forum/#!topic/comp.lang.tcl/Ja01kv75tVk
-Harald
---------------------------------------------------------------------
To unsubscribe, e-mail: rivet-dev-unsubscribe@tcl.apache.org
For additional commands, e-mail: rivet-dev-help@tcl.apache.org
Re: Verify Link using http package
Posted by Harald Oehlmann <ha...@elmicron.de>.
Hello Massimo,
with the -X switch, the issue does not arise, fileevents fire.
Without, they don't.
Within tclsh, anything works fine.
Within Rivet, the event loop itself works fine.
Only fileevents never fire, when "-X" is not specified to httpd.
Thank you,
Harald
Am 28.06.2013 11:40, schrieb Massimo Manghi:
>
>
> On 06/28/2013 10:29 AM, Harald Oehlmann wrote:
>> Hello List,
>>
>> I have tried:
>> strace -o l.txt /usr/sbin/httpd -X
>>
>> Then, the issue does not arise and the script works well.
>>
>> I have tried:
>> strace -o l.txt /usr/sbin/httpd
>> This does not help, as apache goes in daemon mode.
>
> did you run httpd with the -X switch?
>
> I would rather check with tclsh to see what you have to do in order to
> have the event loop working.
>
>
--
ELMICRON Dr. Harald Oehlmann GmbH
Koesener Str. 85
06618 Naumburg
Germany
Phone: +49 (0)3445 78112-0
Fax: +49 (0)3445 78112-19
www.Elmicron.de
German legal references:
Geschaeftsfuehrer: Dr. Harald Oehlmann, Jens Oehlmann
UST Nr. / VAT ID No.: DE206105272
HRB 212803 Stendal
---------------------------------------------------------------------
To unsubscribe, e-mail: rivet-dev-unsubscribe@tcl.apache.org
For additional commands, e-mail: rivet-dev-help@tcl.apache.org
Re: Verify Link using http package
Posted by Massimo Manghi <ma...@unipr.it>.
On 06/28/2013 10:29 AM, Harald Oehlmann wrote:
> Hello List,
>
> I have tried:
> strace -o l.txt /usr/sbin/httpd -X
>
> Then, the issue does not arise and the script works well.
>
> I have tried:
> strace -o l.txt /usr/sbin/httpd
> This does not help, as apache goes in daemon mode.
did you run httpd with the -X switch?
I would rather check with tclsh to see what you have to do in order to
have the event loop working.
--
-- Massimo Manghi
Dipartimento di Neuroscienze
Unità di Biofisica e Fisica Sanitaria
via Volturno 39
43125 Parma
---------------------------------------------------------------------
To unsubscribe, e-mail: rivet-dev-unsubscribe@tcl.apache.org
For additional commands, e-mail: rivet-dev-help@tcl.apache.org
Re: Verify Link using http package
Posted by Harald Oehlmann <ha...@elmicron.de>.
Hello List,
I have tried:
strace -o l.txt /usr/sbin/httpd -X
Then, the issue does not arise and the script works well.
I have tried:
strace -o l.txt /usr/sbin/httpd
This does not help, as apache goes in daemon mode.
So it must be an issue with the deamon mode etc.
Is there any way to debug this ?
Thank you,
Harald
Am 27.06.2013 16:11, schrieb Harald Oehlmann:
> Hi,
>
> I tried to isolate the issue that a ::http::geturl call hangs in rivet
> but works in tclsh.
>
> Could you please verify the following scripts to check if this is the
> same for you ?
>
> --Issue--
>
> I tried to isolate the issue, that the following file works well when
> executed in a tcl8.6 shell and lets an apache thread hang:
> --test_linkcheck.tcl---
> puts [package require http]<br/>
> if { [catch {
> set h [::http::geturl "http://www.elmicron.de/test.txt"]
> } err] } {
> puts " unknown url<br/>"
> } else {
> puts [::http::status $h]<br/>
> ::http::cleanup $h
> }
> --EOF--
>
> % tclsh8.6 test_linkcheck.tcl
> 2.8.7<br/>
> ok<br/>
> %
>
> The file is saved in "/var/www/http" of my test server, and a request of
> the address: "//centostest/test_linkcheck.tcl" lets the client wait
> until I restart the apache server.
>
> An additional option "-timeout 1000" to geturl would let stop geturl in
> a timeout.
>
> The file "http://www.elmicron.de/test.txt" is a 12 byte test file.
>
> ----
>
> Tests I have done:
>
> --T1: Try synchronous socket--
>
> The following script tries to get the data by a synchronous socket:
>
> --test_socket.tcl--
> set h [socket www.elmicron.de 80]
> fconfigure $h -translation crlf -buffering line -blocking 1
> puts $h "GET /test.txt HTTP/1.0\nHost: www.elmicron.de\n"
> while { ! [eof $h] } {
> set data [gets $h]
> if {$data ne ""} {
> puts $data
> }
> }
> close $h
> --eof--
>
> This works well in rivet and in tclsh
>
> --T2: Try asynchronous socket--
>
> The following script loads the data by an asynchoneous method:
>
> --test_socket_async.tcl--
> proc sockin {} {
> global h
> puts [gets $h]
> if {[eof $h]} {
> close $h
> set h ""
> }
> }
>
> set h [socket -async www.elmicron.de 80]
> fconfigure $h -translation crlf -buffering line -blocking 0
> puts $h "GET /test.txt HTTP/1.0\nHost: www.elmicron.de\n"
> fileevent $h readable sockin
> vwait h
> --eof--
>
> This works well in tcl86 and hangs the browser when executed with rivet.
>
> --T3: Event loop--
>
> So I thought the issue is the event loop.
> So I tried a simple event loop example which stays for 1 second in the
> event loop:
>
> --test_vwait.tcl--
> puts "wait 1 sec"
> set h 0
> after 1000 "set h 1"
> vwait h
> puts end
> --eof--
>
> This works well in tclsh and rivet.
>
> ------
>
> Is this a bug or a feature ?
> Do you also see what I see ?
> Are my tests just stupid ?
>
> Thank you for any ideas,
> Harald
>
---------------------------------------------------------------------
To unsubscribe, e-mail: rivet-dev-unsubscribe@tcl.apache.org
For additional commands, e-mail: rivet-dev-help@tcl.apache.org
Re: Verify Link using http package
Posted by Harald Oehlmann <ha...@elmicron.de>.
Hi,
I tried to isolate the issue that a ::http::geturl call hangs in rivet
but works in tclsh.
Could you please verify the following scripts to check if this is the
same for you ?
--Issue--
I tried to isolate the issue, that the following file works well when
executed in a tcl8.6 shell and lets an apache thread hang:
--test_linkcheck.tcl---
puts [package require http]<br/>
if { [catch {
set h [::http::geturl "http://www.elmicron.de/test.txt"]
} err] } {
puts " unknown url<br/>"
} else {
puts [::http::status $h]<br/>
::http::cleanup $h
}
--EOF--
% tclsh8.6 test_linkcheck.tcl
2.8.7<br/>
ok<br/>
%
The file is saved in "/var/www/http" of my test server, and a request of
the address: "//centostest/test_linkcheck.tcl" lets the client wait
until I restart the apache server.
An additional option "-timeout 1000" to geturl would let stop geturl in
a timeout.
The file "http://www.elmicron.de/test.txt" is a 12 byte test file.
----
Tests I have done:
--T1: Try synchronous socket--
The following script tries to get the data by a synchronous socket:
--test_socket.tcl--
set h [socket www.elmicron.de 80]
fconfigure $h -translation crlf -buffering line -blocking 1
puts $h "GET /test.txt HTTP/1.0\nHost: www.elmicron.de\n"
while { ! [eof $h] } {
set data [gets $h]
if {$data ne ""} {
puts $data
}
}
close $h
--eof--
This works well in rivet and in tclsh
--T2: Try asynchronous socket--
The following script loads the data by an asynchoneous method:
--test_socket_async.tcl--
proc sockin {} {
global h
puts [gets $h]
if {[eof $h]} {
close $h
set h ""
}
}
set h [socket -async www.elmicron.de 80]
fconfigure $h -translation crlf -buffering line -blocking 0
puts $h "GET /test.txt HTTP/1.0\nHost: www.elmicron.de\n"
fileevent $h readable sockin
vwait h
--eof--
This works well in tcl86 and hangs the browser when executed with rivet.
--T3: Event loop--
So I thought the issue is the event loop.
So I tried a simple event loop example which stays for 1 second in the
event loop:
--test_vwait.tcl--
puts "wait 1 sec"
set h 0
after 1000 "set h 1"
vwait h
puts end
--eof--
This works well in tclsh and rivet.
------
Is this a bug or a feature ?
Do you also see what I see ?
Are my tests just stupid ?
Thank you for any ideas,
Harald
---------------------------------------------------------------------
To unsubscribe, e-mail: rivet-dev-unsubscribe@tcl.apache.org
For additional commands, e-mail: rivet-dev-help@tcl.apache.org