You are viewing a plain text version of this content. The canonical link for it is here.
Posted to rivet-dev@tcl.apache.org by de...@thelanguageshop.org on 2023/04/20 19:25:07 UTC
REMOVE ME
REMOVE ME!!!
From: Scott Pitcher [mailto:scotty@svptechnicalservices.com.au]
Sent: Sunday, April 16, 2023 8:57 PM
To: Massimo MANGHI <ma...@unipr.it>; rivet-dev@tcl.apache.org
Subject: Re: R: Retrieving the body portion of a POST request
I've added a test to tests/post.test and .../post.rvt. I'm having a little trouble invoking the test scripts. I get an error in gethttpdconf:
scotty@server1:/home/scotty/src/rivet/tcl-rivet/tests$ ./runtests.tcl /usr/sbin/apache2
runtests.tcl is running with auto_path: ./apachetest /usr/share/tcltk/tcl8.6 /usr/share/tcltk /usr/lib /usr/local/lib/tcltk /usr/local/share/tcltk /usr/lib/tcltk/x86_64-linux-gnu /usr/lib/tcltk /usr/lib/tcltk/t
cl8.6
n arguments: 0
httpd_bin: /usr/sbin/apache2
httpd_args:
httpd_version: 2.4.25
mpm: prefork, bridge: default
Tests will be run against apache 2.4.25 version with the prefork module and the default bridge
DEBUG apachetest::gethttpdconf: options= binname=/usr/sbin/apache2 bin=file3
can't read "filename": no such variable
while executing
"file exists $filename"
(procedure "gethttpdconf" line 8)
invoked from within
"gethttpdconf $binname"
(procedure "determinemodules" line 4)
invoked from within
"determinemodules $binname"
(procedure "apachetest::makeconf" line 16)
invoked from within
"apachetest::makeconf server.conf $bridge {
LoadModule rivet_module [file join $CWD .. src/.libs mod_rivet[info sharedlibextension]]
# User an..."
(file "./runtests.tcl" line 124)
scotty@server1:/home/scotty/src/rivet/tcl-rivet/tests$
This seems to be because invoking apache2 with -V option fails due to missing environment configuration:
scotty@server1:~/src/rivet/tcl-rivet/tests$ /usr/sbin/apache2 -V
[Mon Apr 17 10:50:31.830756 2023] [core:warn] [pid 997] AH00111: Config variable ${APACHE_RUN_DIR} is not defined
apache2: Syntax error on line 80 of /etc/apache2/apache2.conf: DefaultRuntimeDir must be a valid directory, absolute or relative to ServerRoot
scotty@server1:~/src/rivet/tcl-rivet/tests$
Looking at /etc/init.d/apache2 there is a little work to do with running apache and setting up environment.
Perhaps it's better to run apache with "...-V -f server.conf"?
Your thoughts?
Kind regards,
Scott
On 14/4/23 18:05, Massimo MANGHI wrote:
Hi Scott
I get the point, if you don't have variable section in the POST request you can't access the data except if you call raw_post. Thank you for the patch, I will integrate it into Rivet. I guess this patch is worth a bugfix release of Rivet.
Since you're working on this type of POST requests do you have at hand some test cases we may add to our test suite? Thank you
-- Massimo
_____
Da: Scott Pitcher
Inviato: Martedì, 11 Aprile, 2023 12:06
A: Massimo MANGHI; rivet-dev@tcl.apache.org <ma...@tcl.apache.org>
Oggetto: Re: Retrieving the body portion of a POST request
Hi Massimo,
I tested "::rivet::var_post all" by inserting it into my client session code:
# DEBUG: What does var_post return?
if {[info exists env(REQUEST_METHOD)] && [string match "POST" $env(REQUEST_METHOD)]} {
::rivet::apache_log_error debug "clientSession: POST, ::rivet::var_post=[::rivet::var_post all]"
}
... but when a Fossil POST request arrives, this function returns a blank string. Probably because the POST request uses no variables. It's main purpose appears to be to pass Fossil the 100 or so bytes in the body of the POST request.
When I ask Fossil to do a http clone and observe it with Wireshark at the server I see the following:
Frame 261: 164 bytes on wire (1312 bits), 164 bytes captured (1312 bits) on interface 0
Ethernet II, Src: d8:5e:d3:0e:20:aa (d8:5e:d3:0e:20:aa), Dst: AsrockIn_e6:6b:35 (bc:5f:f4:e6:6b:35)
Internet Protocol Version 4, Src: 192.168.1.12, Dst: 192.168.1.1
Transmission Control Protocol, Src Port: 33820, Dst Port: 80, Seq: 180, Ack: 1, Len: 98
[2 Reassembled TCP Segments (277 bytes): #259(179), #261(98)]
[Frame: 259, payload: 0-178 (179 bytes)]
[Frame: 261, payload: 179-276 (98 bytes)]
[Segment count: 2]
[Reassembled TCP length: 277]
[Reassembled TCP Data: 504f5354202f666f73732f67697432666f7373696c2e6367...]
Hypertext Transfer Protocol
POST /foss/git2fossil.cgi HTTP/1.0\r\n
Host: [http://www.server2.svpts\r\n <http://www.server2.svpts/r/n> ]www.server2.svpts\r\n <http://www.server2.svpts/r/n>
User-Agent: Fossil/2.20 (2022-11-16 18:46:32 [210e89a059])\r\n
Content-Type: application/x-fossil\r\n
Content-Length: 98\r\n
\r\n
[Full request URI: http://www.server2.svpts/foss/git2fossil.cgi]
[HTTP request 1/1]
[Response in frame: 273]
File Data: 98 bytes
Media Type
Media type: application/x-fossil (98 bytes)
0000 00 00 00 61 78 da 05 c1 41 0a 80 20 10 05 d0 bd ...ax... A.. ....
0010 a7 18 68 1d cc fc 19 35 d7 69 f7 88 90 08 ca a2 ..h....5 .i......
0020 a2 f3 f7 de 75 cf eb 31 d3 b2 6f b5 bd fd 57 ef ....u..1 ..o...W.
0030 67 3b 1b 01 cc 4c 60 40 44 02 c9 60 41 e1 96 fd g;...L`@ D..`A...
0040 6c 95 94 c4 75 c4 d1 4f 01 69 9c 3c 52 56 68 41 l...u..O .i.<RVhA
0050 31 d3 1c 63 32 b3 11 52 82 64 5f 72 74 3f 31 1c 1..c2..R .d_rt?1.
0060 18 6d .m
The 98 bytes in the body of the request after the last "\r\n" is the part I'm trying to get hold of in my website code. With Rivet patched (attached to this email) the ::rivet::raw_post returns the 98 bytes as a Tcl byte array, and I'm able to write that directly through the pipe to Fossil's, and I had the first successful "fossil clone http..." via Rivet and the website last night. I was very happy!
For reference I've also attached my execcgi.tcl module. Much of it is notes and documentation I've gleaned and used as a guide. I've tested it with a few CGI programs not just Fossil. It's really a work in progress but also seems to doing the job now.
Thanks for your help :)
Kind regards,
Scott
On 11/4/23 02:31, Massimo Manghi wrote:
Hi Scott
Thank you for the extensive explanation of your problem. I'm certainly interested in reviewing your patch
Why using the usual command ::rivet::var_post (see https://tcl.apache.org/rivet/manual3.2/var.html) to retrieve the posted data didn't work for you?
-- Massimo
On 09/04/23 01:55, Scott Pitcher wrote:
Hi,
I've been writing a system for our website using Rivet as the foundation. I'd written a proc for running CGI programs from the client request, or, passing the request over. This works well but I'd recently started to incorporate some fossil archives in the website. The fossil CGI method worked except when I tried to clone the repo from a shell. Fossil would report an error code and clone would fail.
I examined the website logs and it Fossil makes a POST request to the CGI program. With wireshark I could see around 100 or so binary bytes in the body of the POST request, below the headers:
0000 50 4f 53 54 20 2f 66 6f 73 73 2f 75 6e 69 6d 61 POST /foss/unima 0010 6b 65 73 63 72 69 70 74 73 2e 63 67 69 2f 69 6e kescripts.cgi/in 0020 64 65 78 20 48 54 54 50 2f 31 2e 30 0d 0a 48 6f dex HTTP/1.0..Ho 0030 73 74 3a 20 77 77 77 2e 73 65 72 76 65 72 32 2e st: www.server2 <http://www.server2> . 0040 73 76 70 74 73 0d 0a 55 73 65 72 2d 41 67 65 6e svpts..User-Agen 0050 74 3a 20 46 6f 73 73 69 6c 2f 32 2e 32 30 20 28 t: Fossil/2.20 ( 0060 32 30 32 32 2d 31 31 2d 31 36 20 31 38 3a 34 36 2022-11-16 18:46 0070 3a 33 32 20 5b 32 31 30 65 38 39 61 30 35 39 5d :32 [210e89a059] 0080 29 0d 0a 43 6f 6e 74 65 6e 74 2d 54 79 70 65 3a )..Content-Type: 0090 20 61 70 70 6c 69 63 61 74 69 6f 6e 2f 78 2d 66 application/x-f 00a0 6f 73 73 69 6c 0d 0a 43 6f 6e 74 65 6e 74 2d 4c ossil..Content-L 00b0 65 6e 67 74 68 3a 20 31 30 30 0d 0a 0d 0a 00 00 ength: 100...... 00c0 00 61 78 da 05 c1 41 0a 80 20 10 00 c0 bb af 58 .axÚ.ÁA.. ..À»¯X 00d0 e8 1c ec 6e ba e9 d1 4c ff 21 21 11 94 86 45 ef è.ìnºéÑLÿ!!...Eï 00e0 6f e6 ee 79 bf 32 6c e7 51 ea 3b 7e a5 3f 47 ab oæîy¿2lçQê;~¥?G« 00f0 c0 8c 88 c0 c8 4c 44 02 64 b5 4c ac b6 b3 d5 02 À..ÀÈLD.dµL¬¶³Õ. 0100 13 90 1a 20 a1 5d 30 08 e9 e0 8c 8e 5e 4b 8c 22 ... ¡]0.éà..^K." 0110 c6 27 f4 f3 6c 53 10 71 d6 38 b7 92 fa 01 36 4f Æ'ôólS.qÖ8·.ú.6O 0120 18 9a
I was able to retrieve the headers using ::rivet::apache_table array_get headers_in. I wasn't able to retrieve the body of the POST request. I tried ::rivet::raw_post but it returns an empty string each time.
I checked out the Rivet source and delved into src/mod_rivet_ng and the Rivet_RawPost() function, and traced down to apache_request.h and ApacheRequest_get_raw_post().
The problem is that Rivet_RawPost() uses Tcl_NewStringObj() to pass the body back to Tcl, and given that the first 3 bytes are NUL, we end with an empty string. I changed it to Tcl_NewByteArrayObj() and added a length parameter which was passed by address all the way down to ApacheRequest_get_raw_post():
data = TclWeb_GetRawPost(private->req, &length); if (!data) { data = ""; } retval = Tcl_NewByteArrayObj((const unsigned char *)data, length); Tcl_SetObjResult(interp, retval);
... and added a raw_length field to the ApacheRequest structure:
const char* temp_dir; char* raw_post; /* Raw post data. */ int raw_length; request_rec* r; int nargs; } ApacheRequest;
The ApacheRequest_parse_urlencoded() function will fill this field in just after filling in the raw_post pointer, and util_read needed a length pointer parameter as well to get that working:
int ApacheRequest_parse_urlencoded(ApacheRequest *req) { ..... if ((rc = util_read(req, &data, &length)) != OK) { return rc; } if (data) { req->raw_post = (char*) data; /* Give people a way of getting at the raw data. */ req->raw_length = length; split_to_parms(req, data); } } return OK; }
I went to the effort because it's a simple change and I think it completes raw_post , if that function can return the complete body.
That's just my opinion. It's important to me because I have a website structure that can handle content, both HTTP and HTTPS at the same time, as well as user base with login and differentiation between local and remote access. It's been developed so we can include private content for our business only and host files for customers both public (e.g. documentation) and private (firmware and software downloads).
I really wanted to get fossil working under it as it streamlines that part of the development for us. I was going to give up and host the fossil archives separately but I persisted and came up with this. I still haven't had time to pass the POST request into fossil but I'll do so when I have time this weekend, but I do expect it to work, one way or another.
Anyway. That's my situation and if it's at all any use to anyone, I'd be more than happy to give you a patch. It's a really a minor set of changes but they give raw_post a bit more utility.
Further: I'd really like Rivet to have a kind of "run_cgi" method, but I've coded one in tcl at the moment and that's good for testing. At some stage I'll write a Rivet command for doing the same which might make environment setup and processing of the CGI output, headers etc, a bit more efficient.
Kind regards, Scott Pitcher
---------------------------------------------------------------------
To unsubscribe, e-mail: rivet-dev-unsubscribe@tcl.apache.org <ma...@tcl.apache.org>
For additional commands, e-mail: rivet-dev-help@tcl.apache.org <ma...@tcl.apache.org>
--
+--------------------------------------------------------------------------------+
| ... will make the wilderness a pool of water, the dry land springs of water ...|
+--------------------------------------------------------------------------------+
| Scott Pitcher Scotty@svptechnicalservices.com.au <ma...@svptechnicalservices.com.au> |
| +61-3-9008-5695 (B) +61-479-042-834 (M) |
| SVP Technical Services ABN 79 220 496 603 |
|PGP: https://gnupg.org/ |
|KEY: https://pgp.mit.edu/pks/lookup?op=vindex <https://pgp.mit.edu/pks/lookup?op=vindex&search=0xEDBE43639EFD884D> &search=0xEDBE43639EFD884D |
+--------------------------------------------------------------------------------+
DISCLAIMER:
The content of this email is confidential and strictly intended only for the
recipient(s) specified in the message. If you have received this message by
mistake it is strictly forbidden to keep this email in any form, electronic or
hard copy, or to share any part of it with any one else, without the prior,
written consent of the author. If you have received this message by mistake,
please let me know by replying to this email, and then destroy it and any copies.
SECURITY:
I prefer to send and receive confidential messages protected by PGP public key
encryption. If you are willing to or are interested in protecting our email
communication with PGP encryption, please refer to https://gnupg.org/.
--
+--------------------------------------------------------------------------------+
| ... will make the wilderness a pool of water, the dry land springs of water ...|
+--------------------------------------------------------------------------------+
| Scott Pitcher Scotty@svptechnicalservices.com.au <ma...@svptechnicalservices.com.au> |
| +61-3-9008-5695 (B) +61-479-042-834 (M) |
| SVP Technical Services ABN 79 220 496 603 |
|PGP: https://gnupg.org/ |
|KEY: https://pgp.mit.edu/pks/lookup?op=vindex <https://pgp.mit.edu/pks/lookup?op=vindex&search=0xEDBE43639EFD884D> &search=0xEDBE43639EFD884D |
+--------------------------------------------------------------------------------+
DISCLAIMER:
The content of this email is confidential and strictly intended only for the
recipient(s) specified in the message. If you have received this message by
mistake it is strictly forbidden to keep this email in any form, electronic or
hard copy, or to share any part of it with any one else, without the prior,
written consent of the author. If you have received this message by mistake,
please let me know by replying to this email, and then destroy it and any copies.
SECURITY:
I prefer to send and receive confidential messages protected by PGP public key
encryption. If you are willing to or are interested in protecting our email
communication with PGP encryption, please refer to https://gnupg.org/.
--
This email has been checked for viruses by AVG antivirus software.
www.avg.com