You are viewing a plain text version of this content. The canonical link for it is here.
Posted to rivet-dev@tcl.apache.org by Massimo Manghi <ma...@unipr.it> on 2023/05/04 06:49:19 UTC
Re: Retrieving the body portion of a POST request
Dear Scott,
thank you. The new improvements to the test scripts were committed to
rivet. I plan to propose a new rivet release soon
-- Massimo
On 30/04/23 13:08, Scott Pitcher wrote:
> I've finished adding a raw_post test to the test suite. Patch is attached.
>
> There is a small change to tests/apachetest/apachetest.tcl. I got it to
> read IncludeOptional config files and to respect the ServerRoot, which
> helped load config.
>
> Also changed tests/rivet.test to get it to look in the environment for
> TestList and the 3 testgroup? variables, so I could override them more
> easily.
>
> I added the new test on the end of post.test:
>
>> ::tcltest::test postvariables-6.1 {::rivet::raw_post} {
>> set payload "\x00\x00\x00\x30\x31\x32\x33\x34\x35\x36\x37\x38\x39"
>> set page [::http::geturl "${urlbase}post.rvt" \
>> -query $payload \
>> -type application/octet-stream]
>> regexp -line {^\[::rivet::raw_post\] =
>> 00000030313233343536373839$} [ ::http::data $page ] match
>> set match
>> } {[::rivet::raw_post] = 00000030313233343536373839}
> And post.rvt has a condition check to handle that and return the
> raw_post result. The leading zeros in the test data will cause the test
> to fail without the bug fix (will return an empty string).
>
> Test passed fine here.
>
> The post.rvt and post.test changes are the important ones and you can
> discard the other changes to apachetest.tcl and rivet.test. I had to add
> those to get the tests to run.
>
> Kind regards,
>
> Scott
>
>
>
>
>
>
> On 27/4/23 09:53, Scott Pitcher wrote:
>> Thank you. That's good news. I haven't finished with the unit test
>> yet. Will come back to it this weekend.
>> Kind regards,
>> Scott
>>
>>
>> On 27/4/23 09:01, Massimo MANGHI wrote:
>>> Your patch fixing the issue with the ::rivet::raw_post command was
>>> commited in the master branch of mod_rivet.
>>>
>>> -- Massimo
>>>
>>>
>>> ------------------------------------------------------------------------
>>> *Da:* Scott Pitcher
>>> *Inviato:* Giovedì, 20 Aprile, 2023 14:59
>>> *A:* Massimo MANGHI; rivet-dev@tcl.apache.org
>>> *Oggetto:* Re: R: R: Retrieving the body portion of a POST request
>>>
>>> No problem at all. I'm working my way through it anyway.
>>>
>>> I found the authz_core_module module error is caused by the
>>> IncludeOptional directives in my /etc/apache2/apache2.conf, which are
>>> missed by the apachetest::getallincludes proc. It regexp's against
>>> Include but not IncludeOptional. I'll rework it to accept both.
>>>
>>> Kind regards,
>>>
>>> Scott
>>>
>>>
>>>
>>> On 20/4/23 19:16, Massimo MANGHI wrote:
>>>> Hi Scott
>>>>
>>>> sorry for reacting to your message so late. If you test rivet
>>>> against your Debian installed apache webserver you have to tweak
>>>> your configuration in order to have all the required modules loaded.
>>>> If you don't want to change it you have to build your own apache
>>>> webserver and have it installed in a custom directory with all the
>>>> configuration (make install does it for you when you build the
>>>> apache HTTP web server)
>>>>
>>>> runtests.tcl run all the tests. If you need to work on your own
>>>> tests then you may disable temporarily the others by putting in the
>>>> TestList variable in tests/rivet.test only what you're actually
>>>> working on
>>>>
>>>> -- Massimo
>>>>
>>>> ------------------------------------------------------------------------
>>>> *Da:* Scott Pitcher
>>>> *Inviato:* Giovedì, 20 Aprile, 2023 05:42
>>>> *A:* Massimo MANGHI; rivet-dev@tcl.apache.org
>>>> <ma...@tcl.apache.org>
>>>> *Oggetto:* Re: R: Retrieving the body portion of a POST request
>>>>
>>>> I've managed to get beyond this problem by upgrading Apache from
>>>> 2.4.25 to 2.4.46 via Debian stretch-backports-sloppy.
>>>>
>>>> Now I have an error -
>>>>> checking /etc/apache2/apache2.conf
>>>>> including files from ports.conf
>>>>> check conf line for authz_core_module module...No LoadModule line
>>>>> for authz_core_module!
>>>>> while executing
>>>>> "error "No LoadModule line for $mod\!""
>>>>> (procedure "getloadmodules" line 12)
>>>>> invoked from within ...
>>>>> scotty@server1:/home/scotty/src/rivet/tcl-rivet/tests$
>>>>
>>>> But I'll work on this when I have some time free later this week.
>>>>
>>>>
>>>> Kind regards,
>>>>
>>>> Scott
>>>>
>>>>
>>>>
>>>>
>>>> On 17/4/23 10:57, Scott Pitcher wrote:
>>>>> I've added a test to tests/post.test and .../post.rvt. I'm having a
>>>>> little trouble invoking the test scripts. I get an error in
>>>>> gethttpdconf:
>>>>>> scotty@server1:/home/scotty/src/rivet/tcl-rivet/tests$
>>>>>> ./runtests.tcl /usr/sbin/apache2
>>>>>> runtests.tcl is running with auto_path: ./apachetest
>>>>>> /usr/share/tcltk/tcl8.6 /usr/share/tcltk /usr/lib
>>>>>> /usr/local/lib/tcltk /usr/local/share/tcltk
>>>>>> /usr/lib/tcltk/x86_64-linux-gnu /usr/lib/tcltk /usr/lib/tcltk/t
>>>>>> cl8.6
>>>>>> n arguments: 0
>>>>>> httpd_bin: /usr/sbin/apache2
>>>>>> httpd_args:
>>>>>> httpd_version: 2.4.25
>>>>>> mpm: prefork, bridge: default
>>>>>> Tests will be run against apache 2.4.25 version with the prefork
>>>>>> module and the default bridge
>>>>>> DEBUG apachetest::gethttpdconf: options= binname=/usr/sbin/apache2
>>>>>> bin=file3
>>>>>> can't read "filename": no such variable
>>>>>> while executing
>>>>>> "file exists $filename"
>>>>>> (procedure "gethttpdconf" line 8)
>>>>>> invoked from within
>>>>>> "gethttpdconf $binname"
>>>>>> (procedure "determinemodules" line 4)
>>>>>> invoked from within
>>>>>> "determinemodules $binname"
>>>>>> (procedure "apachetest::makeconf" line 16)
>>>>>> invoked from within
>>>>>> "apachetest::makeconf server.conf $bridge {
>>>>>> LoadModule rivet_module [file join $CWD .. src/.libs
>>>>>> mod_rivet[info sharedlibextension]]
>>>>>>
>>>>>> # User an..."
>>>>>> (file "./runtests.tcl" line 124)
>>>>>> scotty@server1:/home/scotty/src/rivet/tcl-rivet/tests$
>>>>>>
>>>>>
>>>>> This seems to be because invoking apache2 with -V option fails due
>>>>> to missing environment configuration:
>>>>>
>>>>>> scotty@server1:~/src/rivet/tcl-rivet/tests$ /usr/sbin/apache2 -V
>>>>>> [Mon Apr 17 10:50:31.830756 2023] [core:warn] [pid 997] AH00111:
>>>>>> Config variable ${APACHE_RUN_DIR} is not defined
>>>>>> apache2: Syntax error on line 80 of /etc/apache2/apache2.conf:
>>>>>> DefaultRuntimeDir must be a valid directory, absolute or relative
>>>>>> to ServerRoot
>>>>>> scotty@server1:~/src/rivet/tcl-rivet/tests$
>>>>>
>>>>>
>>>>> Looking at /etc/init.d/apache2 there is a little work to do with
>>>>> running apache and setting up environment.
>>>>>
>>>>> Perhaps it's better to run apache with "...-V -f server.conf"?
>>>>>
>>>>>
>>>>> Your thoughts?
>>>>>
>>>>> Kind regards,
>>>>> Scott
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On 14/4/23 18:05, Massimo MANGHI wrote:
>>>>>> Hi Scott
>>>>>>
>>>>>> I get the point, if you don't have variable section in the POST
>>>>>> request you can't access the data except if you call raw_post.
>>>>>> Thank you for the patch, I will integrate it into Rivet. I guess
>>>>>> this patch is worth a bugfix release of Rivet.
>>>>>>
>>>>>> Since you're working on this type of POST requests do you have at
>>>>>> hand some test cases we may add to our test suite? Thank you
>>>>>>
>>>>>> -- Massimo
>>>>>>
>>>>>>
>>>>>> ------------------------------------------------------------------------
>>>>>> *Da:* Scott Pitcher
>>>>>> *Inviato:* Martedì, 11 Aprile, 2023 12:06
>>>>>> *A:* Massimo MANGHI; rivet-dev@tcl.apache.org
>>>>>> <ma...@tcl.apache.org>
>>>>>> *Oggetto:* Re: Retrieving the body portion of a POST request
>>>>>>
>>>>>> Hi Massimo,
>>>>>>
>>>>>> I tested "::rivet::var_post all" by inserting it into my client
>>>>>> session code:
>>>>>>
>>>>>>> # DEBUG: What does var_post return? if {[info exists
>>>>>>> env(REQUEST_METHOD)] && [string match "POST"
>>>>>>> $env(REQUEST_METHOD)]} { ::rivet::apache_log_error debug
>>>>>>> "clientSession: POST, ::rivet::var_post=[::rivet::var_post all]"
>>>>>>> }
>>>>>> ... but when a Fossil POST request arrives, this function returns
>>>>>> a blank string. Probably because the POST request uses no
>>>>>> variables. It's main purpose appears to be to pass Fossil the 100
>>>>>> or so bytes in the body of the POST request.
>>>>>>
>>>>>> When I ask Fossil to do a http clone and observe it with Wireshark
>>>>>> at the server I see the following:
>>>>>>
>>>>>>> Frame 261: 164 bytes on wire (1312 bits), 164 bytes captured
>>>>>>> (1312 bits) on interface 0
>>>>>>> Ethernet II, Src: d8:5e:d3:0e:20:aa (d8:5e:d3:0e:20:aa), Dst:
>>>>>>> AsrockIn_e6:6b:35 (bc:5f:f4:e6:6b:35)
>>>>>>> Internet Protocol Version 4, Src: 192.168.1.12, Dst: 192.168.1.1
>>>>>>> Transmission Control Protocol, Src Port: 33820, Dst Port: 80,
>>>>>>> Seq: 180, Ack: 1, Len: 98
>>>>>>> [2 Reassembled TCP Segments (277 bytes): #259(179), #261(98)]
>>>>>>> [Frame: 259, payload: 0-178 (179 bytes)]
>>>>>>> [Frame: 261, payload: 179-276 (98 bytes)]
>>>>>>> [Segment count: 2]
>>>>>>> [Reassembled TCP length: 277]
>>>>>>> [Reassembled TCP Data:
>>>>>>> 504f5354202f666f73732f67697432666f7373696c2e6367...]
>>>>>>> Hypertext Transfer Protocol
>>>>>>> POST /foss/git2fossil.cgi HTTP/1.0\r\n
>>>>>>> Host:
>>>>>>> [[[http://www.server2.svpts\r\n]http://www.server2.svpts\r\n][http://www.server2.svpts\r\n]http://www.server2.svpts\r\n]www.server2.svpts\r\n
>>>>>>> User-Agent: Fossil/2.20 (2022-11-16 18:46:32 [210e89a059])\r\n
>>>>>>> Content-Type: application/x-fossil\r\n
>>>>>>> Content-Length: 98\r\n
>>>>>>> \r\n
>>>>>>> [Full request URI:
>>>>>>> http://www.server2.svpts/foss/git2fossil.cgi
>>>>>>> <http://www.server2.svpts/foss/git2fossil.cgi>]
>>>>>>> [HTTP request 1/1]
>>>>>>> [Response in frame: 273]
>>>>>>> File Data: 98 bytes
>>>>>>> Media Type
>>>>>>> Media type: application/x-fossil (98 bytes)
>>>>>>> 0000 00 00 00 61 78 da 05 c1 41 0a 80 20 10 05 d0 bd ...ax...
>>>>>>> A.. ....
>>>>>>> 0010 a7 18 68 1d cc fc 19 35 d7 69 f7 88 90 08 ca a2 ..h....5
>>>>>>> .i......
>>>>>>> 0020 a2 f3 f7 de 75 cf eb 31 d3 b2 6f b5 bd fd 57 ef ....u..1
>>>>>>> ..o...W.
>>>>>>> 0030 67 3b 1b 01 cc 4c 60 40 44 02 c9 60 41 e1 96 fd g;...L`@
>>>>>>> D..`A...
>>>>>>> 0040 6c 95 94 c4 75 c4 d1 4f 01 69 9c 3c 52 56 68 41 l...u..O
>>>>>>> .i.<RVhA
>>>>>>> 0050 31 d3 1c 63 32 b3 11 52 82 64 5f 72 74 3f 31 1c 1..c2..R
>>>>>>> .d_rt?1.
>>>>>>> 0060 18 6d .m
>>>>>>
>>>>>> The 98 bytes in the body of the request after the last "\r\n" is
>>>>>> the part I'm trying to get hold of in my website code. With Rivet
>>>>>> patched (attached to this email) the ::rivet::raw_post returns the
>>>>>> 98 bytes as a Tcl byte array, and I'm able to write that directly
>>>>>> through the pipe to Fossil's, and I had the first successful
>>>>>> "fossil clone http..." via Rivet and the website last night. I was
>>>>>> very happy!
>>>>>>
>>>>>> For reference I've also attached my execcgi.tcl module. Much of it
>>>>>> is notes and documentation I've gleaned and used as a guide. I've
>>>>>> tested it with a few CGI programs not just Fossil. It's really a
>>>>>> work in progress but also seems to doing the job now.
>>>>>>
>>>>>> Thanks for your help :)
>>>>>>
>>>>>> Kind regards,
>>>>>>
>>>>>> Scott
>>>>>>
>>>>>>
>>>>>>
>>>>>> On 11/4/23 02:31, Massimo Manghi wrote:
>>>>>>> Hi Scott
>>>>>>>
>>>>>>> Thank you for the extensive explanation of your problem. I'm
>>>>>>> certainly interested in reviewing your patch
>>>>>>>
>>>>>>> Why using the usual command ::rivet::var_post (see
>>>>>>> https://tcl.apache.org/rivet/manual3.2/var.html
>>>>>>> <https://tcl.apache.org/rivet/manual3.2/var.html>) to retrieve
>>>>>>> the posted data didn't work for you?
>>>>>>>
>>>>>>> -- Massimo
>>>>>>>
>>>>>>>
>>>>>>> On 09/04/23 01:55, Scott Pitcher wrote:
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> I've been writing a system for our website using Rivet as the
>>>>>>>> foundation. I'd written a proc for running CGI programs from the
>>>>>>>> client request, or, passing the request over. This works well
>>>>>>>> but I'd recently started to incorporate some fossil archives in
>>>>>>>> the website. The fossil CGI method worked except when I tried to
>>>>>>>> clone the repo from a shell. Fossil would report an error code
>>>>>>>> and clone would fail.
>>>>>>>>
>>>>>>>> I examined the website logs and it Fossil makes a POST request
>>>>>>>> to the CGI program. With wireshark I could see around 100 or so
>>>>>>>> binary bytes in the body of the POST request, below the headers:
>>>>>>>>
>>>>>>>>> 0000 50 4f 53 54 20 2f 66 6f 73 73 2f 75 6e 69 6d 61 POST
>>>>>>>>> /foss/unima 0010 6b 65 73 63 72 69 70 74 73 2e 63 67 69 2f 69
>>>>>>>>> 6e kescripts.cgi/in 0020 64 65 78 20 48 54 54 50 2f 31 2e
>>>>>>>>> 30 0d 0a 48 6f dex HTTP/1.0..Ho 0030 73 74 3a 20 77 77 77
>>>>>>>>> 2e 73 65 72 76 65 72 32 2e st: www.server2
>>>>>>>>> <http://www.server2>. 0040 73 76 70 74 73 0d 0a 55 73 65 72
>>>>>>>>> 2d 41 67 65 6e svpts..User-Agen 0050 74 3a 20 46 6f 73 73
>>>>>>>>> 69 6c 2f 32 2e 32 30 20 28 t: Fossil/2.20 ( 0060 32 30 32
>>>>>>>>> 32 2d 31 31 2d 31 36 20 31 38 3a 34 36 2022-11-16 18:46
>>>>>>>>> 0070 3a 33 32 20 5b 32 31 30 65 38 39 61 30 35 39 5d :32
>>>>>>>>> [210e89a059] 0080 29 0d 0a 43 6f 6e 74 65 6e 74 2d 54 79 70
>>>>>>>>> 65 3a )..Content-Type: 0090 20 61 70 70 6c 69 63 61 74 69 6f
>>>>>>>>> 6e 2f 78 2d 66 application/x-f 00a0 6f 73 73 69 6c 0d 0a
>>>>>>>>> 43 6f 6e 74 65 6e 74 2d 4c ossil..Content-L 00b0 65 6e 67
>>>>>>>>> 74 68 3a 20 31 30 30 0d 0a 0d 0a 00 00 ength: 100......
>>>>>>>>> 00c0 00 61 78 da 05 c1 41 0a 80 20 10 00 c0 bb af 58
>>>>>>>>> .axÚ.ÁA.. ..À»¯X 00d0 e8 1c ec 6e ba e9 d1 4c ff 21 21 11 94
>>>>>>>>> 86 45 ef è.ìnºéÑLÿ!!...Eï 00e0 6f e6 ee 79 bf 32 6c e7 51
>>>>>>>>> ea 3b 7e a5 3f 47 ab oæîy¿2lçQê;~¥?G« 00f0 c0 8c 88 c0 c8
>>>>>>>>> 4c 44 02 64 b5 4c ac b6 b3 d5 02 À..ÀÈLD.dµL¬¶³Õ. 0100 13 90
>>>>>>>>> 1a 20 a1 5d 30 08 e9 e0 8c 8e 5e 4b 8c 22 ... ¡]0.éà..^K."
>>>>>>>>> 0110 c6 27 f4 f3 6c 53 10 71 d6 38 b7 92 fa 01 36 4f
>>>>>>>>> Æ'ôólS.qÖ8·.ú.6O 0120 18 9a
>>>>>>>>
>>>>>>>> I was able to retrieve the headers using ::rivet::apache_table
>>>>>>>> array_get headers_in. I wasn't able to retrieve the body of the
>>>>>>>> POST request. I tried ::rivet::raw_post but it returns an empty
>>>>>>>> string each time.
>>>>>>>>
>>>>>>>> I checked out the Rivet source and delved into src/mod_rivet_ng
>>>>>>>> and the Rivet_RawPost() function, and traced down to
>>>>>>>> apache_request.h and ApacheRequest_get_raw_post().
>>>>>>>>
>>>>>>>> The problem is that Rivet_RawPost() uses Tcl_NewStringObj() to
>>>>>>>> pass the body back to Tcl, and given that the first 3 bytes are
>>>>>>>> NUL, we end with an empty string. I changed it to
>>>>>>>> Tcl_NewByteArrayObj() and added a length parameter which was
>>>>>>>> passed by address all the way down to ApacheRequest_get_raw_post():
>>>>>>>>
>>>>>>>> data = TclWeb_GetRawPost(private->req, &length); if
>>>>>>>> (!data) { data = ""; } retval =
>>>>>>>> Tcl_NewByteArrayObj((const unsigned char *)data, length);
>>>>>>>> Tcl_SetObjResult(interp, retval);
>>>>>>>>
>>>>>>>> ... and added a raw_length field to the ApacheRequest structure:
>>>>>>>>
>>>>>>>> const char* temp_dir; char* raw_post; /* Raw post data. */ int
>>>>>>>> raw_length; request_rec* r; int nargs; } ApacheRequest;
>>>>>>>>
>>>>>>>> The ApacheRequest_parse_urlencoded() function will fill this
>>>>>>>> field in just after filling in the raw_post pointer, and
>>>>>>>> util_read needed a length pointer parameter as well to get that
>>>>>>>> working:
>>>>>>>>
>>>>>>>> int ApacheRequest_parse_urlencoded(ApacheRequest *req) { .....
>>>>>>>> if ((rc = util_read(req, &data, &length)) != OK) { return rc; }
>>>>>>>> if (data) { req->raw_post = (char*) data; /* Give people a way
>>>>>>>> of getting at the raw data. */ req->raw_length = length;
>>>>>>>> split_to_parms(req, data); } } return OK; }
>>>>>>>>
>>>>>>>> I went to the effort because it's a simple change and I think it
>>>>>>>> completes raw_post , if that function can return the complete body.
>>>>>>>>
>>>>>>>> That's just my opinion. It's important to me because I have a
>>>>>>>> website structure that can handle content, both HTTP and HTTPS
>>>>>>>> at the same time, as well as user base with login and
>>>>>>>> differentiation between local and remote access. It's been
>>>>>>>> developed so we can include private content for our business
>>>>>>>> only and host files for customers both public (e.g.
>>>>>>>> documentation) and private (firmware and software downloads).
>>>>>>>>
>>>>>>>> I really wanted to get fossil working under it as it streamlines
>>>>>>>> that part of the development for us. I was going to give up and
>>>>>>>> host the fossil archives separately but I persisted and came up
>>>>>>>> with this. I still haven't had time to pass the POST request
>>>>>>>> into fossil but I'll do so when I have time this weekend, but I
>>>>>>>> do expect it to work, one way or another.
>>>>>>>>
>>>>>>>> Anyway. That's my situation and if it's at all any use to
>>>>>>>> anyone, I'd be more than happy to give you a patch. It's a
>>>>>>>> really a minor set of changes but they give raw_post a bit more
>>>>>>>> utility.
>>>>>>>>
>>>>>>>> Further: I'd really like Rivet to have a kind of "run_cgi"
>>>>>>>> method, but I've coded one in tcl at the moment and that's good
>>>>>>>> for testing. At some stage I'll write a Rivet command for doing
>>>>>>>> the same which might make environment setup and processing of
>>>>>>>> the CGI output, headers etc, a bit more efficient.
>>>>>>>>
>>>>>>>> Kind regards, Scott Pitcher
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> ---------------------------------------------------------------------
>>>>>>> To unsubscribe, e-mail: rivet-dev-unsubscribe@tcl.apache.org
>>>>>>> <ma...@tcl.apache.org>
>>>>>>> For additional commands, e-mail: rivet-dev-help@tcl.apache.org
>>>>>>> <ma...@tcl.apache.org>
>>>>>>
>>>>>> --
>>>>>> +--------------------------------------------------------------------------------+
>>>>>> | ... will make the wilderness a pool of water, the dry land springs of water ...|
>>>>>> +--------------------------------------------------------------------------------+
>>>>>> | Scott PitcherScotty@svptechnicalservices.com.au <ma...@svptechnicalservices.com.au> |
>>>>>> | +61-3-9008-5695 (B) +61-479-042-834 (M) |
>>>>>> | SVP Technical Services ABN 79 220 496 603 |
>>>>>> |PGP:https://gnupg.org/ <https://gnupg.org/> |
>>>>>> |KEY:https://pgp.mit.edu/pks/lookup?op=vindex&search=0xEDBE43639EFD884D <https://pgp.mit.edu/pks/lookup?op=vindex&search=0xEDBE43639EFD884D> |
>>>>>> +--------------------------------------------------------------------------------+
>>>>>>
>>>>>> DISCLAIMER:
>>>>>> The content of this email is confidential and strictly intended only for the
>>>>>> recipient(s) specified in the message. If you have received this message by
>>>>>> mistake it is strictly forbidden to keep this email in any form, electronic or
>>>>>> hard copy, or to share any part of it with any one else, without the prior,
>>>>>> written consent of the author. If you have received this message by mistake,
>>>>>> please let me know by replying to this email, and then destroy it and any copies.
>>>>>>
>>>>>> SECURITY:
>>>>>> I prefer to send and receive confidential messages protected by PGP public key
>>>>>> encryption. If you are willing to or are interested in protecting our email
>>>>>> communication with PGP encryption, please refer tohttps://gnupg.org/ <https://gnupg.org/>.
>>>>>
>>>>> --
>>>>> +--------------------------------------------------------------------------------+
>>>>> | ... will make the wilderness a pool of water, the dry land springs of water ...|
>>>>> +--------------------------------------------------------------------------------+
>>>>> | Scott PitcherScotty@svptechnicalservices.com.au <ma...@svptechnicalservices.com.au> |
>>>>> | +61-3-9008-5695 (B) +61-479-042-834 (M) |
>>>>> | SVP Technical Services ABN 79 220 496 603 |
>>>>> |PGP:https://gnupg.org/ <https://gnupg.org/> |
>>>>> |KEY:https://pgp.mit.edu/pks/lookup?op=vindex&search=0xEDBE43639EFD884D <https://pgp.mit.edu/pks/lookup?op=vindex&search=0xEDBE43639EFD884D> |
>>>>> +--------------------------------------------------------------------------------+
>>>>>
>>>>> DISCLAIMER:
>>>>> The content of this email is confidential and strictly intended only for the
>>>>> recipient(s) specified in the message. If you have received this message by
>>>>> mistake it is strictly forbidden to keep this email in any form, electronic or
>>>>> hard copy, or to share any part of it with any one else, without the prior,
>>>>> written consent of the author. If you have received this message by mistake,
>>>>> please let me know by replying to this email, and then destroy it and any copies.
>>>>>
>>>>> SECURITY:
>>>>> I prefer to send and receive confidential messages protected by PGP public key
>>>>> encryption. If you are willing to or are interested in protecting our email
>>>>> communication with PGP encryption, please refer tohttps://gnupg.org/ <https://gnupg.org/>.
>>>>
>>>> --
>>>> +--------------------------------------------------------------------------------+
>>>> | ... will make the wilderness a pool of water, the dry land springs of water ...|
>>>> +--------------------------------------------------------------------------------+
>>>> | Scott PitcherScotty@svptechnicalservices.com.au <ma...@svptechnicalservices.com.au> |
>>>> | +61-3-9008-5695 (B) +61-479-042-834 (M) |
>>>> | SVP Technical Services ABN 79 220 496 603 |
>>>> |PGP:https://gnupg.org/ <https://gnupg.org/> |
>>>> |KEY:https://pgp.mit.edu/pks/lookup?op=vindex&search=0xEDBE43639EFD884D <https://pgp.mit.edu/pks/lookup?op=vindex&search=0xEDBE43639EFD884D> |
>>>> +--------------------------------------------------------------------------------+
>>>>
>>>> DISCLAIMER:
>>>> The content of this email is confidential and strictly intended only for the
>>>> recipient(s) specified in the message. If you have received this message by
>>>> mistake it is strictly forbidden to keep this email in any form, electronic or
>>>> hard copy, or to share any part of it with any one else, without the prior,
>>>> written consent of the author. If you have received this message by mistake,
>>>> please let me know by replying to this email, and then destroy it and any copies.
>>>>
>>>> SECURITY:
>>>> I prefer to send and receive confidential messages protected by PGP public key
>>>> encryption. If you are willing to or are interested in protecting our email
>>>> communication with PGP encryption, please refer tohttps://gnupg.org/ <https://gnupg.org/>.
>>>
>>> --
>>> +--------------------------------------------------------------------------------+
>>> | ... will make the wilderness a pool of water, the dry land springs of water ...|
>>> +--------------------------------------------------------------------------------+
>>> | Scott PitcherScotty@svptechnicalservices.com.au <ma...@svptechnicalservices.com.au> |
>>> | +61-3-9008-5695 (B) +61-479-042-834 (M) |
>>> | SVP Technical Services ABN 79 220 496 603 |
>>> |PGP:https://gnupg.org/ <https://gnupg.org/> |
>>> |KEY:https://pgp.mit.edu/pks/lookup?op=vindex&search=0xEDBE43639EFD884D <https://pgp.mit.edu/pks/lookup?op=vindex&search=0xEDBE43639EFD884D> |
>>> +--------------------------------------------------------------------------------+
>>>
>>> DISCLAIMER:
>>> The content of this email is confidential and strictly intended only for the
>>> recipient(s) specified in the message. If you have received this message by
>>> mistake it is strictly forbidden to keep this email in any form, electronic or
>>> hard copy, or to share any part of it with any one else, without the prior,
>>> written consent of the author. If you have received this message by mistake,
>>> please let me know by replying to this email, and then destroy it and any copies.
>>>
>>> SECURITY:
>>> I prefer to send and receive confidential messages protected by PGP public key
>>> encryption. If you are willing to or are interested in protecting our email
>>> communication with PGP encryption, please refer tohttps://gnupg.org/ <https://gnupg.org/>.
>>
>> --
>> +--------------------------------------------------------------------------------+
>> | ... will make the wilderness a pool of water, the dry land springs of water ...|
>> +--------------------------------------------------------------------------------+
>> | Scott PitcherScotty@svptechnicalservices.com.au |
>> | +61-3-9008-5695 (B) +61-479-042-834 (M) |
>> | SVP Technical Services ABN 79 220 496 603 |
>> |PGP:https://gnupg.org/ |
>> |KEY:https://pgp.mit.edu/pks/lookup?op=vindex&search=0xEDBE43639EFD884D |
>> +--------------------------------------------------------------------------------+
>>
>> DISCLAIMER:
>> The content of this email is confidential and strictly intended only for the
>> recipient(s) specified in the message. If you have received this message by
>> mistake it is strictly forbidden to keep this email in any form, electronic or
>> hard copy, or to share any part of it with any one else, without the prior,
>> written consent of the author. If you have received this message by mistake,
>> please let me know by replying to this email, and then destroy it and any copies.
>>
>> SECURITY:
>> I prefer to send and receive confidential messages protected by PGP public key
>> encryption. If you are willing to or are interested in protecting our email
>> communication with PGP encryption, please refer tohttps://gnupg.org/.
>
> --
> +--------------------------------------------------------------------------------+
> | ... will make the wilderness a pool of water, the dry land springs of water ...|
> +--------------------------------------------------------------------------------+
> | Scott PitcherScotty@svptechnicalservices.com.au |
> | +61-3-9008-5695 (B) +61-479-042-834 (M) |
> | SVP Technical Services ABN 79 220 496 603 |
> |PGP:https://gnupg.org/ |
> |KEY:https://pgp.mit.edu/pks/lookup?op=vindex&search=0xEDBE43639EFD884D |
> +--------------------------------------------------------------------------------+
>
> DISCLAIMER:
> The content of this email is confidential and strictly intended only for the
> recipient(s) specified in the message. If you have received this message by
> mistake it is strictly forbidden to keep this email in any form, electronic or
> hard copy, or to share any part of it with any one else, without the prior,
> written consent of the author. If you have received this message by mistake,
> please let me know by replying to this email, and then destroy it and any copies.
>
> SECURITY:
> I prefer to send and receive confidential messages protected by PGP public key
> encryption. If you are willing to or are interested in protecting our email
> communication with PGP encryption, please refer tohttps://gnupg.org/.
>
---------------------------------------------------------------------
To unsubscribe, e-mail: rivet-dev-unsubscribe@tcl.apache.org
For additional commands, e-mail: rivet-dev-help@tcl.apache.org
Re: Retrieving the body portion of a POST request
Posted by Scott Pitcher <sc...@svptechnicalservices.com.au>.
Thanks Massimo :)
On 4/5/23 16:49, Massimo Manghi wrote:
> Dear Scott,
>
> thank you. The new improvements to the test scripts were committed to
> rivet. I plan to propose a new rivet release soon
>
> -- Massimo
>
>
>
> On 30/04/23 13:08, Scott Pitcher wrote:
>> I've finished adding a raw_post test to the test suite. Patch is
>> attached.
>>
>> There is a small change to tests/apachetest/apachetest.tcl. I got it
>> to read IncludeOptional config files and to respect the ServerRoot,
>> which helped load config.
>>
>> Also changed tests/rivet.test to get it to look in the environment
>> for TestList and the 3 testgroup? variables, so I could override them
>> more easily.
>>
>> I added the new test on the end of post.test:
>>
>>> ::tcltest::test postvariables-6.1 {::rivet::raw_post} {
>>> set payload "\x00\x00\x00\x30\x31\x32\x33\x34\x35\x36\x37\x38\x39"
>>> set page [::http::geturl "${urlbase}post.rvt" \
>>> -query $payload \
>>> -type application/octet-stream]
>>> regexp -line {^\[::rivet::raw_post\] =
>>> 00000030313233343536373839$} [ ::http::data $page ] match
>>> set match
>>> } {[::rivet::raw_post] = 00000030313233343536373839}
>> And post.rvt has a condition check to handle that and return the
>> raw_post result. The leading zeros in the test data will cause the
>> test to fail without the bug fix (will return an empty string).
>>
>> Test passed fine here.
>>
>> The post.rvt and post.test changes are the important ones and you can
>> discard the other changes to apachetest.tcl and rivet.test. I had to
>> add those to get the tests to run.
>>
>> Kind regards,
>>
>> Scott
>>
>>
>>
>>
>>
>>
>> On 27/4/23 09:53, Scott Pitcher wrote:
>>> Thank you. That's good news. I haven't finished with the unit test
>>> yet. Will come back to it this weekend.
>>> Kind regards,
>>> Scott
>>>
>>>
>>> On 27/4/23 09:01, Massimo MANGHI wrote:
>>>> Your patch fixing the issue with the ::rivet::raw_post command was
>>>> commited in the master branch of mod_rivet.
>>>>
>>>> -- Massimo
>>>>
>>>>
>>>> ------------------------------------------------------------------------
>>>>
>>>> *Da:* Scott Pitcher
>>>> *Inviato:* Giovedì, 20 Aprile, 2023 14:59
>>>> *A:* Massimo MANGHI; rivet-dev@tcl.apache.org
>>>> *Oggetto:* Re: R: R: Retrieving the body portion of a POST request
>>>>
>>>> No problem at all. I'm working my way through it anyway.
>>>>
>>>> I found the authz_core_module module error is caused by the
>>>> IncludeOptional directives in my /etc/apache2/apache2.conf, which
>>>> are missed by the apachetest::getallincludes proc. It regexp's
>>>> against Include but not IncludeOptional. I'll rework it to accept
>>>> both.
>>>>
>>>> Kind regards,
>>>>
>>>> Scott
>>>>
>>>>
>>>>
>>>> On 20/4/23 19:16, Massimo MANGHI wrote:
>>>>> Hi Scott
>>>>>
>>>>> sorry for reacting to your message so late. If you test rivet
>>>>> against your Debian installed apache webserver you have to tweak
>>>>> your configuration in order to have all the required modules
>>>>> loaded. If you don't want to change it you have to build your own
>>>>> apache webserver and have it installed in a custom directory with
>>>>> all the configuration (make install does it for you when you build
>>>>> the apache HTTP web server)
>>>>>
>>>>> runtests.tcl run all the tests. If you need to work on your own
>>>>> tests then you may disable temporarily the others by putting in
>>>>> the TestList variable in tests/rivet.test only what you're
>>>>> actually working on
>>>>>
>>>>> -- Massimo
>>>>>
>>>>> ------------------------------------------------------------------------
>>>>>
>>>>> *Da:* Scott Pitcher
>>>>> *Inviato:* Giovedì, 20 Aprile, 2023 05:42
>>>>> *A:* Massimo MANGHI; rivet-dev@tcl.apache.org
>>>>> <ma...@tcl.apache.org>
>>>>> *Oggetto:* Re: R: Retrieving the body portion of a POST request
>>>>>
>>>>> I've managed to get beyond this problem by upgrading Apache from
>>>>> 2.4.25 to 2.4.46 via Debian stretch-backports-sloppy.
>>>>>
>>>>> Now I have an error -
>>>>>> checking /etc/apache2/apache2.conf
>>>>>> including files from ports.conf
>>>>>> check conf line for authz_core_module module...No LoadModule line
>>>>>> for authz_core_module!
>>>>>> while executing
>>>>>> "error "No LoadModule line for $mod\!""
>>>>>> (procedure "getloadmodules" line 12)
>>>>>> invoked from within ...
>>>>>> scotty@server1:/home/scotty/src/rivet/tcl-rivet/tests$
>>>>>
>>>>> But I'll work on this when I have some time free later this week.
>>>>>
>>>>>
>>>>> Kind regards,
>>>>>
>>>>> Scott
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On 17/4/23 10:57, Scott Pitcher wrote:
>>>>>> I've added a test to tests/post.test and .../post.rvt. I'm having
>>>>>> a little trouble invoking the test scripts. I get an error in
>>>>>> gethttpdconf:
>>>>>>> scotty@server1:/home/scotty/src/rivet/tcl-rivet/tests$
>>>>>>> ./runtests.tcl /usr/sbin/apache2
>>>>>>> runtests.tcl is running with auto_path: ./apachetest
>>>>>>> /usr/share/tcltk/tcl8.6 /usr/share/tcltk /usr/lib
>>>>>>> /usr/local/lib/tcltk /usr/local/share/tcltk
>>>>>>> /usr/lib/tcltk/x86_64-linux-gnu /usr/lib/tcltk /usr/lib/tcltk/t
>>>>>>> cl8.6
>>>>>>> n arguments: 0
>>>>>>> httpd_bin: /usr/sbin/apache2
>>>>>>> httpd_args:
>>>>>>> httpd_version: 2.4.25
>>>>>>> mpm: prefork, bridge: default
>>>>>>> Tests will be run against apache 2.4.25 version with the prefork
>>>>>>> module and the default bridge
>>>>>>> DEBUG apachetest::gethttpdconf: options=
>>>>>>> binname=/usr/sbin/apache2 bin=file3
>>>>>>> can't read "filename": no such variable
>>>>>>> while executing
>>>>>>> "file exists $filename"
>>>>>>> (procedure "gethttpdconf" line 8)
>>>>>>> invoked from within
>>>>>>> "gethttpdconf $binname"
>>>>>>> (procedure "determinemodules" line 4)
>>>>>>> invoked from within
>>>>>>> "determinemodules $binname"
>>>>>>> (procedure "apachetest::makeconf" line 16)
>>>>>>> invoked from within
>>>>>>> "apachetest::makeconf server.conf $bridge {
>>>>>>> LoadModule rivet_module [file join $CWD .. src/.libs
>>>>>>> mod_rivet[info sharedlibextension]]
>>>>>>>
>>>>>>> # User an..."
>>>>>>> (file "./runtests.tcl" line 124)
>>>>>>> scotty@server1:/home/scotty/src/rivet/tcl-rivet/tests$
>>>>>>>
>>>>>>
>>>>>> This seems to be because invoking apache2 with -V option fails
>>>>>> due to missing environment configuration:
>>>>>>
>>>>>>> scotty@server1:~/src/rivet/tcl-rivet/tests$ /usr/sbin/apache2 -V
>>>>>>> [Mon Apr 17 10:50:31.830756 2023] [core:warn] [pid 997] AH00111:
>>>>>>> Config variable ${APACHE_RUN_DIR} is not defined
>>>>>>> apache2: Syntax error on line 80 of /etc/apache2/apache2.conf:
>>>>>>> DefaultRuntimeDir must be a valid directory, absolute or
>>>>>>> relative to ServerRoot
>>>>>>> scotty@server1:~/src/rivet/tcl-rivet/tests$
>>>>>>
>>>>>>
>>>>>> Looking at /etc/init.d/apache2 there is a little work to do with
>>>>>> running apache and setting up environment.
>>>>>>
>>>>>> Perhaps it's better to run apache with "...-V -f server.conf"?
>>>>>>
>>>>>>
>>>>>> Your thoughts?
>>>>>>
>>>>>> Kind regards,
>>>>>> Scott
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On 14/4/23 18:05, Massimo MANGHI wrote:
>>>>>>> Hi Scott
>>>>>>>
>>>>>>> I get the point, if you don't have variable section in the POST
>>>>>>> request you can't access the data except if you call raw_post.
>>>>>>> Thank you for the patch, I will integrate it into Rivet. I guess
>>>>>>> this patch is worth a bugfix release of Rivet.
>>>>>>>
>>>>>>> Since you're working on this type of POST requests do you have
>>>>>>> at hand some test cases we may add to our test suite? Thank you
>>>>>>>
>>>>>>> -- Massimo
>>>>>>>
>>>>>>>
>>>>>>> ------------------------------------------------------------------------
>>>>>>>
>>>>>>> *Da:* Scott Pitcher
>>>>>>> *Inviato:* Martedì, 11 Aprile, 2023 12:06
>>>>>>> *A:* Massimo MANGHI; rivet-dev@tcl.apache.org
>>>>>>> <ma...@tcl.apache.org>
>>>>>>> *Oggetto:* Re: Retrieving the body portion of a POST request
>>>>>>>
>>>>>>> Hi Massimo,
>>>>>>>
>>>>>>> I tested "::rivet::var_post all" by inserting it into my client
>>>>>>> session code:
>>>>>>>
>>>>>>>> # DEBUG: What does var_post return? if {[info exists
>>>>>>>> env(REQUEST_METHOD)] && [string match "POST"
>>>>>>>> $env(REQUEST_METHOD)]} { ::rivet::apache_log_error debug
>>>>>>>> "clientSession: POST, ::rivet::var_post=[::rivet::var_post
>>>>>>>> all]" }
>>>>>>> ... but when a Fossil POST request arrives, this function
>>>>>>> returns a blank string. Probably because the POST request uses
>>>>>>> no variables. It's main purpose appears to be to pass Fossil the
>>>>>>> 100 or so bytes in the body of the POST request.
>>>>>>>
>>>>>>> When I ask Fossil to do a http clone and observe it with
>>>>>>> Wireshark at the server I see the following:
>>>>>>>
>>>>>>>> Frame 261: 164 bytes on wire (1312 bits), 164 bytes captured
>>>>>>>> (1312 bits) on interface 0
>>>>>>>> Ethernet II, Src: d8:5e:d3:0e:20:aa (d8:5e:d3:0e:20:aa), Dst:
>>>>>>>> AsrockIn_e6:6b:35 (bc:5f:f4:e6:6b:35)
>>>>>>>> Internet Protocol Version 4, Src: 192.168.1.12, Dst: 192.168.1.1
>>>>>>>> Transmission Control Protocol, Src Port: 33820, Dst Port: 80,
>>>>>>>> Seq: 180, Ack: 1, Len: 98
>>>>>>>> [2 Reassembled TCP Segments (277 bytes): #259(179), #261(98)]
>>>>>>>> [Frame: 259, payload: 0-178 (179 bytes)]
>>>>>>>> [Frame: 261, payload: 179-276 (98 bytes)]
>>>>>>>> [Segment count: 2]
>>>>>>>> [Reassembled TCP length: 277]
>>>>>>>> [Reassembled TCP Data:
>>>>>>>> 504f5354202f666f73732f67697432666f7373696c2e6367...]
>>>>>>>> Hypertext Transfer Protocol
>>>>>>>> POST /foss/git2fossil.cgi HTTP/1.0\r\n
>>>>>>>> Host:
>>>>>>>> [[[http://www.server2.svpts\r\n]http://www.server2.svpts\r\n][http://www.server2.svpts\r\n]http://www.server2.svpts\r\n]www.server2.svpts\r\n
>>>>>>>> User-Agent: Fossil/2.20 (2022-11-16 18:46:32 [210e89a059])\r\n
>>>>>>>> Content-Type: application/x-fossil\r\n
>>>>>>>> Content-Length: 98\r\n
>>>>>>>> \r\n
>>>>>>>> [Full request URI:
>>>>>>>> http://www.server2.svpts/foss/git2fossil.cgi
>>>>>>>> <http://www.server2.svpts/foss/git2fossil.cgi>]
>>>>>>>> [HTTP request 1/1]
>>>>>>>> [Response in frame: 273]
>>>>>>>> File Data: 98 bytes
>>>>>>>> Media Type
>>>>>>>> Media type: application/x-fossil (98 bytes)
>>>>>>>> 0000 00 00 00 61 78 da 05 c1 41 0a 80 20 10 05 d0 bd
>>>>>>>> ...ax... A.. ....
>>>>>>>> 0010 a7 18 68 1d cc fc 19 35 d7 69 f7 88 90 08 ca a2
>>>>>>>> ..h....5 .i......
>>>>>>>> 0020 a2 f3 f7 de 75 cf eb 31 d3 b2 6f b5 bd fd 57 ef
>>>>>>>> ....u..1 ..o...W.
>>>>>>>> 0030 67 3b 1b 01 cc 4c 60 40 44 02 c9 60 41 e1 96 fd
>>>>>>>> g;...L`@ D..`A...
>>>>>>>> 0040 6c 95 94 c4 75 c4 d1 4f 01 69 9c 3c 52 56 68 41
>>>>>>>> l...u..O .i.<RVhA
>>>>>>>> 0050 31 d3 1c 63 32 b3 11 52 82 64 5f 72 74 3f 31 1c
>>>>>>>> 1..c2..R .d_rt?1.
>>>>>>>> 0060 18 6d .m
>>>>>>>
>>>>>>> The 98 bytes in the body of the request after the last "\r\n" is
>>>>>>> the part I'm trying to get hold of in my website code. With
>>>>>>> Rivet patched (attached to this email) the ::rivet::raw_post
>>>>>>> returns the 98 bytes as a Tcl byte array, and I'm able to write
>>>>>>> that directly through the pipe to Fossil's, and I had the first
>>>>>>> successful "fossil clone http..." via Rivet and the website last
>>>>>>> night. I was very happy!
>>>>>>>
>>>>>>> For reference I've also attached my execcgi.tcl module. Much of
>>>>>>> it is notes and documentation I've gleaned and used as a guide.
>>>>>>> I've tested it with a few CGI programs not just Fossil. It's
>>>>>>> really a work in progress but also seems to doing the job now.
>>>>>>>
>>>>>>> Thanks for your help :)
>>>>>>>
>>>>>>> Kind regards,
>>>>>>>
>>>>>>> Scott
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On 11/4/23 02:31, Massimo Manghi wrote:
>>>>>>>> Hi Scott
>>>>>>>>
>>>>>>>> Thank you for the extensive explanation of your problem. I'm
>>>>>>>> certainly interested in reviewing your patch
>>>>>>>>
>>>>>>>> Why using the usual command ::rivet::var_post (see
>>>>>>>> https://tcl.apache.org/rivet/manual3.2/var.html
>>>>>>>> <https://tcl.apache.org/rivet/manual3.2/var.html>) to retrieve
>>>>>>>> the posted data didn't work for you?
>>>>>>>>
>>>>>>>> -- Massimo
>>>>>>>>
>>>>>>>>
>>>>>>>> On 09/04/23 01:55, Scott Pitcher wrote:
>>>>>>>>> Hi,
>>>>>>>>>
>>>>>>>>> I've been writing a system for our website using Rivet as the
>>>>>>>>> foundation. I'd written a proc for running CGI programs from
>>>>>>>>> the client request, or, passing the request over. This works
>>>>>>>>> well but I'd recently started to incorporate some fossil
>>>>>>>>> archives in the website. The fossil CGI method worked except
>>>>>>>>> when I tried to clone the repo from a shell. Fossil would
>>>>>>>>> report an error code and clone would fail.
>>>>>>>>>
>>>>>>>>> I examined the website logs and it Fossil makes a POST request
>>>>>>>>> to the CGI program. With wireshark I could see around 100 or
>>>>>>>>> so binary bytes in the body of the POST request, below the
>>>>>>>>> headers:
>>>>>>>>>
>>>>>>>>>> 0000 50 4f 53 54 20 2f 66 6f 73 73 2f 75 6e 69 6d 61 POST
>>>>>>>>>> /foss/unima 0010 6b 65 73 63 72 69 70 74 73 2e 63 67 69 2f 69
>>>>>>>>>> 6e kescripts.cgi/in 0020 64 65 78 20 48 54 54 50 2f 31 2e
>>>>>>>>>> 30 0d 0a 48 6f dex HTTP/1.0..Ho 0030 73 74 3a 20 77 77 77
>>>>>>>>>> 2e 73 65 72 76 65 72 32 2e st: www.server2
>>>>>>>>>> <http://www.server2>. 0040 73 76 70 74 73 0d 0a 55 73 65 72
>>>>>>>>>> 2d 41 67 65 6e svpts..User-Agen 0050 74 3a 20 46 6f 73 73
>>>>>>>>>> 69 6c 2f 32 2e 32 30 20 28 t: Fossil/2.20 ( 0060 32 30 32
>>>>>>>>>> 32 2d 31 31 2d 31 36 20 31 38 3a 34 36 2022-11-16 18:46
>>>>>>>>>> 0070 3a 33 32 20 5b 32 31 30 65 38 39 61 30 35 39 5d :32
>>>>>>>>>> [210e89a059] 0080 29 0d 0a 43 6f 6e 74 65 6e 74 2d 54 79 70
>>>>>>>>>> 65 3a )..Content-Type: 0090 20 61 70 70 6c 69 63 61 74 69
>>>>>>>>>> 6f 6e 2f 78 2d 66 application/x-f 00a0 6f 73 73 69 6c 0d 0a
>>>>>>>>>> 43 6f 6e 74 65 6e 74 2d 4c ossil..Content-L 00b0 65 6e 67
>>>>>>>>>> 74 68 3a 20 31 30 30 0d 0a 0d 0a 00 00 ength: 100......
>>>>>>>>>> 00c0 00 61 78 da 05 c1 41 0a 80 20 10 00 c0 bb af 58
>>>>>>>>>> .axÚ.ÁA.. ..À»¯X 00d0 e8 1c ec 6e ba e9 d1 4c ff 21 21 11
>>>>>>>>>> 94 86 45 ef è.ìnºéÑLÿ!!...Eï 00e0 6f e6 ee 79 bf 32 6c e7
>>>>>>>>>> 51 ea 3b 7e a5 3f 47 ab oæîy¿2lçQê;~¥?G« 00f0 c0 8c 88 c0
>>>>>>>>>> c8 4c 44 02 64 b5 4c ac b6 b3 d5 02 À..ÀÈLD.dµL¬¶³Õ. 0100 13
>>>>>>>>>> 90 1a 20 a1 5d 30 08 e9 e0 8c 8e 5e 4b 8c 22 ...
>>>>>>>>>> ¡]0.éà..^K." 0110 c6 27 f4 f3 6c 53 10 71 d6 38 b7 92 fa 01
>>>>>>>>>> 36 4f Æ'ôólS.qÖ8·.ú.6O 0120 18 9a
>>>>>>>>>
>>>>>>>>> I was able to retrieve the headers using ::rivet::apache_table
>>>>>>>>> array_get headers_in. I wasn't able to retrieve the body of
>>>>>>>>> the POST request. I tried ::rivet::raw_post but it returns an
>>>>>>>>> empty string each time.
>>>>>>>>>
>>>>>>>>> I checked out the Rivet source and delved into
>>>>>>>>> src/mod_rivet_ng and the Rivet_RawPost() function, and traced
>>>>>>>>> down to apache_request.h and ApacheRequest_get_raw_post().
>>>>>>>>>
>>>>>>>>> The problem is that Rivet_RawPost() uses Tcl_NewStringObj() to
>>>>>>>>> pass the body back to Tcl, and given that the first 3 bytes
>>>>>>>>> are NUL, we end with an empty string. I changed it to
>>>>>>>>> Tcl_NewByteArrayObj() and added a length parameter which was
>>>>>>>>> passed by address all the way down to
>>>>>>>>> ApacheRequest_get_raw_post():
>>>>>>>>>
>>>>>>>>> data = TclWeb_GetRawPost(private->req, &length); if
>>>>>>>>> (!data) { data = ""; } retval =
>>>>>>>>> Tcl_NewByteArrayObj((const unsigned char *)data, length);
>>>>>>>>> Tcl_SetObjResult(interp, retval);
>>>>>>>>>
>>>>>>>>> ... and added a raw_length field to the ApacheRequest structure:
>>>>>>>>>
>>>>>>>>> const char* temp_dir; char* raw_post; /* Raw post data. */ int
>>>>>>>>> raw_length; request_rec* r; int nargs; } ApacheRequest;
>>>>>>>>>
>>>>>>>>> The ApacheRequest_parse_urlencoded() function will fill this
>>>>>>>>> field in just after filling in the raw_post pointer, and
>>>>>>>>> util_read needed a length pointer parameter as well to get
>>>>>>>>> that working:
>>>>>>>>>
>>>>>>>>> int ApacheRequest_parse_urlencoded(ApacheRequest *req) { .....
>>>>>>>>> if ((rc = util_read(req, &data, &length)) != OK) { return rc;
>>>>>>>>> } if (data) { req->raw_post = (char*) data; /* Give people a
>>>>>>>>> way of getting at the raw data. */ req->raw_length = length;
>>>>>>>>> split_to_parms(req, data); } } return OK; }
>>>>>>>>>
>>>>>>>>> I went to the effort because it's a simple change and I think
>>>>>>>>> it completes raw_post , if that function can return the
>>>>>>>>> complete body.
>>>>>>>>>
>>>>>>>>> That's just my opinion. It's important to me because I have a
>>>>>>>>> website structure that can handle content, both HTTP and HTTPS
>>>>>>>>> at the same time, as well as user base with login and
>>>>>>>>> differentiation between local and remote access. It's been
>>>>>>>>> developed so we can include private content for our business
>>>>>>>>> only and host files for customers both public (e.g.
>>>>>>>>> documentation) and private (firmware and software downloads).
>>>>>>>>>
>>>>>>>>> I really wanted to get fossil working under it as it
>>>>>>>>> streamlines that part of the development for us. I was going
>>>>>>>>> to give up and host the fossil archives separately but I
>>>>>>>>> persisted and came up with this. I still haven't had time to
>>>>>>>>> pass the POST request into fossil but I'll do so when I have
>>>>>>>>> time this weekend, but I do expect it to work, one way or
>>>>>>>>> another.
>>>>>>>>>
>>>>>>>>> Anyway. That's my situation and if it's at all any use to
>>>>>>>>> anyone, I'd be more than happy to give you a patch. It's a
>>>>>>>>> really a minor set of changes but they give raw_post a bit
>>>>>>>>> more utility.
>>>>>>>>>
>>>>>>>>> Further: I'd really like Rivet to have a kind of "run_cgi"
>>>>>>>>> method, but I've coded one in tcl at the moment and that's
>>>>>>>>> good for testing. At some stage I'll write a Rivet command for
>>>>>>>>> doing the same which might make environment setup and
>>>>>>>>> processing of the CGI output, headers etc, a bit more efficient.
>>>>>>>>>
>>>>>>>>> Kind regards, Scott Pitcher
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>
>>>>>>>> To unsubscribe, e-mail: rivet-dev-unsubscribe@tcl.apache.org
>>>>>>>> <ma...@tcl.apache.org>
>>>>>>>> For additional commands, e-mail: rivet-dev-help@tcl.apache.org
>>>>>>>> <ma...@tcl.apache.org>
>>>>>>>
>>>>>>> --
>>>>>>> +--------------------------------------------------------------------------------+
>>>>>>>
>>>>>>> | ... will make the wilderness a pool of water, the dry land
>>>>>>> springs of water ...|
>>>>>>> +--------------------------------------------------------------------------------+
>>>>>>>
>>>>>>> | Scott PitcherScotty@svptechnicalservices.com.au
>>>>>>> <ma...@svptechnicalservices.com.au> |
>>>>>>> | +61-3-9008-5695 (B) +61-479-042-834 (M) |
>>>>>>> | SVP Technical Services
>>>>>>> ABN 79 220 496 603 |
>>>>>>> |PGP:https://gnupg.org/ <https://gnupg.org/> |
>>>>>>> |KEY:https://pgp.mit.edu/pks/lookup?op=vindex&search=0xEDBE43639EFD884D
>>>>>>> <https://pgp.mit.edu/pks/lookup?op=vindex&search=0xEDBE43639EFD884D>
>>>>>>> |
>>>>>>> +--------------------------------------------------------------------------------+
>>>>>>>
>>>>>>>
>>>>>>> DISCLAIMER:
>>>>>>> The content of this email is confidential and strictly intended
>>>>>>> only for the
>>>>>>> recipient(s) specified in the message. If you have received this
>>>>>>> message by
>>>>>>> mistake it is strictly forbidden to keep this email in any form,
>>>>>>> electronic or
>>>>>>> hard copy, or to share any part of it with any one else, without
>>>>>>> the prior,
>>>>>>> written consent of the author. If you have received this message
>>>>>>> by mistake,
>>>>>>> please let me know by replying to this email, and then destroy
>>>>>>> it and any copies.
>>>>>>>
>>>>>>> SECURITY:
>>>>>>> I prefer to send and receive confidential messages protected by
>>>>>>> PGP public key
>>>>>>> encryption. If you are willing to or are interested in
>>>>>>> protecting our email
>>>>>>> communication with PGP encryption, please refer
>>>>>>> tohttps://gnupg.org/ <https://gnupg.org/>.
>>>>>>
>>>>>> --
>>>>>> +--------------------------------------------------------------------------------+
>>>>>>
>>>>>> | ... will make the wilderness a pool of water, the dry land
>>>>>> springs of water ...|
>>>>>> +--------------------------------------------------------------------------------+
>>>>>>
>>>>>> | Scott PitcherScotty@svptechnicalservices.com.au
>>>>>> <ma...@svptechnicalservices.com.au> |
>>>>>> | +61-3-9008-5695 (B) +61-479-042-834 (M) |
>>>>>> | SVP Technical Services ABN
>>>>>> 79 220 496 603 |
>>>>>> |PGP:https://gnupg.org/ <https://gnupg.org/> |
>>>>>> |KEY:https://pgp.mit.edu/pks/lookup?op=vindex&search=0xEDBE43639EFD884D
>>>>>> <https://pgp.mit.edu/pks/lookup?op=vindex&search=0xEDBE43639EFD884D>
>>>>>> |
>>>>>> +--------------------------------------------------------------------------------+
>>>>>>
>>>>>>
>>>>>> DISCLAIMER:
>>>>>> The content of this email is confidential and strictly intended
>>>>>> only for the
>>>>>> recipient(s) specified in the message. If you have received this
>>>>>> message by
>>>>>> mistake it is strictly forbidden to keep this email in any form,
>>>>>> electronic or
>>>>>> hard copy, or to share any part of it with any one else, without
>>>>>> the prior,
>>>>>> written consent of the author. If you have received this message
>>>>>> by mistake,
>>>>>> please let me know by replying to this email, and then destroy it
>>>>>> and any copies.
>>>>>>
>>>>>> SECURITY:
>>>>>> I prefer to send and receive confidential messages protected by
>>>>>> PGP public key
>>>>>> encryption. If you are willing to or are interested in protecting
>>>>>> our email
>>>>>> communication with PGP encryption, please refer
>>>>>> tohttps://gnupg.org/ <https://gnupg.org/>.
>>>>>
>>>>> --
>>>>> +--------------------------------------------------------------------------------+
>>>>>
>>>>> | ... will make the wilderness a pool of water, the dry land
>>>>> springs of water ...|
>>>>> +--------------------------------------------------------------------------------+
>>>>>
>>>>> | Scott PitcherScotty@svptechnicalservices.com.au
>>>>> <ma...@svptechnicalservices.com.au> |
>>>>> | +61-3-9008-5695 (B)
>>>>> +61-479-042-834 (M) |
>>>>> | SVP Technical Services ABN
>>>>> 79 220 496 603 |
>>>>> |PGP:https://gnupg.org/ <https://gnupg.org/> |
>>>>> |KEY:https://pgp.mit.edu/pks/lookup?op=vindex&search=0xEDBE43639EFD884D
>>>>> <https://pgp.mit.edu/pks/lookup?op=vindex&search=0xEDBE43639EFD884D>
>>>>> |
>>>>> +--------------------------------------------------------------------------------+
>>>>>
>>>>>
>>>>> DISCLAIMER:
>>>>> The content of this email is confidential and strictly intended
>>>>> only for the
>>>>> recipient(s) specified in the message. If you have received this
>>>>> message by
>>>>> mistake it is strictly forbidden to keep this email in any form,
>>>>> electronic or
>>>>> hard copy, or to share any part of it with any one else, without
>>>>> the prior,
>>>>> written consent of the author. If you have received this message
>>>>> by mistake,
>>>>> please let me know by replying to this email, and then destroy it
>>>>> and any copies.
>>>>>
>>>>> SECURITY:
>>>>> I prefer to send and receive confidential messages protected by
>>>>> PGP public key
>>>>> encryption. If you are willing to or are interested in protecting
>>>>> our email
>>>>> communication with PGP encryption, please refer
>>>>> tohttps://gnupg.org/ <https://gnupg.org/>.
>>>>
>>>> --
>>>> +--------------------------------------------------------------------------------+
>>>>
>>>> | ... will make the wilderness a pool of water, the dry land
>>>> springs of water ...|
>>>> +--------------------------------------------------------------------------------+
>>>>
>>>> | Scott PitcherScotty@svptechnicalservices.com.au
>>>> <ma...@svptechnicalservices.com.au> |
>>>> | +61-3-9008-5695 (B) +61-479-042-834 (M) |
>>>> | SVP Technical Services ABN
>>>> 79 220 496 603 |
>>>> |PGP:https://gnupg.org/ <https://gnupg.org/> |
>>>> |KEY:https://pgp.mit.edu/pks/lookup?op=vindex&search=0xEDBE43639EFD884D
>>>> <https://pgp.mit.edu/pks/lookup?op=vindex&search=0xEDBE43639EFD884D> |
>>>> +--------------------------------------------------------------------------------+
>>>>
>>>>
>>>> DISCLAIMER:
>>>> The content of this email is confidential and strictly intended
>>>> only for the
>>>> recipient(s) specified in the message. If you have received this
>>>> message by
>>>> mistake it is strictly forbidden to keep this email in any form,
>>>> electronic or
>>>> hard copy, or to share any part of it with any one else, without
>>>> the prior,
>>>> written consent of the author. If you have received this message by
>>>> mistake,
>>>> please let me know by replying to this email, and then destroy it
>>>> and any copies.
>>>>
>>>> SECURITY:
>>>> I prefer to send and receive confidential messages protected by PGP
>>>> public key
>>>> encryption. If you are willing to or are interested in protecting
>>>> our email
>>>> communication with PGP encryption, please refer
>>>> tohttps://gnupg.org/ <https://gnupg.org/>.
>>>
>>> --
>>> +--------------------------------------------------------------------------------+
>>>
>>> | ... will make the wilderness a pool of water, the dry land springs
>>> of water ...|
>>> +--------------------------------------------------------------------------------+
>>>
>>> | Scott PitcherScotty@svptechnicalservices.com.au |
>>> | +61-3-9008-5695 (B) +61-479-042-834 (M) |
>>> | SVP Technical Services ABN 79 220 496 603 |
>>> |PGP:https://gnupg.org/ |
>>> |KEY:https://pgp.mit.edu/pks/lookup?op=vindex&search=0xEDBE43639EFD884D
>>> |
>>> +--------------------------------------------------------------------------------+
>>>
>>>
>>> DISCLAIMER:
>>> The content of this email is confidential and strictly intended only
>>> for the
>>> recipient(s) specified in the message. If you have received this
>>> message by
>>> mistake it is strictly forbidden to keep this email in any form,
>>> electronic or
>>> hard copy, or to share any part of it with any one else, without the
>>> prior,
>>> written consent of the author. If you have received this message by
>>> mistake,
>>> please let me know by replying to this email, and then destroy it
>>> and any copies.
>>>
>>> SECURITY:
>>> I prefer to send and receive confidential messages protected by PGP
>>> public key
>>> encryption. If you are willing to or are interested in protecting
>>> our email
>>> communication with PGP encryption, please refer tohttps://gnupg.org/.
>>
>> --
>> +--------------------------------------------------------------------------------+
>>
>> | ... will make the wilderness a pool of water, the dry land springs
>> of water ...|
>> +--------------------------------------------------------------------------------+
>>
>> | Scott PitcherScotty@svptechnicalservices.com.au |
>> | +61-3-9008-5695 (B) +61-479-042-834 (M) |
>> | SVP Technical Services ABN 79 220 496 603 |
>> |PGP:https://gnupg.org/ |
>> |KEY:https://pgp.mit.edu/pks/lookup?op=vindex&search=0xEDBE43639EFD884D
>> |
>> +--------------------------------------------------------------------------------+
>>
>>
>> DISCLAIMER:
>> The content of this email is confidential and strictly intended only
>> for the
>> recipient(s) specified in the message. If you have received this
>> message by
>> mistake it is strictly forbidden to keep this email in any form,
>> electronic or
>> hard copy, or to share any part of it with any one else, without the
>> prior,
>> written consent of the author. If you have received this message by
>> mistake,
>> please let me know by replying to this email, and then destroy it and
>> any copies.
>>
>> SECURITY:
>> I prefer to send and receive confidential messages protected by PGP
>> public key
>> encryption. If you are willing to or are interested in protecting our
>> email
>> communication with PGP encryption, please refer tohttps://gnupg.org/.
>>
--
+--------------------------------------------------------------------------------+
| ... will make the wilderness a pool of water, the dry land springs of water ...|
+--------------------------------------------------------------------------------+
| Scott PitcherScotty@svptechnicalservices.com.au |
| +61-3-9008-5695 (B) +61-479-042-834 (M) |
| SVP Technical Services ABN 79 220 496 603 |
|PGP:https://gnupg.org/ |
|KEY:https://pgp.mit.edu/pks/lookup?op=vindex&search=0xEDBE43639EFD884D |
+--------------------------------------------------------------------------------+
DISCLAIMER:
The content of this email is confidential and strictly intended only for the
recipient(s) specified in the message. If you have received this message by
mistake it is strictly forbidden to keep this email in any form, electronic or
hard copy, or to share any part of it with any one else, without the prior,
written consent of the author. If you have received this message by mistake,
please let me know by replying to this email, and then destroy it and any copies.
SECURITY:
I prefer to send and receive confidential messages protected by PGP public key
encryption. If you are willing to or are interested in protecting our email
communication with PGP encryption, please refer tohttps://gnupg.org/.