You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@httpd.apache.org by dean gaudet <de...@arctic.org> on 2001/01/07 20:51:47 UTC
woah, "GET /" with autoindex
woah. ok there's some serious problems here.
this is from a "GET / HTTP/1.0" which results in an autoindex (dunno why
it's not grabbing one of the index.htmls, but i'm not worrying about that,
the autoindex is probably far more interesting).
ryan, here's the memory allocation you were talking about i think. this
does do the allocation every time through. aiee. (and it looks like
linux returns memory to the system each request, note the final brk()
which truncates the heap.)
also, this is a perfect example of why zero-copy can be totally wrong.
passing 100 bytes to the kernel in each writev() in 8 elements is way
slower than passing the kernel 4096 byte buffers. i haven't written up
the test to prove it to you, but this is one of the exact problems i had
to deal with on the last zero-copy library i worked with.
this one is showstopper material.
in general you want to buffer when modules are doing dynamic generation
and you want to zero-copy when modules are sending bulk data.
-dean
accept(5, {sin_family=AF_INET, sin_port=htons(2511), sin_addr=inet_addr("204.107.140.52")}}, [16]) = 8
rt_sigaction(SIGUSR1, {SIG_IGN}, {0x807f774, [], SA_INTERRUPT|0x4000000}, 8) = 0
setsockopt(8, IPPROTO_TCP1, [1], 4) = 0
getsockname(8, {sin_family=AF_INET, sin_port=htons(8888), sin_addr=inet_addr("204.107.140.52")}}, [16]) = 0
fcntl(8, F_GETFL) = 0x2 (flags O_RDWR)
fcntl(8, F_SETFL, O_RDWR|O_NONBLOCK) = 0
read(8, "GET / HTTP/1.0\nHost: arctic.org:"..., 8192) = 38
gettimeofday({978896760, 331262}, NULL) = 0
stat("/home/dean/ap2/htdocs", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
open("/home/dean/ap2/htdocs/", O_RDONLY|O_NONBLOCK|O_DIRECTORY) = 9
fstat(9, {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
fcntl(9, F_SETFD, FD_CLOEXEC) = 0
brk(0x8152000) = 0x8152000
stat("/home/dean/ap2/htdocs/HEADER", 0xbffff6e0) = -1 ENOENT (No such file or directory)
stat("/home/dean/ap2/htdocs", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
brk(0x8154000) = 0x8154000
brk(0x8156000) = 0x8156000
brk(0x8158000) = 0x8158000
getdents(9, /* 38 entries */, 3933) = 1016
stat("/home/dean/ap2/htdocs/..", {st_mode=S_IFDIR|0775, st_size=4096, ...}) = 0
stat("/home/dean/ap2/htdocs/apache_pb.gif", {st_mode=S_IFREG|0664, st_size=2326, ...}) = 0
stat("/home/dean/ap2/htdocs/index.html.ca", {st_mode=S_IFREG|0664, st_size=1864, ...}) = 0
stat("/home/dean/ap2/htdocs/index.html.cz", {st_mode=S_IFREG|0664, st_size=1623, ...}) = 0
stat("/home/dean/ap2/htdocs/index.html.de", {st_mode=S_IFREG|0664, st_size=2250, ...}) = 0
stat("/home/dean/ap2/htdocs/index.html.dk", {st_mode=S_IFREG|0664, st_size=1547, ...}) = 0
stat("/home/dean/ap2/htdocs/index.html.ee", {st_mode=S_IFREG|0664, st_size=1867, ...}) = 0
stat("/home/dean/ap2/htdocs/index.html.el", {st_mode=S_IFREG|0664, st_size=1657, ...}) = 0
stat("/home/dean/ap2/htdocs/index.html.en", {st_mode=S_IFREG|0664, st_size=1349, ...}) = 0
stat("/home/dean/ap2/htdocs/index.html.es", {st_mode=S_IFREG|0664, st_size=1772, ...}) = 0
stat("/home/dean/ap2/htdocs/index.html.et", {st_mode=S_IFREG|0664, st_size=1867, ...}) = 0
stat("/home/dean/ap2/htdocs/index.html.fr", {st_mode=S_IFREG|0664, st_size=1559, ...}) = 0
stat("/home/dean/ap2/htdocs/index.html.he.iso8859-8", {st_mode=S_IFREG|0664, st_size=4115, ...}) = 0
stat("/home/dean/ap2/htdocs/index.html.it", {st_mode=S_IFREG|0664, st_size=1818, ...}) = 0
stat("/home/dean/ap2/htdocs/index.html.ja.iso2022-jp", {st_mode=S_IFREG|0664, st_size=1610, ...}) = 0
stat("/home/dean/ap2/htdocs/index.html.ja.jis", {st_mode=S_IFREG|0664, st_size=1610, ...}) = 0
stat("/home/dean/ap2/htdocs/index.html.kr.iso-kr", {st_mode=S_IFREG|0664, st_size=1303, ...}) = 0
stat("/home/dean/ap2/htdocs/index.html.kr.iso2022-kr", {st_mode=S_IFREG|0664, st_size=1303, ...}) = 0
stat("/home/dean/ap2/htdocs/index.html.ltz", {st_mode=S_IFREG|0664, st_size=1876, ...}) = 0
stat("/home/dean/ap2/htdocs/index.html.lu", {st_mode=S_IFREG|0664, st_size=1876, ...}) = 0
stat("/home/dean/ap2/htdocs/index.html.nl", {st_mode=S_IFREG|0664, st_size=2007, ...}) = 0
stat("/home/dean/ap2/htdocs/index.html.no", {st_mode=S_IFREG|0664, st_size=1504, ...}) = 0
stat("/home/dean/ap2/htdocs/index.html.po.iso-pl", {st_mode=S_IFREG|0664, st_size=1480, ...}) = 0
stat("/home/dean/ap2/htdocs/index.html.pt", {st_mode=S_IFREG|0664, st_size=1812, ...}) = 0
stat("/home/dean/ap2/htdocs/index.html.pt-br", {st_mode=S_IFREG|0664, st_size=1917, ...}) = 0
stat("/home/dean/ap2/htdocs/index.html.ru.cp-1251", {st_mode=S_IFREG|0664, st_size=1561, ...}) = 0
stat("/home/dean/ap2/htdocs/index.html.ru.cp866", {st_mode=S_IFREG|0664, st_size=1555, ...}) = 0
stat("/home/dean/ap2/htdocs/index.html.ru.iso-ru", {st_mode=S_IFREG|0664, st_size=1559, ...}) = 0
stat("/home/dean/ap2/htdocs/index.html.ru.koi8-r", {st_mode=S_IFREG|0664, st_size=1555, ...}) = 0
stat("/home/dean/ap2/htdocs/index.html.ru.ucs2", {st_mode=S_IFREG|0664, st_size=3134, ...}) = 0
stat("/home/dean/ap2/htdocs/index.html.ru.ucs4", {st_mode=S_IFREG|0664, st_size=6268, ...}) = 0
stat("/home/dean/ap2/htdocs/index.html.ru.utf8", {st_mode=S_IFREG|0664, st_size=2288, ...}) = 0
stat("/home/dean/ap2/htdocs/index.html.se", {st_mode=S_IFREG|0664, st_size=1670, ...}) = 0
stat("/home/dean/ap2/htdocs/index.html.tw", {st_mode=S_IFREG|0664, st_size=1032, ...}) = 0
stat("/home/dean/ap2/htdocs/index.html.tw.Big5", {st_mode=S_IFREG|0664, st_size=1032, ...}) = 0
stat("/home/dean/ap2/htdocs/manual", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
getdents(9, /* 0 entries */, 3933) = 0
brk(0x815a000) = 0x815a000
brk(0x815c000) = 0x815c000
brk(0x815e000) = 0x815e000
brk(0x8160000) = 0x8160000
brk(0x8162000) = 0x8162000
brk(0x8164000) = 0x8164000
brk(0x8166000) = 0x8166000
brk(0x8168000) = 0x8168000
brk(0x816a000) = 0x816a000
brk(0x816c000) = 0x816c000
brk(0x816e000) = 0x816e000
brk(0x8170000) = 0x8170000
brk(0x8172000) = 0x8172000
brk(0x8174000) = 0x8174000
brk(0x8176000) = 0x8176000
brk(0x8178000) = 0x8178000
brk(0x817a000) = 0x817a000
brk(0x817c000) = 0x817c000
brk(0x817e000) = 0x817e000
brk(0x8180000) = 0x8180000
brk(0x8182000) = 0x8182000
brk(0x8184000) = 0x8184000
brk(0x8186000) = 0x8186000
brk(0x8188000) = 0x8188000
brk(0x818a000) = 0x818a000
brk(0x818c000) = 0x818c000
brk(0x818e000) = 0x818e000
brk(0x8190000) = 0x8190000
brk(0x8191000) = 0x8191000
brk(0x8193000) = 0x8193000
brk(0x8195000) = 0x8195000
brk(0x8197000) = 0x8197000
brk(0x8199000) = 0x8199000
brk(0x819b000) = 0x819b000
brk(0x819d000) = 0x819d000
brk(0x819f000) = 0x819f000
brk(0x81a1000) = 0x81a1000
brk(0x81a3000) = 0x81a3000
brk(0x81a5000) = 0x81a5000
brk(0x81a7000) = 0x81a7000
brk(0x81a9000) = 0x81a9000
brk(0x81ab000) = 0x81ab000
brk(0x81ad000) = 0x81ad000
brk(0x81af000) = 0x81af000
brk(0x81b1000) = 0x81b1000
brk(0x81b3000) = 0x81b3000
brk(0x81b5000) = 0x81b5000
brk(0x81b7000) = 0x81b7000
brk(0x81b9000) = 0x81b9000
brk(0x81bb000) = 0x81bb000
brk(0x81bd000) = 0x81bd000
brk(0x81bf000) = 0x81bf000
brk(0x81c1000) = 0x81c1000
brk(0x81c3000) = 0x81c3000
brk(0x81c5000) = 0x81c5000
brk(0x81c7000) = 0x81c7000
brk(0x81c9000) = 0x81c9000
brk(0x81cb000) = 0x81cb000
brk(0x81cd000) = 0x81cd000
brk(0x81cf000) = 0x81cf000
brk(0x81d1000) = 0x81d1000
brk(0x81d2000) = 0x81d2000
brk(0x81d4000) = 0x81d4000
brk(0x81d6000) = 0x81d6000
brk(0x81d8000) = 0x81d8000
brk(0x81da000) = 0x81da000
brk(0x81dc000) = 0x81dc000
brk(0x81de000) = 0x81de000
brk(0x81e0000) = 0x81e0000
brk(0x81e2000) = 0x81e2000
brk(0x81e4000) = 0x81e4000
brk(0x81e6000) = 0x81e6000
brk(0x81e8000) = 0x81e8000
brk(0x81ea000) = 0x81ea000
brk(0x81ec000) = 0x81ec000
brk(0x81ee000) = 0x81ee000
brk(0x81f0000) = 0x81f0000
brk(0x81f2000) = 0x81f2000
brk(0x81f4000) = 0x81f4000
brk(0x81f6000) = 0x81f6000
brk(0x81f8000) = 0x81f8000
brk(0x81fa000) = 0x81fa000
brk(0x81fc000) = 0x81fc000
brk(0x81fe000) = 0x81fe000
brk(0x8200000) = 0x8200000
brk(0x8202000) = 0x8202000
brk(0x8204000) = 0x8204000
brk(0x8206000) = 0x8206000
brk(0x8208000) = 0x8208000
brk(0x820a000) = 0x820a000
brk(0x820c000) = 0x820c000
brk(0x820e000) = 0x820e000
brk(0x8210000) = 0x8210000
brk(0x8212000) = 0x8212000
brk(0x8213000) = 0x8213000
brk(0x8215000) = 0x8215000
brk(0x8217000) = 0x8217000
brk(0x8219000) = 0x8219000
brk(0x821b000) = 0x821b000
brk(0x821d000) = 0x821d000
brk(0x821f000) = 0x821f000
brk(0x8221000) = 0x8221000
brk(0x8223000) = 0x8223000
brk(0x8225000) = 0x8225000
brk(0x8227000) = 0x8227000
brk(0x8229000) = 0x8229000
brk(0x822b000) = 0x822b000
brk(0x822d000) = 0x822d000
brk(0x822f000) = 0x822f000
brk(0x8231000) = 0x8231000
brk(0x8233000) = 0x8233000
brk(0x8235000) = 0x8235000
brk(0x8237000) = 0x8237000
brk(0x8239000) = 0x8239000
brk(0x823b000) = 0x823b000
brk(0x823d000) = 0x823d000
brk(0x823f000) = 0x823f000
brk(0x8241000) = 0x8241000
brk(0x8243000) = 0x8243000
brk(0x8245000) = 0x8245000
brk(0x8247000) = 0x8247000
brk(0x8249000) = 0x8249000
brk(0x824b000) = 0x824b000
brk(0x824d000) = 0x824d000
brk(0x824f000) = 0x824f000
brk(0x8251000) = 0x8251000
brk(0x8253000) = 0x8253000
brk(0x8254000) = 0x8254000
brk(0x8256000) = 0x8256000
brk(0x8258000) = 0x8258000
brk(0x825a000) = 0x825a000
brk(0x825c000) = 0x825c000
brk(0x825e000) = 0x825e000
brk(0x8260000) = 0x8260000
brk(0x8262000) = 0x8262000
brk(0x8264000) = 0x8264000
brk(0x8266000) = 0x8266000
brk(0x8268000) = 0x8268000
brk(0x826a000) = 0x826a000
brk(0x826c000) = 0x826c000
brk(0x826e000) = 0x826e000
brk(0x8270000) = 0x8270000
brk(0x8272000) = 0x8272000
brk(0x8274000) = 0x8274000
brk(0x8276000) = 0x8276000
brk(0x8278000) = 0x8278000
brk(0x827a000) = 0x827a000
brk(0x827c000) = 0x827c000
brk(0x827e000) = 0x827e000
brk(0x8280000) = 0x8280000
brk(0x8282000) = 0x8282000
brk(0x8284000) = 0x8284000
brk(0x8286000) = 0x8286000
brk(0x8288000) = 0x8288000
brk(0x828a000) = 0x828a000
brk(0x828c000) = 0x828c000
brk(0x828e000) = 0x828e000
brk(0x8290000) = 0x8290000
brk(0x8292000) = 0x8292000
brk(0x8294000) = 0x8294000
brk(0x8295000) = 0x8295000
brk(0x8297000) = 0x8297000
brk(0x8299000) = 0x8299000
brk(0x829b000) = 0x829b000
brk(0x829d000) = 0x829d000
brk(0x829f000) = 0x829f000
brk(0x82a1000) = 0x82a1000
brk(0x82a3000) = 0x82a3000
brk(0x82a5000) = 0x82a5000
brk(0x82a7000) = 0x82a7000
brk(0x82a9000) = 0x82a9000
brk(0x82ab000) = 0x82ab000
brk(0x82ad000) = 0x82ad000
brk(0x82af000) = 0x82af000
brk(0x82b1000) = 0x82b1000
brk(0x82b3000) = 0x82b3000
brk(0x82b5000) = 0x82b5000
brk(0x82b7000) = 0x82b7000
brk(0x82b9000) = 0x82b9000
brk(0x82bb000) = 0x82bb000
brk(0x82bd000) = 0x82bd000
brk(0x82bf000) = 0x82bf000
brk(0x82c1000) = 0x82c1000
brk(0x82c3000) = 0x82c3000
brk(0x82c5000) = 0x82c5000
brk(0x82c7000) = 0x82c7000
brk(0x82c9000) = 0x82c9000
brk(0x82cb000) = 0x82cb000
brk(0x82cd000) = 0x82cd000
brk(0x82cf000) = 0x82cf000
brk(0x82d1000) = 0x82d1000
brk(0x82d3000) = 0x82d3000
brk(0x82d5000) = 0x82d5000
brk(0x82d6000) = 0x82d6000
brk(0x82d8000) = 0x82d8000
brk(0x82da000) = 0x82da000
brk(0x82dc000) = 0x82dc000
brk(0x82de000) = 0x82de000
brk(0x82e0000) = 0x82e0000
brk(0x82e2000) = 0x82e2000
brk(0x82e4000) = 0x82e4000
brk(0x82e6000) = 0x82e6000
brk(0x82e8000) = 0x82e8000
brk(0x82ea000) = 0x82ea000
brk(0x82ec000) = 0x82ec000
brk(0x82ee000) = 0x82ee000
brk(0x82f0000) = 0x82f0000
brk(0x82f2000) = 0x82f2000
brk(0x82f4000) = 0x82f4000
brk(0x82f6000) = 0x82f6000
brk(0x82f8000) = 0x82f8000
brk(0x82fa000) = 0x82fa000
brk(0x82fc000) = 0x82fc000
brk(0x82fe000) = 0x82fe000
brk(0x8300000) = 0x8300000
brk(0x8302000) = 0x8302000
brk(0x8304000) = 0x8304000
brk(0x8306000) = 0x8306000
brk(0x8308000) = 0x8308000
brk(0x830a000) = 0x830a000
brk(0x830c000) = 0x830c000
brk(0x830e000) = 0x830e000
brk(0x8310000) = 0x8310000
brk(0x8312000) = 0x8312000
brk(0x8314000) = 0x8314000
brk(0x8316000) = 0x8316000
brk(0x8317000) = 0x8317000
brk(0x8319000) = 0x8319000
brk(0x831b000) = 0x831b000
brk(0x831d000) = 0x831d000
brk(0x831f000) = 0x831f000
brk(0x8321000) = 0x8321000
brk(0x8323000) = 0x8323000
brk(0x8325000) = 0x8325000
brk(0x8327000) = 0x8327000
brk(0x8329000) = 0x8329000
brk(0x832b000) = 0x832b000
brk(0x832d000) = 0x832d000
brk(0x832f000) = 0x832f000
brk(0x8331000) = 0x8331000
brk(0x8333000) = 0x8333000
brk(0x8335000) = 0x8335000
brk(0x8337000) = 0x8337000
brk(0x8339000) = 0x8339000
brk(0x833b000) = 0x833b000
brk(0x833d000) = 0x833d000
brk(0x833f000) = 0x833f000
brk(0x8341000) = 0x8341000
brk(0x8343000) = 0x8343000
brk(0x8345000) = 0x8345000
brk(0x8347000) = 0x8347000
brk(0x8349000) = 0x8349000
brk(0x834b000) = 0x834b000
brk(0x834d000) = 0x834d000
brk(0x834f000) = 0x834f000
brk(0x8351000) = 0x8351000
brk(0x8353000) = 0x8353000
brk(0x8355000) = 0x8355000
brk(0x8357000) = 0x8357000
brk(0x8358000) = 0x8358000
brk(0x835a000) = 0x835a000
brk(0x835c000) = 0x835c000
brk(0x835e000) = 0x835e000
brk(0x8360000) = 0x8360000
brk(0x8362000) = 0x8362000
brk(0x8364000) = 0x8364000
brk(0x8366000) = 0x8366000
brk(0x8368000) = 0x8368000
brk(0x836a000) = 0x836a000
brk(0x836c000) = 0x836c000
brk(0x836e000) = 0x836e000
brk(0x8370000) = 0x8370000
brk(0x8372000) = 0x8372000
brk(0x8374000) = 0x8374000
brk(0x8376000) = 0x8376000
brk(0x8378000) = 0x8378000
brk(0x837a000) = 0x837a000
brk(0x837c000) = 0x837c000
brk(0x837e000) = 0x837e000
brk(0x8380000) = 0x8380000
brk(0x8382000) = 0x8382000
brk(0x8384000) = 0x8384000
brk(0x8386000) = 0x8386000
brk(0x8388000) = 0x8388000
brk(0x838a000) = 0x838a000
brk(0x838c000) = 0x838c000
brk(0x838e000) = 0x838e000
brk(0x8390000) = 0x8390000
brk(0x8392000) = 0x8392000
brk(0x8394000) = 0x8394000
brk(0x8396000) = 0x8396000
brk(0x8398000) = 0x8398000
brk(0x8399000) = 0x8399000
brk(0x839b000) = 0x839b000
brk(0x839d000) = 0x839d000
brk(0x839f000) = 0x839f000
brk(0x83a1000) = 0x83a1000
brk(0x83a3000) = 0x83a3000
brk(0x83a5000) = 0x83a5000
brk(0x83a7000) = 0x83a7000
brk(0x83a9000) = 0x83a9000
brk(0x83ab000) = 0x83ab000
brk(0x83ad000) = 0x83ad000
brk(0x83af000) = 0x83af000
brk(0x83b1000) = 0x83b1000
brk(0x83b3000) = 0x83b3000
brk(0x83b5000) = 0x83b5000
brk(0x83b7000) = 0x83b7000
brk(0x83b9000) = 0x83b9000
brk(0x83bb000) = 0x83bb000
brk(0x83bd000) = 0x83bd000
brk(0x83bf000) = 0x83bf000
brk(0x83c1000) = 0x83c1000
brk(0x83c3000) = 0x83c3000
brk(0x83c5000) = 0x83c5000
brk(0x83c7000) = 0x83c7000
brk(0x83c9000) = 0x83c9000
brk(0x83cb000) = 0x83cb000
brk(0x83cd000) = 0x83cd000
brk(0x83cf000) = 0x83cf000
brk(0x83d1000) = 0x83d1000
brk(0x83d3000) = 0x83d3000
brk(0x83d5000) = 0x83d5000
brk(0x83d7000) = 0x83d7000
brk(0x83d9000) = 0x83d9000
brk(0x83da000) = 0x83da000
brk(0x83dc000) = 0x83dc000
brk(0x83de000) = 0x83de000
brk(0x83e0000) = 0x83e0000
brk(0x83e2000) = 0x83e2000
brk(0x83e4000) = 0x83e4000
brk(0x83e6000) = 0x83e6000
close(9) = 0
stat("/home/dean/ap2/htdocs/README", 0xbffff6e4) = -1 ENOENT (No such file or directory)
stat("/home/dean/ap2/htdocs", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
brk(0x83e8000) = 0x83e8000
writev(8, [{"HTTP/1.1 200 OK\r\nDate: Sun, 07 J"..., 251}, {"<!DOCTYPE HTML PUBLIC \"-//W3C//D"..., 56}, {"<HTML>\n <HEAD>\n <TITLE>Index of"..., 33}, {"/", 1}, {"</TITLE>\n </HEAD>\n <BODY>\n", 26}, {"<H1>Index of ", 13}, {"/", 1}, {"</H1>\n", 6}, {"<PRE>", 5}, {"<IMG SRC=\"", 10}, {"/icons/blank.gif", 16}, {"\" ALT=\" \"", 13}, {"> ", 2}, {"<A HREF=\"", 9}, {"?N=D", 4}, {"\">", 2}], 16) = 448
writev(8, [{"Name", 4}, {"</A>", 4}, {" ", 19}, {" ", 1}, {"<A HREF=\"", 9}, {"?M=A", 4}, {"\">", 2}, {"Last modified", 13}, {"</A>", 4}, {" ", 7}, {"<A HREF=\"", 9}, {"?S=A", 4}, {"\">", 2}, {"Size", 4}, {"</A>", 4}, {" ", 2}], 16) = 92
writev(8, [{"<A HREF=\"", 9}, {"?D=A", 4}, {"\">", 2}, {"Description", 11}, {"</A>", 4}, {"\n<HR>\n", 6}, {"<IMG SRC=\"", 10}, {"/icons/back.gif", 15}, {"\" ALT=\"[", 8}, {"DIR", 3}, {"]\"", 2}, {">", 1}, {" <A HREF=\"", 10}, {"/", 1}, {"\">", 2}, {"Parent Directory", 16}], 16) = 104
writev(8, [{"</A>", 4}, {" ", 7}, {" ", 1}, {"07-Jan-2001 11:24 ", 19}, {" -", 5}, {" ", 2}, {"\n", 1}, {"<IMG SRC=\"", 10}, {"/icons/image2.gif", 17}, {"\" ALT=\"[", 8}, {"IMG", 3}, {"]\"", 2}, {">", 1}, {" <A HREF=\"", 10}, {"apache_pb.gif", 13}, {"\">", 2}], 16) = 105
writev(8, [{"apache_pb.gif", 13}, {"</A>", 4}, {" ", 10}, {" ", 1}, {"02-Jul-1996 23:18 ", 19}, {" 2k", 5}, {" ", 2}, {"\n", 1}, {"<IMG SRC=\"", 10}, {"/icons/text.gif", 15}, {"\" ALT=\"[", 8}, {"TXT", 3}, {"]\"", 2}, {">", 1}, {" <A HREF=\"", 10}, {"index.html.ca", 13}], 16) = 117
writev(8, [{"\">", 2}, {"index.html.ca", 13}, {"</A>", 4}, {" ", 10}, {" ", 1}, {"03-Dec-2000 08:44 ", 19}, {" 2k", 5}, {" ", 2}, {"\n", 1}, {"<IMG SRC=\"", 10}, {"/icons/text.gif", 15}, {"\" ALT=\"[", 8}, {"TXT", 3}, {"]\"", 2}, {">", 1}, {" <A HREF=\"", 10}], 16) = 106
writev(8, [{"index.html.cz", 13}, {"\">", 2}, {"index.html.cz", 13}, {"</A>", 4}, {" ", 10}, {" ", 1}, {"14-Nov-2000 17:11 ", 19}, {" 2k", 5}, {" ", 2}, {"\n", 1}, {"<IMG SRC=\"", 10}, {"/icons/text.gif", 15}, {"\" ALT=\"[", 8}, {"TXT", 3}, {"]\"", 2}, {">", 1}], 16) = 109
writev(8, [{" <A HREF=\"", 10}, {"index.html.de", 13}, {"\">", 2}, {"index.html.de", 13}, {"</A>", 4}, {" ", 10}, {" ", 1}, {"03-Dec-2000 08:44 ", 19}, {" 2k", 5}, {" ", 2}, {"\n", 1}, {"<IMG SRC=\"", 10}, {"/icons/text.gif", 15}, {"\" ALT=\"[", 8}, {"TXT", 3}, {"]\"", 2}], 16) = 118
writev(8, [{">", 1}, {" <A HREF=\"", 10}, {"index.html.dk", 13}, {"\">", 2}, {"index.html.dk", 13}, {"</A>", 4}, {" ", 10}, {" ", 1}, {"14-Nov-2000 17:11 ", 19}, {" 2k", 5}, {" ", 2}, {"\n", 1}, {"<IMG SRC=\"", 10}, {"/icons/unknown.gif", 18}, {"\" ALT=\"[", 8}, {" ", 3}], 16) = 120
writev(8, [{"]\"", 2}, {">", 1}, {" <A HREF=\"", 10}, {"index.html.ee", 13}, {"\">", 2}, {"index.html.ee", 13}, {"</A>", 4}, {" ", 10}, {" ", 1}, {"14-Nov-2000 17:11 ", 19}, {" 2k", 5}, {" ", 2}, {"\n", 1}, {"<IMG SRC=\"", 10}, {"/icons/text.gif", 15}, {"\" ALT=\"[", 8}], 16) = 116
writev(8, [{"TXT", 3}, {"]\"", 2}, {">", 1}, {" <A HREF=\"", 10}, {"index.html.el", 13}, {"\">", 2}, {"index.html.el", 13}, {"</A>", 4}, {" ", 10}, {" ", 1}, {"14-Dec-2000 19:46 ", 19}, {" 2k", 5}, {" ", 2}, {"\n", 1}, {"<IMG SRC=\"", 10}, {"/icons/text.gif", 15}], 16) = 111
writev(8, [{"\" ALT=\"[", 8}, {"TXT", 3}, {"]\"", 2}, {">", 1}, {" <A HREF=\"", 10}, {"index.html.en", 13}, {"\">", 2}, {"index.html.en", 13}, {"</A>", 4}, {" ", 10}, {" ", 1}, {"03-Dec-2000 08:44 ", 19}, {" 1k", 5}, {" ", 2}, {"\n", 1}, {"<IMG SRC=\"", 10}], 16) = 104
writev(8, [{"/icons/text.gif", 15}, {"\" ALT=\"[", 8}, {"TXT", 3}, {"]\"", 2}, {">", 1}, {" <A HREF=\"", 10}, {"index.html.es", 13}, {"\">", 2}, {"index.html.es", 13}, {"</A>", 4}, {" ", 10}, {" ", 1}, {"14-Nov-2000 17:11 ", 19}, {" 2k", 5}, {" ", 2}, {"\n", 1}], 16) = 109
writev(8, [{"<IMG SRC=\"", 10}, {"/icons/text.gif", 15}, {"\" ALT=\"[", 8}, {"TXT", 3}, {"]\"", 2}, {">", 1}, {" <A HREF=\"", 10}, {"index.html.et", 13}, {"\">", 2}, {"index.html.et", 13}, {"</A>", 4}, {" ", 10}, {" ", 1}, {"14-Nov-2000 17:28 ", 19}, {" 2k", 5}, {" ", 2}], 16) = 118
writev(8, [{"\n", 1}, {"<IMG SRC=\"", 10}, {"/icons/text.gif", 15}, {"\" ALT=\"[", 8}, {"TXT", 3}, {"]\"", 2}, {">", 1}, {" <A HREF=\"", 10}, {"index.html.fr", 13}, {"\">", 2}, {"index.html.fr", 13}, {"</A>", 4}, {" ", 10}, {" ", 1}, {"04-Jan-2001 04:02 ", 19}, {" 2k", 5}], 16) = 117
writev(8, [{" ", 2}, {"\n", 1}, {"<IMG SRC=\"", 10}, {"/icons/unknown.gif", 18}, {"\" ALT=\"[", 8}, {" ", 3}, {"]\"", 2}, {">", 1}, {" <A HREF=\"", 10}, {"index.html.he.iso8859-8", 23}, {"\">", 2}, {"index.html.he.iso8859-8", 23}, {"</A>", 4}, {" ", 1}, {"03-Dec-2000 08:48 ", 19}, {" 4k", 5}], 16) = 132
writev(8, [{" ", 2}, {"\n", 1}, {"<IMG SRC=\"", 10}, {"/icons/text.gif", 15}, {"\" ALT=\"[", 8}, {"TXT", 3}, {"]\"", 2}, {">", 1}, {" <A HREF=\"", 10}, {"index.html.it", 13}, {"\">", 2}, {"index.html.it", 13}, {"</A>", 4}, {" ", 10}, {" ", 1}, {"03-Dec-2000 08:44 ", 19}], 16) = 114
writev(8, [{" 2k", 5}, {" ", 2}, {"\n", 1}, {"<IMG SRC=\"", 10}, {"/icons/text.gif", 15}, {"\" ALT=\"[", 8}, {"TXT", 3}, {"]\"", 2}, {">", 1}, {" <A HREF=\"", 10}, {"index.html.ja.iso2022-jp", 24}, {"\">", 2}, {"index.html.ja.iso202..>", 26}, {"</A>", 4}, {" ", 1}, {"03-Dec-2000 08:48 ", 19}], 16) = 133
writev(8, [{" 2k", 5}, {" ", 2}, {"\n", 1}, {"<IMG SRC=\"", 10}, {"/icons/text.gif", 15}, {"\" ALT=\"[", 8}, {"TXT", 3}, {"]\"", 2}, {">", 1}, {" <A HREF=\"", 10}, {"index.html.ja.jis", 17}, {"\">", 2}, {"index.html.ja.jis", 17}, {"</A>", 4}, {" ", 6}, {" ", 1}], 16) = 104
writev(8, [{"03-Dec-2000 08:48 ", 19}, {" 2k", 5}, {" ", 2}, {"\n", 1}, {"<IMG SRC=\"", 10}, {"/icons/unknown.gif", 18}, {"\" ALT=\"[", 8}, {" ", 3}, {"]\"", 2}, {">", 1}, {" <A HREF=\"", 10}, {"index.html.kr.iso-kr", 20}, {"\">", 2}, {"index.html.kr.iso-kr", 20}, {"</A>", 4}, {" ", 3}], 16) = 128
writev(8, [{" ", 1}, {"03-Dec-2000 08:48 ", 19}, {" 1k", 5}, {" ", 2}, {"\n", 1}, {"<IMG SRC=\"", 10}, {"/icons/text.gif", 15}, {"\" ALT=\"[", 8}, {"TXT", 3}, {"]\"", 2}, {">", 1}, {" <A HREF=\"", 10}, {"index.html.kr.iso2022-kr", 24}, {"\">", 2}, {"index.html.kr.iso202..>", 26}, {"</A>", 4}], 16) = 133
writev(8, [{" ", 1}, {"03-Dec-2000 08:48 ", 19}, {" 1k", 5}, {" ", 2}, {"\n", 1}, {"<IMG SRC=\"", 10}, {"/icons/text.gif", 15}, {"\" ALT=\"[", 8}, {"TXT", 3}, {"]\"", 2}, {">", 1}, {" <A HREF=\"", 10}, {"index.html.ltz", 14}, {"\">", 2}, {"index.html.ltz", 14}, {"</A>", 4}], 16) = 111
writev(8, [{" ", 9}, {" ", 1}, {"03-Dec-2000 08:44 ", 19}, {" 2k", 5}, {" ", 2}, {"\n", 1}, {"<IMG SRC=\"", 10}, {"/icons/unknown.gif", 18}, {"\" ALT=\"[", 8}, {" ", 3}, {"]\"", 2}, {">", 1}, {" <A HREF=\"", 10}, {"index.html.lu", 13}, {"\">", 2}, {"index.html.lu", 13}], 16) = 117
writev(8, [{"</A>", 4}, {" ", 10}, {" ", 1}, {"03-Dec-2000 08:44 ", 19}, {" 2k", 5}, {" ", 2}, {"\n", 1}, {"<IMG SRC=\"", 10}, {"/icons/text.gif", 15}, {"\" ALT=\"[", 8}, {"TXT", 3}, {"]\"", 2}, {">", 1}, {" <A HREF=\"", 10}, {"index.html.nl", 13}, {"\">", 2}], 16) = 106
writev(8, [{"index.html.nl", 13}, {"</A>", 4}, {" ", 10}, {" ", 1}, {"14-Nov-2000 17:11 ", 19}, {" 2k", 5}, {" ", 2}, {"\n", 1}, {"<IMG SRC=\"", 10}, {"/icons/text.gif", 15}, {"\" ALT=\"[", 8}, {"TXT", 3}, {"]\"", 2}, {">", 1}, {" <A HREF=\"", 10}, {"index.html.no", 13}], 16) = 117
writev(8, [{"\">", 2}, {"index.html.no", 13}, {"</A>", 4}, {" ", 10}, {" ", 1}, {"19-Oct-2000 12:18 ", 19}, {" 1k", 5}, {" ", 2}, {"\n", 1}, {"<IMG SRC=\"", 10}, {"/icons/unknown.gif", 18}, {"\" ALT=\"[", 8}, {" ", 3}, {"]\"", 2}, {">", 1}, {" <A HREF=\"", 10}], 16) = 109
writev(8, [{"index.html.po.iso-pl", 20}, {"\">", 2}, {"index.html.po.iso-pl", 20}, {"</A>", 4}, {" ", 3}, {" ", 1}, {"19-Dec-2000 13:30 ", 19}, {" 1k", 5}, {" ", 2}, {"\n", 1}, {"<IMG SRC=\"", 10}, {"/icons/text.gif", 15}, {"\" ALT=\"[", 8}, {"TXT", 3}, {"]\"", 2}, {">", 1}], 16) = 116
writev(8, [{" <A HREF=\"", 10}, {"index.html.pt", 13}, {"\">", 2}, {"index.html.pt", 13}, {"</A>", 4}, {" ", 10}, {" ", 1}, {"03-Dec-2000 08:44 ", 19}, {" 2k", 5}, {" ", 2}, {"\n", 1}, {"<IMG SRC=\"", 10}, {"/icons/text.gif", 15}, {"\" ALT=\"[", 8}, {"TXT", 3}, {"]\"", 2}], 16) = 118
writev(8, [{">", 1}, {" <A HREF=\"", 10}, {"index.html.pt-br", 16}, {"\">", 2}, {"index.html.pt-br", 16}, {"</A>", 4}, {" ", 7}, {" ", 1}, {"14-Nov-2000 17:11 ", 19}, {" 2k", 5}, {" ", 2}, {"\n", 1}, {"<IMG SRC=\"", 10}, {"/icons/text.gif", 15}, {"\" ALT=\"[", 8}, {"TXT", 3}], 16) = 120
writev(8, [{"]\"", 2}, {">", 1}, {" <A HREF=\"", 10}, {"index.html.ru.cp-1251", 21}, {"\">", 2}, {"index.html.ru.cp-1251", 21}, {"</A>", 4}, {" ", 2}, {" ", 1}, {"03-Dec-2000 08:48 ", 19}, {" 2k", 5}, {" ", 2}, {"\n", 1}, {"<IMG SRC=\"", 10}, {"/icons/text.gif", 15}, {"\" ALT=\"[", 8}], 16) = 124
writev(8, [{"TXT", 3}, {"]\"", 2}, {">", 1}, {" <A HREF=\"", 10}, {"index.html.ru.cp866", 19}, {"\">", 2}, {"index.html.ru.cp866", 19}, {"</A>", 4}, {" ", 4}, {" ", 1}, {"03-Dec-2000 08:48 ", 19}, {" 2k", 5}, {" ", 2}, {"\n", 1}, {"<IMG SRC=\"", 10}, {"/icons/text.gif", 15}], 16) = 117
writev(8, [{"\" ALT=\"[", 8}, {"TXT", 3}, {"]\"", 2}, {">", 1}, {" <A HREF=\"", 10}, {"index.html.ru.iso-ru", 20}, {"\">", 2}, {"index.html.ru.iso-ru", 20}, {"</A>", 4}, {" ", 3}, {" ", 1}, {"03-Dec-2000 08:48 ", 19}, {" 2k", 5}, {" ", 2}, {"\n", 1}, {"<IMG SRC=\"", 10}], 16) = 111
writev(8, [{"/icons/text.gif", 15}, {"\" ALT=\"[", 8}, {"TXT", 3}, {"]\"", 2}, {">", 1}, {" <A HREF=\"", 10}, {"index.html.ru.koi8-r", 20}, {"\">", 2}, {"index.html.ru.koi8-r", 20}, {"</A>", 4}, {" ", 3}, {" ", 1}, {"03-Dec-2000 08:48 ", 19}, {" 2k", 5}, {" ", 2}, {"\n", 1}], 16) = 116
writev(8, [{"<IMG SRC=\"", 10}, {"/icons/text.gif", 15}, {"\" ALT=\"[", 8}, {"TXT", 3}, {"]\"", 2}, {">", 1}, {" <A HREF=\"", 10}, {"index.html.ru.ucs2", 18}, {"\">", 2}, {"index.html.ru.ucs2", 18}, {"</A>", 4}, {" ", 5}, {" ", 1}, {"04-Apr-2000 08:41 ", 19}, {" 3k", 5}, {" ", 2}], 16) = 123
writev(8, [{"\n", 1}, {"<IMG SRC=\"", 10}, {"/icons/text.gif", 15}, {"\" ALT=\"[", 8}, {"TXT", 3}, {"]\"", 2}, {">", 1}, {" <A HREF=\"", 10}, {"index.html.ru.ucs4", 18}, {"\">", 2}, {"index.html.ru.ucs4", 18}, {"</A>", 4}, {" ", 5}, {" ", 1}, {"04-Apr-2000 08:41 ", 19}, {" 6k", 5}], 16) = 122
writev(8, [{" ", 2}, {"\n", 1}, {"<IMG SRC=\"", 10}, {"/icons/text.gif", 15}, {"\" ALT=\"[", 8}, {"TXT", 3}, {"]\"", 2}, {">", 1}, {" <A HREF=\"", 10}, {"index.html.ru.utf8", 18}, {"\">", 2}, {"index.html.ru.utf8", 18}, {"</A>", 4}, {" ", 5}, {" ", 1}, {"03-Dec-2000 08:48 ", 19}], 16) = 119
writev(8, [{" 2k", 5}, {" ", 2}, {"\n", 1}, {"<IMG SRC=\"", 10}, {"/icons/text.gif", 15}, {"\" ALT=\"[", 8}, {"TXT", 3}, {"]\"", 2}, {">", 1}, {" <A HREF=\"", 10}, {"index.html.se", 13}, {"\">", 2}, {"index.html.se", 13}, {"</A>", 4}, {" ", 10}, {" ", 1}], 16) = 100
writev(8, [{"03-Dec-2000 08:44 ", 19}, {" 2k", 5}, {" ", 2}, {"\n", 1}, {"<IMG SRC=\"", 10}, {"/icons/text.gif", 15}, {"\" ALT=\"[", 8}, {"TXT", 3}, {"]\"", 2}, {">", 1}, {" <A HREF=\"", 10}, {"index.html.tw", 13}, {"\">", 2}, {"index.html.tw", 13}, {"</A>", 4}, {" ", 10}], 16) = 118
writev(8, [{" ", 1}, {"03-Dec-2000 08:44 ", 19}, {" 1k", 5}, {" ", 2}, {"\n", 1}, {"<IMG SRC=\"", 10}, {"/icons/text.gif", 15}, {"\" ALT=\"[", 8}, {"TXT", 3}, {"]\"", 2}, {">", 1}, {" <A HREF=\"", 10}, {"index.html.tw.Big5", 18}, {"\">", 2}, {"index.html.tw.Big5", 18}, {"</A>", 4}], 16) = 119
writev(8, [{" ", 5}, {" ", 1}, {"03-Dec-2000 08:48 ", 19}, {" 1k", 5}, {" ", 2}, {"\n", 1}, {"<IMG SRC=\"", 10}, {"/icons/folder.gif", 17}, {"\" ALT=\"[", 8}, {"DIR", 3}, {"]\"", 2}, {">", 1}, {" <A HREF=\"", 10}, {"manual/", 7}, {"\">", 2}, {"manual/", 7}], 16) = 100
read(8, 0x814cfb0, 8192) = -1 EAGAIN (Resource temporarily unavailable)
writev(8, [{"</A>", 4}, {" ", 16}, {" ", 1}, {"07-Jan-2001 10:36 ", 19}, {" -", 5}, {" ", 2}, {"\n", 1}, {"</PRE>", 6}, {"<HR>\n", 5}, {"<ADDRESS>Apache/2.0b1-dev Server"..., 76}, {"</BODY></HTML>\n", 15}], 11) = 150
brk(0x8150000) = 0x8150000
write(4, "204.107.140.52 - - [07/Jan/2001:"..., 74) = 74
shutdown(8, 1 /* send */) = 0
gettimeofday({978896760, 422093}, NULL) = 0
read(8, 0xbffff760, 512) = -1 EAGAIN (Resource temporarily unavailable)
select(9, [8], NULL, NULL, {30, 0}) = 1 (in [8], left {29, 980000})
read(8, "", 512) = 0
close(8) = 0
rt_sigaction(SIGUSR1, {0x807f774, [], SA_INTERRUPT|0x4000000}, {SIG_IGN}, 8) = 0
accept(5, <unfinished ...>
Re: woah, "GET /" with autoindex
Posted by rb...@covalent.net.
On Sun, 7 Jan 2001 rbb@covalent.net wrote:
>
> > > I agree this is a showstopper for the release, I believe it is fine to
> > > release a beta with this issue. There are multiple attacks for solving
> > > this problem.
> >
> > my main concern with calling it a showstopper is that i don't know
> > what the APIs look like yet... and maybe they'll need changing to fix
> > these bugs. if you're cool with API changes during beta then it's not
> > a beta showstopper.
>
> The API's shouldn't change, because
Ignore that.
Ryan
_______________________________________________________________________________
Ryan Bloom rbb@apache.org
406 29th St.
San Francisco, CA 94131
-------------------------------------------------------------------------------
Re: woah, "GET /" with autoindex
Posted by rb...@covalent.net.
> > I agree this is a showstopper for the release, I believe it is fine to
> > release a beta with this issue. There are multiple attacks for solving
> > this problem.
>
> my main concern with calling it a showstopper is that i don't know
> what the APIs look like yet... and maybe they'll need changing to fix
> these bugs. if you're cool with API changes during beta then it's not
> a beta showstopper.
The API's shouldn't change, because
> > #1, the sbrk's. We need to keep a list of free buckets, and not allocate
> > a new bucket each time we create one. At some point, each process will
> > create a maximum number of buckets, and we will stop allocating new ones.
>
> are you allocating the buckets out of the request or connection pool?
> or via malloc? if it's malloc or otherwise shared across multiple
> threads then we'll run into lock contention on the list/malloc (on
> multi-CPU boxes only).
The buckets are allocated using malloc, because they need to be able to
move from the request to the connection. The bucket_list will need to be
locked, preferably using a simple atomic increment/decrement lock.
> > This solves half the problem, but we are still allocating FAR too many
> > buckets for auto-index's. The problem is that auto-index was written for
> > the old API, and we haven't tried to optimize that API yet. The ap_r*
> > functions should all buffer data, so that they don't create a bucket per
> > char.
>
> i don't really see ap_r* as the problem, i see ap_bucket_putstr/printf
> as the problem. fixing ap_r* just helps folks writing code within httpd,
> it wouldn't help folks using buckets in other apps.
The thing is, our bucket implementation really relies on people knowing a
lot about their data. I think that ap_bucket_putstr/printf should just go
away. They don't really work, and they were added long before we arrived
at our current API for the buckets. There is a reason nobody ever used
them when porting our modules.
> > We don't force zero-copy on anybody, but the core needs to figure out how
> > to buffer the data, which is just going to take somebody putting the logic
> > in. It's not terribly complex, but nobody has had the time/inclination.
>
> yeah, there's not much logic. if a module uses ap_r{put,printf} then it
> should be buffered (one-copied), if it uses ap_rwrite/sendmmap/sendfile
> then it should be zero-copied. that's essentially what 1.3 does... and
> seems to work fine.
Modules really shouldn't be using those interfaces anymore. Any module
that uses those interfaces is essentially going to be doing
one-copy. Modules that really want to perform will create buckets, and
store data in the buckets directly. Or, they will create their own bucket
types, and just use those.
Ryan
_______________________________________________________________________________
Ryan Bloom rbb@apache.org
406 29th St.
San Francisco, CA 94131
-------------------------------------------------------------------------------
Re: woah, "GET /" with autoindex
Posted by dean gaudet <dg...@arctic.org>.
On Sun, 7 Jan 2001 rbb@covalent.net wrote:
> I agree this is a showstopper for the release, I believe it is fine to
> release a beta with this issue. There are multiple attacks for solving
> this problem.
my main concern with calling it a showstopper is that i don't know
what the APIs look like yet... and maybe they'll need changing to fix
these bugs. if you're cool with API changes during beta then it's not
a beta showstopper.
> #1, the sbrk's. We need to keep a list of free buckets, and not allocate
> a new bucket each time we create one. At some point, each process will
> create a maximum number of buckets, and we will stop allocating new ones.
are you allocating the buckets out of the request or connection pool?
or via malloc? if it's malloc or otherwise shared across multiple
threads then we'll run into lock contention on the list/malloc (on
multi-CPU boxes only).
> This solves half the problem, but we are still allocating FAR too many
> buckets for auto-index's. The problem is that auto-index was written for
> the old API, and we haven't tried to optimize that API yet. The ap_r*
> functions should all buffer data, so that they don't create a bucket per
> char.
i don't really see ap_r* as the problem, i see ap_bucket_putstr/printf
as the problem. fixing ap_r* just helps folks writing code within httpd,
it wouldn't help folks using buckets in other apps.
> We don't force zero-copy on anybody, but the core needs to figure out how
> to buffer the data, which is just going to take somebody putting the logic
> in. It's not terribly complex, but nobody has had the time/inclination.
yeah, there's not much logic. if a module uses ap_r{put,printf} then it
should be buffered (one-copied), if it uses ap_rwrite/sendmmap/sendfile
then it should be zero-copied. that's essentially what 1.3 does... and
seems to work fine.
obviously someone could deliberately write code which does printfs of
massive data, but they deserve to break.
-dean
Re: woah, "GET /" with autoindex
Posted by rb...@covalent.net.
Ok, I can answer a lot of these questions for you.
> woah. ok there's some serious problems here.
We know, and many of them are fixed, but not enabled, because they tend to
hide other problems, or the fix is designed, but not implemented.
> this is from a "GET / HTTP/1.0" which results in an autoindex (dunno why
> it's not grabbing one of the index.htmls, but i'm not worrying about that,
> the autoindex is probably far more interesting).
This is a known bug. Ben Laurie is working on the fix.
> also, this is a perfect example of why zero-copy can be totally wrong.
> passing 100 bytes to the kernel in each writev() in 8 elements is way
> slower than passing the kernel 4096 byte buffers. i haven't written up
> the test to prove it to you, but this is one of the exact problems i had
> to deal with on the last zero-copy library i worked with.
>
> this one is showstopper material.
I agree this is a showstopper for the release, I believe it is fine to
release a beta with this issue. There are multiple attacks for solving
this problem.
#1, the sbrk's. We need to keep a list of free buckets, and not allocate
a new bucket each time we create one. At some point, each process will
create a maximum number of buckets, and we will stop allocating new ones.
This solves half the problem, but we are still allocating FAR too many
buckets for auto-index's. The problem is that auto-index was written for
the old API, and we haven't tried to optimize that API yet. The ap_r*
functions should all buffer data, so that they don't create a bucket per
char.
We have a coalesce filter someplace, but I'm not sure that it has ever
been turned on.
In reality, I believe that mod_autoindex should be completely re-written
to be MUCH cleaner. This would allow it to be used for HTTP and FTP, by
just changing one simple function, the thing that formats the results of
the stat into a string of HTML or raw data. This has been designed, we
just need time to implement it.
> in general you want to buffer when modules are doing dynamic generation
> and you want to zero-copy when modules are sending bulk data.
We don't force zero-copy on anybody, but the core needs to figure out how
to buffer the data, which is just going to take somebody putting the logic
in. It's not terribly complex, but nobody has had the time/inclination.
Ryan
_______________________________________________________________________________
Ryan Bloom rbb@apache.org
406 29th St.
San Francisco, CA 94131
-------------------------------------------------------------------------------
Re: woah, "GET /" with autoindex
Posted by Greg Stein <gs...@lyra.org>.
On Thu, Jan 11, 2001 at 02:18:02PM -0800, rbb@covalent.net wrote:
>...
> It is an optimization. We don't need it today. Let's table this until
> after the beta.
Reminder: if you want to table something, then add a note to STATUS.
Cheers,
-g
--
Greg Stein, http://www.lyra.org/
Re: woah, "GET /" with autoindex
Posted by rb...@covalent.net.
> > The answer is to optimize ap_r* to not require so much allocation. It might
> > also be a good idea to insert a COALESCE filter IFF ap_r* gets called (e.g.
> > insert the filter on the first ap_r* call).
>
> Sorry, but I don't believe that's the _only_ answer, or optimal either.
> Perhaps something like an ap_rflush triggered somehow by this mixture of
> calls you mentioned above.
Dean is 100% correct about the COALESCE filter. It is the wrong solution
99.9% of the time. It has the correct goal, but in general, it means that
we copy the same data two or even three times, instead of once or never.
I have been thinking along different lines. What if we remove the
coalesce filter altogether, and add a char * to the brigade structure.
The char * points to a 4K buffer that is allocated when needed. When we
create heap, or transient, or pool buckets, we copy the data into the
buffer, if it makes sense. Then, when the buffer is full, we create an
actual bucket, and put it at the end of the brigade. This has a few
issues, and it will require smudging the line between buckets and filters,
but it could work.
It is an optimization. We don't need it today. Let's table this until
after the beta.
Ryan
_______________________________________________________________________________
Ryan Bloom rbb@apache.org
406 29th St.
San Francisco, CA 94131
-------------------------------------------------------------------------------
Re: woah, "GET /" with autoindex
Posted by Greg Ames <gr...@raleigh.ibm.com>.
Greg Stein wrote:
>
> This kind of solution won't work. If you end up mixing ap_r* with bucket
> calls, then you're screwed. The bucket would hit the chain before the
> contents of the coalesce buffer.
>
> And note that it would be quite easy for the mixture to happen. Handler H
> starts writing some content with ap_r*(). Then, it calls ap_cool_utility(r)
> which dumps some content via buckets. Oops!
>
good call...thanks!
> The answer is to optimize ap_r* to not require so much allocation. It might
> also be a good idea to insert a COALESCE filter IFF ap_r* gets called (e.g.
> insert the filter on the first ap_r* call).
>
Sorry, but I don't believe that's the _only_ answer, or optimal either.
Perhaps something like an ap_rflush triggered somehow by this mixture of
calls you mentioned above.
> There was a question about whether other modules use ap_r* for lots of bits.
> Absolutely... mod_dav is a simple example. PHP uses ap_r*, too.
>
ooooops...my apologies for not being more clear. I was trying to ask
the opposite question: would there be any known cases of lots of
"tinygrams" going thru buckets directly, once we do the Right Thing in
ap_r* ?
...but the mod_dav case is interesting also. Wouldn't that primarily be
big data (file uploads/downloads) mixed with a few small handshake type
packets?
Thanks,
Greg
Re: woah, "GET /" with autoindex
Posted by Greg Stein <gs...@lyra.org>.
This kind of solution won't work. If you end up mixing ap_r* with bucket
calls, then you're screwed. The bucket would hit the chain before the
contents of the coalesce buffer.
And note that it would be quite easy for the mixture to happen. Handler H
starts writing some content with ap_r*(). Then, it calls ap_cool_utility(r)
which dumps some content via buckets. Oops!
The answer is to optimize ap_r* to not require so much allocation. It might
also be a good idea to insert a COALESCE filter IFF ap_r* gets called (e.g.
insert the filter on the first ap_r* call).
There was a question about whether other modules use ap_r* for lots of bits.
Absolutely... mod_dav is a simple example. PHP uses ap_r*, too.
Cheers,
-g
On Wed, Jan 10, 2001 at 05:24:58PM -0500, Victor J. Orlikowski wrote:
> (*blink blink*)
> Huh?
> (victor scrounges around. darn - not on his backup cd's...maybe in
> new-httpd archives - yes....)
>
> Sorry - re-packaging old source from another project rots the brain.
>
> This patch was only a preliminary idea, and showed slightly better
> performance, but it is still up for improvement. I've updated it to
> work against the current build tree.
>...
--
Greg Stein, http://www.lyra.org/
Re: woah, "GET /" with autoindex
Posted by "Victor J. Orlikowski" <v....@gte.net>.
(*blink blink*)
Huh?
(victor scrounges around. darn - not on his backup cd's...maybe in
new-httpd archives - yes....)
Sorry - re-packaging old source from another project rots the brain.
This patch was only a preliminary idea, and showed slightly better
performance, but it is still up for improvement. I've updated it to
work against the current build tree.
Index: include/httpd.h
===================================================================
RCS file: /cvs/apache/httpd-2.0/include/httpd.h,v
retrieving revision 1.129
diff -u -d -r1.129 httpd.h
--- httpd.h 2001/01/05 20:44:39 1.129
+++ httpd.h 2001/01/10 22:25:26
@@ -809,6 +809,18 @@
/** A flag to determine if the eos bucket has been sent yet
* @defvar int eos_sent */
int eos_sent;
+ /** A buffer to hold data to be coalesced into a reasonable size bucket
+ * @defvar char *coalesce_buf */
+ char *coalesce_buf;
+ /** A pointer to the current position in coalesce_buf
+ * @defvar char *coalesce_buf_cur */
+ char *coalesce_buf_cur;
+ /** Space used in coalesce_buf
+ * @defvar apr_size_t coalesce_cnt */
+ apr_size_t coalesce_cnt;
+ /** Space remaining in coalesce_buf
+ * @defvar apr_size_t coalesce_avail */
+ apr_size_t coalesce_avail;
/* Things placed at the end of the record to avoid breaking binary
* compatibility. It would be nice to remember to reorder the entire
Index: modules/http/http_protocol.c
===================================================================
RCS file: /cvs/apache/httpd-2.0/modules/http/http_protocol.c,v
retrieving revision 1.265
diff -u -d -r1.265 http_protocol.c
--- http_protocol.c 2001/01/02 19:22:09 1.265
+++ http_protocol.c 2001/01/10 22:25:29
@@ -1466,6 +1466,10 @@
r->the_request = NULL;
r->output_filters = conn->output_filters;
r->input_filters = conn->input_filters;
+ r->coalesce_buf = apr_pcalloc(r->pool, MAX_STRING_LEN);
+ r->coalesce_buf_cur = r->coalesce_buf;
+ r->coalesce_cnt = 0;
+ r->coalesce_avail = MAX_STRING_LEN;
apr_setsocketopt(conn->client_socket, APR_SO_TIMEOUT,
(int)(conn->keptalive
@@ -2678,6 +2682,7 @@
r = r->next;
}
/* tell the filter chain there is no more content coming */
+ ap_rflush(r);
if (!r->eos_sent) {
end_output_stream(r);
}
@@ -2995,29 +3000,70 @@
}
#endif /* APR_HAS_MMAP */
+#define RMIN_BUCKET_SIZE 4096
+static void coalesce_buf_create_bucket(request_rec *r, ap_bucket_brigade *bb)
+{
+ ap_bucket *b = ap_bucket_create_transient(r->coalesce_buf, r->coalesce_cnt);
+ AP_BRIGADE_INSERT_TAIL(bb, b);
+ r->coalesce_buf_cur = r->coalesce_buf;
+ r->coalesce_cnt = 0;
+ r->coalesce_avail = MAX_STRING_LEN;
+}
+
+static void coalesce_buf_insertc(request_rec *r, char c)
+{
+ *r->coalesce_buf_cur = c;
+ r->coalesce_buf_cur++;
+ r->coalesce_cnt++;
+ r->coalesce_avail--;
+}
+
+static void coalesce_buf_insertb(request_rec *r, ap_bucket_brigade *bb,
+ const void *buf, int len)
+{
+ ap_bucket *b;
+
+ if (len < RMIN_BUCKET_SIZE) {
+ if (len > r->coalesce_avail) {
+ bb = ap_brigade_create(r->pool);
+ coalesce_buf_create_bucket(r, bb);
+ }
+ memcpy(r->coalesce_buf_cur, buf, len);
+ r->coalesce_cnt += len;
+ r->coalesce_buf_cur += len;
+ r->coalesce_avail -= len;
+ }
+ else {
+ bb = ap_brigade_create(r->pool);
+ coalesce_buf_create_bucket(r, bb);
+ b = ap_bucket_create_transient(buf, len);
+ AP_BRIGADE_INSERT_TAIL(bb, b);
+ }
+}
+
AP_DECLARE(int) ap_rputc(int c, request_rec *r)
{
ap_bucket_brigade *bb = NULL;
- ap_bucket *b;
char c2 = (char)c;
if (r->connection->aborted) {
return EOF;
}
+ if (r->coalesce_avail == 0) {
bb = ap_brigade_create(r->pool);
- b = ap_bucket_create_transient(&c2, 1);
- AP_BRIGADE_INSERT_TAIL(bb, b);
+ coalesce_buf_create_bucket(r, bb);
ap_pass_brigade(r->output_filters, bb);
+ }
+ coalesce_buf_insertc(r, c2);
return c;
}
AP_DECLARE(int) ap_rputs(const char *str, request_rec *r)
{
ap_bucket_brigade *bb = NULL;
- ap_bucket *b;
- apr_size_t len;
+ int len;
if (r->connection->aborted)
return EOF;
@@ -3025,9 +3071,9 @@
return 0;
len = strlen(str);
- bb = ap_brigade_create(r->pool);
- b = ap_bucket_create_transient(str, len);
- AP_BRIGADE_INSERT_TAIL(bb, b);
+
+ coalesce_buf_insertb(r, bb, str, len);
+ if (bb)
ap_pass_brigade(r->output_filters, bb);
return len;
@@ -3036,17 +3082,16 @@
AP_DECLARE(int) ap_rwrite(const void *buf, int nbyte, request_rec *r)
{
ap_bucket_brigade *bb = NULL;
- ap_bucket *b;
if (r->connection->aborted)
return EOF;
if (nbyte == 0)
return 0;
- bb = ap_brigade_create(r->pool);
- b = ap_bucket_create_transient(buf, nbyte);
- AP_BRIGADE_INSERT_TAIL(bb, b);
+ coalesce_buf_insertb(r, bb, buf, nbyte);
+ if (bb)
ap_pass_brigade(r->output_filters, bb);
+
return nbyte;
}
@@ -3054,13 +3099,18 @@
{
ap_bucket_brigade *bb = NULL;
apr_size_t written;
+ int send_it = 0;
if (r->connection->aborted)
return EOF;
bb = ap_brigade_create(r->pool);
+ if (r->coalesce_cnt > 0) {
+ coalesce_buf_create_bucket(r, bb);
+ send_it = 1;
+ }
written = ap_brigade_vprintf(bb, fmt, va);
- if (written != 0)
+ if (written != 0 || send_it != 0)
ap_pass_brigade(r->output_filters, bb);
return written;
}
@@ -3084,28 +3134,43 @@
}
AP_DECLARE_NONSTD(int) ap_rvputs(request_rec *r, ...)
-{
+ {
ap_bucket_brigade *bb = NULL;
apr_size_t written;
+ const char *str;
+ int len;
va_list va;
if (r->connection->aborted)
return EOF;
- bb = ap_brigade_create(r->pool);
+
va_start(va, r);
- written = ap_brigade_vputstrs(bb, va);
+ for (written = 0;;) {
+ str = va_arg(va, const char *);
+ if (str == NULL)
+ break;
+ len = strlen(str);
+ coalesce_buf_insertb(r, bb, str, len);
+ written += len;
+ }
va_end(va);
- if (written != 0)
+ if (written != 0 && bb)
ap_pass_brigade(r->output_filters, bb);
return written;
}
AP_DECLARE(int) ap_rflush(request_rec *r)
{
- /* we should be using a flush bucket to flush the stack, not buff code. */
ap_bucket_brigade *bb;
ap_bucket *b;
+ if (r->coalesce_cnt > 0) {
+ bb = ap_brigade_create(r->pool);
+ coalesce_buf_create_bucket(r, bb);
+ ap_pass_brigade(r->output_filters, bb);
+ }
+
+ /* we should be using a flush bucket to flush the stack, not buff code. */
bb = ap_brigade_create(r->pool);
b = ap_bucket_create_flush();
AP_BRIGADE_INSERT_TAIL(bb, b);
Victor
--
Victor J. Orlikowski
======================
v.j.orlikowski@gte.net
vjo@raleigh.ibm.com
vjo@us.ibm.com
Re: woah, "GET /" with autoindex
Posted by Greg Ames <gr...@raleigh.ibm.com>.
rbb@covalent.net wrote:
>
> > > > since the underlying ap_bucket_putstr and ap_bucket_printf are implemented
> > > > as one-copy, coalesce will result in a two-copy implementation... which is
> > > > less than 1.3's one-copy implementation.
> > > >
> > > > not to mention that the current ap_bucket_{putstr,printf} result in far
> > > > too many ap_bucket_ts being allocated.
[....]
> They are close, but they aren't exactly the same. If module is using the
> ap_r* functions, then we can assume that it isn't using the buckets
> directly, and we can buffer.
Didn't Victor post a patch not too long ago that did exactly this? IIRC
it was vetoed. Victor, if I'm not imagining things, could you please
re-post to save everybody some time?
> If a module is using the buckets directly,
> then where do we put the buffer?
What buffer? I thought we were talking about lots of itty bitty data,
like dynamic html.
Assuming we fix ap_r*, are there any modules out there now that generate
tons of itty bitty data directly to buckets, or any modules likely to do
it?
(I would think you would go directly to buckets with big data - files,
mmaps, probably pipes - where you typically wouldn't use a buffer above
the kernel.)
If I'm correct, what would a new buffer in the buckets path buy us?
Rasmus, which API do you see mod_php using? Does PHP buffer internally?
Greg
Re: woah, "GET /" with autoindex
Posted by rb...@covalent.net.
> > > since the underlying ap_bucket_putstr and ap_bucket_printf are implemented
> > > as one-copy, coalesce will result in a two-copy implementation... which is
> > > less than 1.3's one-copy implementation.
> > >
> > > not to mention that the current ap_bucket_{putstr,printf} result in far
> > > too many ap_bucket_ts being allocated.
> >
> > But like I said, those functions aren't being used in this case.
>
> ap_rprintf is essentially an ap_bucket_printf. so it is being used.
>
> ap_rputs does essentially what ap_bucket_putstr does... unless i'm totally
> misreading the code. ap_rputs creates a transient bucket which is a
> one-copy. ap_bucket_putstr creates a heap bucket... with copy.
They are close, but they aren't exactly the same. If module is using the
ap_r* functions, then we can assume that it isn't using the buckets
directly, and we can buffer. If a module is using the buckets directly,
then where do we put the buffer?
Ryan
_______________________________________________________________________________
Ryan Bloom rbb@apache.org
406 29th St.
San Francisco, CA 94131
-------------------------------------------------------------------------------
Re: woah, "GET /" with autoindex
Posted by dean gaudet <dg...@arctic.org>.
On Sun, 7 Jan 2001 rbb@covalent.net wrote:
>
> > since the underlying ap_bucket_putstr and ap_bucket_printf are implemented
> > as one-copy, coalesce will result in a two-copy implementation... which is
> > less than 1.3's one-copy implementation.
> >
> > not to mention that the current ap_bucket_{putstr,printf} result in far
> > too many ap_bucket_ts being allocated.
>
> But like I said, those functions aren't being used in this case.
ap_rprintf is essentially an ap_bucket_printf. so it is being used.
ap_rputs does essentially what ap_bucket_putstr does... unless i'm totally
misreading the code. ap_rputs creates a transient bucket which is a
one-copy. ap_bucket_putstr creates a heap bucket... with copy.
-dean
Re: woah, "GET /" with autoindex
Posted by rb...@covalent.net.
> since the underlying ap_bucket_putstr and ap_bucket_printf are implemented
> as one-copy, coalesce will result in a two-copy implementation... which is
> less than 1.3's one-copy implementation.
>
> not to mention that the current ap_bucket_{putstr,printf} result in far
> too many ap_bucket_ts being allocated.
But like I said, those functions aren't being used in this case.
> p.s. what's the syntax for turning on a filter? or should i rtfm?
ap_add_output_filter(FILTER_NAME, NULL, r, c)
in the code someplace
OR
AddOutOutFilter COALESCE
In the config file.
Ryan
>
> On Sun, 7 Jan 2001 rbb@covalent.net wrote:
>
> >
> > > 1.3 would generate three 1448 byte packets plus one straggler.
> > >
> > > on linux you could CORK in 2.0 and unCORK at the end and it'd generate the
> > > right packets... but that's not portable. so the fix needs to live
> > > somewhere in the i/o layering... which i'm not familiar enough with yet.
> >
> > Try looking for the COALESCE filter, and turning that on. This should fix
> > the sub-optimal packets.
> >
> > Ryan
> >
> > _______________________________________________________________________________
> > Ryan Bloom rbb@apache.org
> > 406 29th St.
> > San Francisco, CA 94131
> > -------------------------------------------------------------------------------
> >
> >
>
>
_______________________________________________________________________________
Ryan Bloom rbb@apache.org
406 29th St.
San Francisco, CA 94131
-------------------------------------------------------------------------------
Re: woah, "GET /" with autoindex
Posted by dean gaudet <dg...@arctic.org>.
since the underlying ap_bucket_putstr and ap_bucket_printf are implemented
as one-copy, coalesce will result in a two-copy implementation... which is
less than 1.3's one-copy implementation.
not to mention that the current ap_bucket_{putstr,printf} result in far
too many ap_bucket_ts being allocated.
coalesce sounds like the wrong fix.
-dean
p.s. what's the syntax for turning on a filter? or should i rtfm?
On Sun, 7 Jan 2001 rbb@covalent.net wrote:
>
> > 1.3 would generate three 1448 byte packets plus one straggler.
> >
> > on linux you could CORK in 2.0 and unCORK at the end and it'd generate the
> > right packets... but that's not portable. so the fix needs to live
> > somewhere in the i/o layering... which i'm not familiar enough with yet.
>
> Try looking for the COALESCE filter, and turning that on. This should fix
> the sub-optimal packets.
>
> Ryan
>
> _______________________________________________________________________________
> Ryan Bloom rbb@apache.org
> 406 29th St.
> San Francisco, CA 94131
> -------------------------------------------------------------------------------
>
>
Re: woah, "GET /" with autoindex
Posted by rb...@covalent.net.
> 1.3 would generate three 1448 byte packets plus one straggler.
>
> on linux you could CORK in 2.0 and unCORK at the end and it'd generate the
> right packets... but that's not portable. so the fix needs to live
> somewhere in the i/o layering... which i'm not familiar enough with yet.
Try looking for the COALESCE filter, and turning that on. This should fix
the sub-optimal packets.
Ryan
_______________________________________________________________________________
Ryan Bloom rbb@apache.org
406 29th St.
San Francisco, CA 94131
-------------------------------------------------------------------------------
Re: woah, "GET /" with autoindex
Posted by dean gaudet <dg...@arctic.org>.
On Sun, 7 Jan 2001, dean gaudet wrote:
> woah. ok there's some serious problems here.
>
> this is from a "GET / HTTP/1.0" which results in an autoindex (dunno why
> it's not grabbing one of the index.htmls, but i'm not worrying about that,
> the autoindex is probably far more interesting).
btw, the tcpdump for this shows suboptimal packets as well. this is
because without nagle, we've told the kernel that each of our writev()s is
to be put on the network.
11:53:12.839439 eth0 < client.1131 > server.8888: S 1366931848:1366931848(0) win 32120 <mss 1460,sackOK,timestamp 983960148 0,nop,wscale 0> (DF)
11:53:12.839531 eth0 > server.8888 > client.1131: S 1369163235:1369163235(0) ack 1366931849 win 32120 <mss 1460,sackOK,timestamp 769294182 983960148,nop,wscale 0> (DF)
11:53:12.856197 eth0 < client.1131 > server.8888: . 1:1(0) ack 1 win 32120 <nop,nop,timestamp 983960150 769294182> (DF)
11:53:14.969574 eth0 < client.1131 > server.8888: P 1:17(16) ack 1 win 32120 <nop,nop,timestamp 983960361 769294182> (DF)
11:53:14.969626 eth0 > server.8888 > client.1131: . 1:1(0) ack 17 win 32120 <nop,nop,timestamp 769294395 983960361> (DF)
11:53:15.109544 eth0 < client.1131 > server.8888: P 17:19(2) ack 1 win 32120 <nop,nop,timestamp 983960375 769294395> (DF)
11:53:15.119937 eth0 > server.8888 > client.1131: . 1:1(0) ack 19 win 32120 <nop,nop,timestamp 769294411 983960375> (DF)
11:53:15.139046 eth0 > server.8888 > client.1131: P 1:449(448) ack 19 win 32120 <nop,nop,timestamp 769294412 983960375> (DF)
11:53:15.139164 eth0 > server.8888 > client.1131: P 449:541(92) ack 19 win 32120 <nop,nop,timestamp 769294412 983960375> (DF)
11:53:15.167745 eth0 < client.1131 > server.8888: . 19:19(0) ack 449 win 32120 <nop,nop,timestamp 983960381 769294412> (DF)
11:53:15.167806 eth0 > server.8888 > client.1131: P 541:1989(1448) ack 19 win 32120 <nop,nop,timestamp 769294415 983960381> (DF)
11:53:15.167821 eth0 > server.8888 > client.1131: P 1989:3437(1448) ack 19 win 32120 <nop,nop,timestamp 769294415 983960381> (DF)
11:53:15.174817 eth0 < client.1131 > server.8888: . 19:19(0) ack 541 win 32120 <nop,nop,timestamp 983960382 769294412> (DF)
11:53:15.174882 eth0 > server.8888 > client.1131: P 3437:4885(1448) ack 19 win 32120 <nop,nop,timestamp 769294416 983960382> (DF)
11:53:15.174904 eth0 > server.8888 > client.1131: FP 4885:5072(187) ack 19 win 32120 <nop,nop,timestamp 769294416 983960382> (DF)
11:53:15.207674 eth0 < client.1131 > server.8888: . 19:19(0) ack 3437 win 31856 <nop,nop,timestamp 983960385 769294415> (DF)
11:53:15.227760 eth0 < client.1131 > server.8888: . 19:19(0) ack 5073 win 30408 <nop,nop,timestamp 983960387 769294416> (DF)
11:53:15.238239 eth0 < client.1131 > server.8888: F 19:19(0) ack 5073 win 31856 <nop,nop,timestamp 983960388 769294416> (DF)
11:53:15.238296 eth0 > server.8888 > client.1131: . 5073:5073(0) ack 20 win 32120 <nop,nop,timestamp 769294422 983960388> (DF)
1.3 would generate three 1448 byte packets plus one straggler.
on linux you could CORK in 2.0 and unCORK at the end and it'd generate the
right packets... but that's not portable. so the fix needs to live
somewhere in the i/o layering... which i'm not familiar enough with yet.
-dean