You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@couchdb.apache.org by daniel williams <da...@gmail.com> on 2011/11/02 22:16:38 UTC
couchdb .NET 4.0 client
Hi,
I posted an question the other day in regard to latency leveraging the
Redbranch Hammock library on windows. The codebase leverage the .NET 4.0
HttpWebRequest class which seems to have some serious issues in it's
communication with CouchDB. I've noticed in a tcpdump on Mac with code
compiled in mono that a PUT/Save takes roughly 1~4ms total. The same code
run on Mono or .NET takes about 200 ms and in analyzing the tcpdump on
windows (through mono or .NET making the execution) that after the Http
Continue 100 is received the .NET/Mono code on Windows 64 bit will wait
200ms roughly before sending the ACK to receive the response payload. See
below, I've changed the machine names:
10:03:04.552113 IP machine > server:5984: P 1:190(189) ack 1 win 256
E...X.@.......oX..o....`.eQ.Ug.bP...7\..PUT
/test/TestEntity-8cdddd58-1860-429e-90
10:03:04.554113 IP server:5984 > machine: P 1:26(25) ack 190 win 65346
E..AI.@...zC..o...oX.`..Ug.b.eR.P..B....HTTP/1.1 100 Continue
10:03:04.554220 IP machine > server:5984: P 190:602(412) ack 26 win 256
E...X.@
.......oX..o....`.eR.Ug.{P...8;..{"Foo1":"foo","Foo2":"foo","Foo3":"foo","F
10:03:04.558000 IP server:5984 > machine: P 26:351(325) ack 602 win 64934
E..mI.@...y...o...oX.`..Ug.{.eTDP.......HTTP/1.1 201 Created
Server: CouchDB/1.1.
10:03:04.751642 IP machine > server:5984: . ack 351 win 255
E..(X.@.......oX..o....`.eTDUg..P...6...
10:03:04.752131 IP server:5984 > machine: P 351:461(110) ack 602 win 64934
E...I.@
...y...o...oX.`..Ug...eTDP....Y..{"ok":true,"id":"TestEntity-8cdddd58-1860-
In doing a test in isolation on Windows 64 bit and .NET 4.0 I wrote a
TcpClient implementation that meets HTTP standards and I was able to PUT
4000 records successfully in less than 40 secs. Does anyone out there know
of a more efficient library that can be used to communicate with the
CouchDB API on Windows 64 bit servers running .NET 4.0? Looking at the
results of my tests run it would seem the most efficient to do straight
socket communication.
Thanks.
dan
Re: couchdb .NET 4.0 client
Posted by Dale Ragan <da...@sinesignal.com>.
I started a library called Ottoman last year and ran into issues with
WebRequest and REST in general, also limitations with System.Uri. I
decided to take a step back and do some research and my conclusion was
to write my own HttpClient using straight sockets that's a little more
REST friendly and without the System.Uri limitations. I'm not working
on it full time, but just something to pass the time when I feel bored.
I hope to release an alpha build here soon and then work it into Ottoman
and continue finishing it.
Dale
Re: couchdb .NET 4.0 client
Posted by daniel williams <da...@gmail.com>.
Hi everyone,
To wrap up my initial query to the group. After doing some more research
with my coworkers we were able to discover some additional obscure
information about the windows tcp stack. I'll recap
- our initial tests took code and ran it in .NET where the tcpdump
showed the delay on the ack
- we took the same code and compiled it in mono and ran it on mac and
ran tcpdump and did not see the issue
- we ran the same .exe that we ran on mac's version of mono and ran it
through windows version of mono which showed the same delay. This makes
sense as there is still an OS call to get the tcp socket for the
connectivity. This was run against many variations of windows (server 2003
32 bit, windows 7 64 bit, 2008 64 bit) all displaying the same results
roughly. The ack after the 201 Created had a delay of roughly 200 ms.
We did the majority of our searches with queries similar to "windows
httpwebresponse getresponse slow". A shot in the dark my boss searched
"tcp no delay windwows" and came across the following url:
http://lifeandcode.net/2009/05/reduce-game-network-latency-in-windows-7-or-vista/
After following the instructions we ran the same test and we were able to
product the following tcpdump with a vast improvement in speed:
4:12:46.*070*172 IP client.49761 > server.5984: P 20797:20962(165) ack
16561 win 63780
.....@.......oX..o..a.`|.....j.P..$7D..PUT
/test/testentity-f5719971-374c-4b61-93
4:12:46.071512 IP server.5984 > client.49761: P 16561:16586(25) ack 20962
win 65370
..A..@.......o...oX.`.a..j.|...P..Z;...HTTP/1.1 100 Continue
4:12:46.071545 IP client.49761 > server.5984: . ack 16586 win 63755
..(..@.......oX..o..a.`|.....j.P...6...
4:12:46.072854 IP client.49761 > server.5984: P 20962:21374(412) ack 16586
win 63755
.....@.......oX..o..a.`|.....j.P...8;..{"Foo1":"foo","Foo2":"foo","Foo3":"foo","F
4:12:46.077300 IP server.5984 > client.49761: P 16586:16911(325) ack 21374
win 64958
..m..@....v..o...oX.`.a..j.|..?P...>Q..HTTP/1.1 201 Created
erver: CouchDB/1.1.
4:12:46.077355 IP client.49761 > server.5984: . ack 16911 win 63430
..(..@.......oX..o..a.`|..?..lBP...6...
4:12:46.*077*687 IP server.5984 > client.49761: P 16911:17021(110) ack
21374 win 64958
.....@....L..o...oX.`.a..lB|..?P....c..{"ok":true,"id":"testentity-f5719971-374c-
With that said, this good be greatly beneficial to couchdb replication
schemes as well. We've already seen marked improvement in couch processing
over http in testing scenarios.
Thanks.
dan
On Thu, Nov 3, 2011 at 10:56 AM, Dennis Redfield <
dennis.lee.redfield@gmail.com> wrote:
> wow. I am going to work on this myself when time allows, very interesting.
>
> On Thursday, November 3, 2011, daniel williams <da...@gmail.com>
> wrote:
> > Oddly enough that is how I'm doing it. Initially I thought that the
> > problem was with the Hammock library and began digging through the code
> and
> > noticed that they were creating the HttpWebRequest via the
> > WebRequest.Create. In noticing this I thought there could be some
> > additional caching overhead in the Hammock library. Pulling out exactly
> as
> > you have suggested produced the tcpdump I provided. It is the Windows 64
> > bit / .NET 4.0 platform. I ran the same tcpdump with Hammock and it has
> > produced the same results. When I wrote the raw HTTP stream to a socket
> I
> > was able to do 4000 PUTs in 40s to a single instance of couchdb with the
> > 100-continue expect in line.
> >
> > I may simply rewrite GetById and Save to use a Socket instead of
> WebRequest
> > for the time being unless someone on this list knows of a better
> strategy.
> >
> > Thanks.
> >
> > dan
> >
> > On Thu, Nov 3, 2011 at 6:34 AM, Dennis Redfield <
> > dennis.lee.redfield@gmail.com> wrote:
> >
> >> I am running on Windows XP 32 bit and am making my own calls to couchdb
> >> using Framework 4.0 and they are very fast. Nothing as slow as the
> number
> >> you are reporting. I am thinking it problem is with the library not the
> >> underlying system - Why not write your own calls like:
> >>
> >> HttpWebRequest myRequest =( HttpWebRequest ) WebRequest.Create(
> requestURL
> >> );
> >> myRequest.MaximumAutomaticRedirections = _MaxRedirect;
> >> myRequest.MaximumResponseHeadersLength = _MaxHeadersLen;
> >> response = ( HttpWebResponse ) myRequest.GetResponse( );
> >> Stream receiveStream=response.GetResponseStream( );
> >> StreamReader readStream = new StreamReader( receiveStream,
> >> Encoding.UTF8 );//we shall see if this is right
> >> return readStream.ReadToEnd( );
> >>
> >> Hope this helps.
> >>
> >> On Wednesday, November 2, 2011, daniel williams <
> daniel.williams@gmail.com
> >> >
> >> wrote:
> >> > Hi,
> >> >
> >> > I posted an question the other day in regard to latency leveraging the
> >> > Redbranch Hammock library on windows. The codebase leverage the .NET
> 4.0
> >> > HttpWebRequest class which seems to have some serious issues in it's
> >> > communication with CouchDB. I've noticed in a tcpdump on Mac with
> code
> >> > compiled in mono that a PUT/Save takes roughly 1~4ms total. The same
> >> code
> >> > run on Mono or .NET takes about 200 ms and in analyzing the tcpdump on
> >> > windows (through mono or .NET making the execution) that after the
> Http
> >> > Continue 100 is received the .NET/Mono code on Windows 64 bit will
> wait
> >> > 200ms roughly before sending the ACK to receive the response payload.
> >> See
> >> > below, I've changed the machine names:
> >> >
> >> > 10:03:04.552113 IP machine > server:5984: P 1:190(189) ack 1 win 256
> >> > E...X.@.......oX..o....`.eQ.Ug.bP...7\..PUT
> >> > /test/TestEntity-8cdddd58-1860-429e-90
> >> > 10:03:04.554113 IP server:5984 > machine: P 1:26(25) ack 190 win 65346
> >> > E..AI.@...zC..o...oX.`..Ug.b.eR.P..B....HTTP/1.1 100 Continue
> >> >
> >> >
> >> > 10:03:04.554220 IP machine > server:5984: P 190:602(412) ack 26 win
> 256
> >> > E...X.@
> >> >
> >>
> .......oX..o....`.eR.Ug.{P...8;..{"Foo1":"foo","Foo2":"foo","Foo3":"foo","F
> >> > 10:03:04.558000 IP server:5984 > machine: P 26:351(325) ack 602 win
> 64934
> >> > E..mI.@...y...o...oX.`..Ug.{.eTDP.......HTTP/1.1 201 Created
> >> > Server: CouchDB/1.1.
> >> > 10:03:04.751642 IP machine > server:5984: . ack 351 win 255
> >> > E..(X.@.......oX..o....`.eTDUg..P...6...
> >> > 10:03:04.752131 IP server:5984 > machine: P 351:461(110) ack 602 win
> >> 64934
> >> > E...I.@
> >> >
> >>
> ...y...o...oX.`..Ug...eTDP....Y..{"ok":true,"id":"TestEntity-8cdddd58-1860-
> >> >
> >> > In doing a test in isolation on Windows 64 bit and .NET 4.0 I wrote a
> >> > TcpClient implementation that meets HTTP standards and I was able to
> PUT
> >> > 4000 records successfully in less than 40 secs. Does anyone out there
> >> know
> >> > of a more efficient library that can be used to communicate with the
> >> > CouchDB API on Windows 64 bit servers running .NET 4.0? Looking at
> the
> >> > results of my tests run it would seem the most efficient to do
> straight
> >> > socket communication.
> >> >
> >> > Thanks.
> >> >
> >> > dan
> >> >
> >>
> >> --
> >> Trust No One
> >>
> >
> >
> >
> > --
> > -dan
> >
>
> --
> Trust No One
>
--
-dan
Re: couchdb .NET 4.0 client
Posted by Dennis Redfield <de...@gmail.com>.
wow. I am going to work on this myself when time allows, very interesting.
On Thursday, November 3, 2011, daniel williams <da...@gmail.com>
wrote:
> Oddly enough that is how I'm doing it. Initially I thought that the
> problem was with the Hammock library and began digging through the code
and
> noticed that they were creating the HttpWebRequest via the
> WebRequest.Create. In noticing this I thought there could be some
> additional caching overhead in the Hammock library. Pulling out exactly
as
> you have suggested produced the tcpdump I provided. It is the Windows 64
> bit / .NET 4.0 platform. I ran the same tcpdump with Hammock and it has
> produced the same results. When I wrote the raw HTTP stream to a socket I
> was able to do 4000 PUTs in 40s to a single instance of couchdb with the
> 100-continue expect in line.
>
> I may simply rewrite GetById and Save to use a Socket instead of
WebRequest
> for the time being unless someone on this list knows of a better strategy.
>
> Thanks.
>
> dan
>
> On Thu, Nov 3, 2011 at 6:34 AM, Dennis Redfield <
> dennis.lee.redfield@gmail.com> wrote:
>
>> I am running on Windows XP 32 bit and am making my own calls to couchdb
>> using Framework 4.0 and they are very fast. Nothing as slow as the
number
>> you are reporting. I am thinking it problem is with the library not the
>> underlying system - Why not write your own calls like:
>>
>> HttpWebRequest myRequest =( HttpWebRequest ) WebRequest.Create(
requestURL
>> );
>> myRequest.MaximumAutomaticRedirections = _MaxRedirect;
>> myRequest.MaximumResponseHeadersLength = _MaxHeadersLen;
>> response = ( HttpWebResponse ) myRequest.GetResponse( );
>> Stream receiveStream=response.GetResponseStream( );
>> StreamReader readStream = new StreamReader( receiveStream,
>> Encoding.UTF8 );//we shall see if this is right
>> return readStream.ReadToEnd( );
>>
>> Hope this helps.
>>
>> On Wednesday, November 2, 2011, daniel williams <
daniel.williams@gmail.com
>> >
>> wrote:
>> > Hi,
>> >
>> > I posted an question the other day in regard to latency leveraging the
>> > Redbranch Hammock library on windows. The codebase leverage the .NET
4.0
>> > HttpWebRequest class which seems to have some serious issues in it's
>> > communication with CouchDB. I've noticed in a tcpdump on Mac with code
>> > compiled in mono that a PUT/Save takes roughly 1~4ms total. The same
>> code
>> > run on Mono or .NET takes about 200 ms and in analyzing the tcpdump on
>> > windows (through mono or .NET making the execution) that after the Http
>> > Continue 100 is received the .NET/Mono code on Windows 64 bit will wait
>> > 200ms roughly before sending the ACK to receive the response payload.
>> See
>> > below, I've changed the machine names:
>> >
>> > 10:03:04.552113 IP machine > server:5984: P 1:190(189) ack 1 win 256
>> > E...X.@.......oX..o....`.eQ.Ug.bP...7\..PUT
>> > /test/TestEntity-8cdddd58-1860-429e-90
>> > 10:03:04.554113 IP server:5984 > machine: P 1:26(25) ack 190 win 65346
>> > E..AI.@...zC..o...oX.`..Ug.b.eR.P..B....HTTP/1.1 100 Continue
>> >
>> >
>> > 10:03:04.554220 IP machine > server:5984: P 190:602(412) ack 26 win 256
>> > E...X.@
>> >
>>
.......oX..o....`.eR.Ug.{P...8;..{"Foo1":"foo","Foo2":"foo","Foo3":"foo","F
>> > 10:03:04.558000 IP server:5984 > machine: P 26:351(325) ack 602 win
64934
>> > E..mI.@...y...o...oX.`..Ug.{.eTDP.......HTTP/1.1 201 Created
>> > Server: CouchDB/1.1.
>> > 10:03:04.751642 IP machine > server:5984: . ack 351 win 255
>> > E..(X.@.......oX..o....`.eTDUg..P...6...
>> > 10:03:04.752131 IP server:5984 > machine: P 351:461(110) ack 602 win
>> 64934
>> > E...I.@
>> >
>>
...y...o...oX.`..Ug...eTDP....Y..{"ok":true,"id":"TestEntity-8cdddd58-1860-
>> >
>> > In doing a test in isolation on Windows 64 bit and .NET 4.0 I wrote a
>> > TcpClient implementation that meets HTTP standards and I was able to
PUT
>> > 4000 records successfully in less than 40 secs. Does anyone out there
>> know
>> > of a more efficient library that can be used to communicate with the
>> > CouchDB API on Windows 64 bit servers running .NET 4.0? Looking at the
>> > results of my tests run it would seem the most efficient to do straight
>> > socket communication.
>> >
>> > Thanks.
>> >
>> > dan
>> >
>>
>> --
>> Trust No One
>>
>
>
>
> --
> -dan
>
--
Trust No One
Re: couchdb .NET 4.0 client
Posted by daniel williams <da...@gmail.com>.
Oddly enough that is how I'm doing it. Initially I thought that the
problem was with the Hammock library and began digging through the code and
noticed that they were creating the HttpWebRequest via the
WebRequest.Create. In noticing this I thought there could be some
additional caching overhead in the Hammock library. Pulling out exactly as
you have suggested produced the tcpdump I provided. It is the Windows 64
bit / .NET 4.0 platform. I ran the same tcpdump with Hammock and it has
produced the same results. When I wrote the raw HTTP stream to a socket I
was able to do 4000 PUTs in 40s to a single instance of couchdb with the
100-continue expect in line.
I may simply rewrite GetById and Save to use a Socket instead of WebRequest
for the time being unless someone on this list knows of a better strategy.
Thanks.
dan
On Thu, Nov 3, 2011 at 6:34 AM, Dennis Redfield <
dennis.lee.redfield@gmail.com> wrote:
> I am running on Windows XP 32 bit and am making my own calls to couchdb
> using Framework 4.0 and they are very fast. Nothing as slow as the number
> you are reporting. I am thinking it problem is with the library not the
> underlying system - Why not write your own calls like:
>
> HttpWebRequest myRequest =( HttpWebRequest ) WebRequest.Create( requestURL
> );
> myRequest.MaximumAutomaticRedirections = _MaxRedirect;
> myRequest.MaximumResponseHeadersLength = _MaxHeadersLen;
> response = ( HttpWebResponse ) myRequest.GetResponse( );
> Stream receiveStream=response.GetResponseStream( );
> StreamReader readStream = new StreamReader( receiveStream,
> Encoding.UTF8 );//we shall see if this is right
> return readStream.ReadToEnd( );
>
> Hope this helps.
>
> On Wednesday, November 2, 2011, daniel williams <daniel.williams@gmail.com
> >
> wrote:
> > Hi,
> >
> > I posted an question the other day in regard to latency leveraging the
> > Redbranch Hammock library on windows. The codebase leverage the .NET 4.0
> > HttpWebRequest class which seems to have some serious issues in it's
> > communication with CouchDB. I've noticed in a tcpdump on Mac with code
> > compiled in mono that a PUT/Save takes roughly 1~4ms total. The same
> code
> > run on Mono or .NET takes about 200 ms and in analyzing the tcpdump on
> > windows (through mono or .NET making the execution) that after the Http
> > Continue 100 is received the .NET/Mono code on Windows 64 bit will wait
> > 200ms roughly before sending the ACK to receive the response payload.
> See
> > below, I've changed the machine names:
> >
> > 10:03:04.552113 IP machine > server:5984: P 1:190(189) ack 1 win 256
> > E...X.@.......oX..o....`.eQ.Ug.bP...7\..PUT
> > /test/TestEntity-8cdddd58-1860-429e-90
> > 10:03:04.554113 IP server:5984 > machine: P 1:26(25) ack 190 win 65346
> > E..AI.@...zC..o...oX.`..Ug.b.eR.P..B....HTTP/1.1 100 Continue
> >
> >
> > 10:03:04.554220 IP machine > server:5984: P 190:602(412) ack 26 win 256
> > E...X.@
> >
> .......oX..o....`.eR.Ug.{P...8;..{"Foo1":"foo","Foo2":"foo","Foo3":"foo","F
> > 10:03:04.558000 IP server:5984 > machine: P 26:351(325) ack 602 win 64934
> > E..mI.@...y...o...oX.`..Ug.{.eTDP.......HTTP/1.1 201 Created
> > Server: CouchDB/1.1.
> > 10:03:04.751642 IP machine > server:5984: . ack 351 win 255
> > E..(X.@.......oX..o....`.eTDUg..P...6...
> > 10:03:04.752131 IP server:5984 > machine: P 351:461(110) ack 602 win
> 64934
> > E...I.@
> >
> ...y...o...oX.`..Ug...eTDP....Y..{"ok":true,"id":"TestEntity-8cdddd58-1860-
> >
> > In doing a test in isolation on Windows 64 bit and .NET 4.0 I wrote a
> > TcpClient implementation that meets HTTP standards and I was able to PUT
> > 4000 records successfully in less than 40 secs. Does anyone out there
> know
> > of a more efficient library that can be used to communicate with the
> > CouchDB API on Windows 64 bit servers running .NET 4.0? Looking at the
> > results of my tests run it would seem the most efficient to do straight
> > socket communication.
> >
> > Thanks.
> >
> > dan
> >
>
> --
> Trust No One
>
--
-dan
Re: couchdb .NET 4.0 client
Posted by Dennis Redfield <de...@gmail.com>.
I am running on Windows XP 32 bit and am making my own calls to couchdb
using Framework 4.0 and they are very fast. Nothing as slow as the number
you are reporting. I am thinking it problem is with the library not the
underlying system - Why not write your own calls like:
HttpWebRequest myRequest =( HttpWebRequest ) WebRequest.Create( requestURL
);
myRequest.MaximumAutomaticRedirections = _MaxRedirect;
myRequest.MaximumResponseHeadersLength = _MaxHeadersLen;
response = ( HttpWebResponse ) myRequest.GetResponse( );
Stream receiveStream=response.GetResponseStream( );
StreamReader readStream = new StreamReader( receiveStream,
Encoding.UTF8 );//we shall see if this is right
return readStream.ReadToEnd( );
Hope this helps.
On Wednesday, November 2, 2011, daniel williams <da...@gmail.com>
wrote:
> Hi,
>
> I posted an question the other day in regard to latency leveraging the
> Redbranch Hammock library on windows. The codebase leverage the .NET 4.0
> HttpWebRequest class which seems to have some serious issues in it's
> communication with CouchDB. I've noticed in a tcpdump on Mac with code
> compiled in mono that a PUT/Save takes roughly 1~4ms total. The same code
> run on Mono or .NET takes about 200 ms and in analyzing the tcpdump on
> windows (through mono or .NET making the execution) that after the Http
> Continue 100 is received the .NET/Mono code on Windows 64 bit will wait
> 200ms roughly before sending the ACK to receive the response payload. See
> below, I've changed the machine names:
>
> 10:03:04.552113 IP machine > server:5984: P 1:190(189) ack 1 win 256
> E...X.@.......oX..o....`.eQ.Ug.bP...7\..PUT
> /test/TestEntity-8cdddd58-1860-429e-90
> 10:03:04.554113 IP server:5984 > machine: P 1:26(25) ack 190 win 65346
> E..AI.@...zC..o...oX.`..Ug.b.eR.P..B....HTTP/1.1 100 Continue
>
>
> 10:03:04.554220 IP machine > server:5984: P 190:602(412) ack 26 win 256
> E...X.@
>
.......oX..o....`.eR.Ug.{P...8;..{"Foo1":"foo","Foo2":"foo","Foo3":"foo","F
> 10:03:04.558000 IP server:5984 > machine: P 26:351(325) ack 602 win 64934
> E..mI.@...y...o...oX.`..Ug.{.eTDP.......HTTP/1.1 201 Created
> Server: CouchDB/1.1.
> 10:03:04.751642 IP machine > server:5984: . ack 351 win 255
> E..(X.@.......oX..o....`.eTDUg..P...6...
> 10:03:04.752131 IP server:5984 > machine: P 351:461(110) ack 602 win 64934
> E...I.@
>
...y...o...oX.`..Ug...eTDP....Y..{"ok":true,"id":"TestEntity-8cdddd58-1860-
>
> In doing a test in isolation on Windows 64 bit and .NET 4.0 I wrote a
> TcpClient implementation that meets HTTP standards and I was able to PUT
> 4000 records successfully in less than 40 secs. Does anyone out there
know
> of a more efficient library that can be used to communicate with the
> CouchDB API on Windows 64 bit servers running .NET 4.0? Looking at the
> results of my tests run it would seem the most efficient to do straight
> socket communication.
>
> Thanks.
>
> dan
>
--
Trust No One