You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@couchdb.apache.org by Cliffano Subagio <cl...@gmail.com> on 2013/06/18 02:12:01 UTC
large replication URL
Hi,
I have a CouchDB database which replicates from npm registry and I notice
that the replication URL can get large due to the revisions array in
atts_since parameter, to a point that its too large that the proxy I'm
using doesn't let the GET request through.
Here's a sample of the large URL
https://gist.github.com/cliffano/5726639/raw/ea0e416dbbd8928ade4809c38e5a3214a736c35c/gistfile1.txt
Is there any particular reason why this is a GET instead of a POST?
Is there any way to change the request to use POST?
Cheers,
Cliff.
Re: large replication URL
Posted by Cliffano Subagio <cl...@gmail.com>.
Hi Jens,
Specific version: Blue Coat SG510 Series, Version: SGOS 6.2.10.3, Release
id: 90684 Proxy Edition
HTH.
On Tue, Jun 18, 2013 at 6:00 PM, Cliffano Subagio <cl...@gmail.com>wrote:
> Hi Jens,
>
> It's an older Bluecoat SG.
>
> Btw, I found this line on RFC 2616 re proxy and URI length support (party
> like it's 1999 :p)
>
> Note: Servers ought to be cautious about depending on URI lengths
> above 255 bytes, because some older client or proxy
> implementations might not properly support these lengths.
>
>
>
> On Tue, Jun 18, 2013 at 12:13 PM, Jens Alfke <je...@couchbase.com> wrote:
>
>>
>> On Jun 17, 2013, at 5:12 PM, Cliffano Subagio <cl...@gmail.com> wrote:
>>
>> > I have a CouchDB database which replicates from npm registry and I
>> notice
>> > that the replication URL can get large due to the revisions array in
>> > atts_since parameter, to a point that its too large that the proxy I'm
>> > using doesn't let the GET request through.
>>
>> That URL is only 6273 bytes long; I wouldn’t expect a proxy to reject it.
>> Do you know what type of proxy server this is?
>>
>> (I have an interest in this as I’m the main author of a
>> CouchDB-compatible database, which has its own heuristics for how long to
>> let such a URL get.)
>>
>> —Jens
>
>
>
Re: large replication URL
Posted by Cliffano Subagio <cl...@gmail.com>.
Hi Jens,
It's an older Bluecoat SG.
Btw, I found this line on RFC 2616 re proxy and URI length support (party
like it's 1999 :p)
Note: Servers ought to be cautious about depending on URI lengths
above 255 bytes, because some older client or proxy
implementations might not properly support these lengths.
On Tue, Jun 18, 2013 at 12:13 PM, Jens Alfke <je...@couchbase.com> wrote:
>
> On Jun 17, 2013, at 5:12 PM, Cliffano Subagio <cl...@gmail.com> wrote:
>
> > I have a CouchDB database which replicates from npm registry and I notice
> > that the replication URL can get large due to the revisions array in
> > atts_since parameter, to a point that its too large that the proxy I'm
> > using doesn't let the GET request through.
>
> That URL is only 6273 bytes long; I wouldn’t expect a proxy to reject it.
> Do you know what type of proxy server this is?
>
> (I have an interest in this as I’m the main author of a CouchDB-compatible
> database, which has its own heuristics for how long to let such a URL get.)
>
> —Jens
Re: large replication URL
Posted by Jens Alfke <je...@couchbase.com>.
On Jun 17, 2013, at 5:12 PM, Cliffano Subagio <cl...@gmail.com> wrote:
> I have a CouchDB database which replicates from npm registry and I notice
> that the replication URL can get large due to the revisions array in
> atts_since parameter, to a point that its too large that the proxy I'm
> using doesn't let the GET request through.
That URL is only 6273 bytes long; I wouldn’t expect a proxy to reject it. Do you know what type of proxy server this is?
(I have an interest in this as I’m the main author of a CouchDB-compatible database, which has its own heuristics for how long to let such a URL get.)
—Jens
Re: large replication URL
Posted by Cliffano Subagio <cl...@gmail.com>.
Thanks for the explanation and the hint to MAX_URL_LEN, Adam.
We do build from source, will patch manually for now to get around this
issue.
On Tue, Jun 18, 2013 at 10:26 AM, Adam Kocoloski <ko...@apache.org>wrote:
> Well, a GET request is the RESTful choice. I do admit that the limit on
> the length of a URL is going to be much more stringent than the limit on
> the size of a POST body.
>
> CouchDB already has code to split the request up if the generated URL is
> "too large". Currently the value of "too large" is hard-coded at 7000
> bytes:
>
>
> https://github.com/apache/couchdb/blob/1.3.0/src/couch_replicator/src/couch_replicator_api_wrap.erl#L473
>
> If you happen to be building from source you can lower that value to
> something your proxy will accept.
>
> For dev@ -- it would absolutely make sense to patch this so that the user
> could supply a parameter like max_url_length in the replication spec and
> override the default. It may also make sense to turn the default into a
> config setting.
>
> Adam
>
> On Jun 17, 2013, at 8:12 PM, Cliffano Subagio <cl...@gmail.com> wrote:
>
> > Hi,
> >
> > I have a CouchDB database which replicates from npm registry and I notice
> > that the replication URL can get large due to the revisions array in
> > atts_since parameter, to a point that its too large that the proxy I'm
> > using doesn't let the GET request through.
> >
> > Here's a sample of the large URL
> >
> https://gist.github.com/cliffano/5726639/raw/ea0e416dbbd8928ade4809c38e5a3214a736c35c/gistfile1.txt
> >
> > Is there any particular reason why this is a GET instead of a POST?
> > Is there any way to change the request to use POST?
> >
> > Cheers,
> > Cliff.
>
>
Re: large replication URL
Posted by Adam Kocoloski <ko...@apache.org>.
Well, a GET request is the RESTful choice. I do admit that the limit on the length of a URL is going to be much more stringent than the limit on the size of a POST body.
CouchDB already has code to split the request up if the generated URL is "too large". Currently the value of "too large" is hard-coded at 7000 bytes:
https://github.com/apache/couchdb/blob/1.3.0/src/couch_replicator/src/couch_replicator_api_wrap.erl#L473
If you happen to be building from source you can lower that value to something your proxy will accept.
For dev@ -- it would absolutely make sense to patch this so that the user could supply a parameter like max_url_length in the replication spec and override the default. It may also make sense to turn the default into a config setting.
Adam
On Jun 17, 2013, at 8:12 PM, Cliffano Subagio <cl...@gmail.com> wrote:
> Hi,
>
> I have a CouchDB database which replicates from npm registry and I notice
> that the replication URL can get large due to the revisions array in
> atts_since parameter, to a point that its too large that the proxy I'm
> using doesn't let the GET request through.
>
> Here's a sample of the large URL
> https://gist.github.com/cliffano/5726639/raw/ea0e416dbbd8928ade4809c38e5a3214a736c35c/gistfile1.txt
>
> Is there any particular reason why this is a GET instead of a POST?
> Is there any way to change the request to use POST?
>
> Cheers,
> Cliff.