You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@subversion.apache.org by Justin Erenkrantz <ju...@erenkrantz.com> on 2008/10/02 16:16:41 UTC

Re: svn commit: r33398 - trunk/notes

On Thu, Oct 2, 2008 at 8:06 AM,  <su...@tigris.org> wrote:
> +  XML is out : there's a huge performance penalty for producing and
> +  consuming it, which is why companies like Facebook and Google have
> +  released 'fast wire serialization' libraries like Thrift and
> +  Protocol Buffers.  Unfortunately, these libraries require entire
> +  structures to be held in memory in order to serialize/deserialze
> +  them, and this isn't an option when dealing with something
> +  (potentially) infinitely large like an editor-drive.

Then perhaps we should look at improving Thrift parsing to be streamy.
 I'd rather use a library than a custom serialization format - as you
said, one of the goals should be used to make it simpler to
understand.   And, one of the ways to do that is to go "Learn Thrift"
rather than read our custom (de)serialization code.  =)

>  In the list of commands, all commands are assumed to be attached as
> -?cmd=command to GET requests on the repository root URI, with the
> -exception of {commit, ...}, which are attached to POST requests.
> +?cmd=command to the request URI.  Command parameters are all
> +query-encoded (&parm=val), and optional parameters are listed in
> +square brackets.  Server response values are assumed to be in response
> +bodies.

So, a question is how much we want to push the HTTP envelope.  But,
doing query parameters is kinda lame from a performance standpoint.

The diff kinda butchered it, but you had:
---
  get-latest-rev

    GET /repos[/path]?cmd=get-latest-rev

    response: revnum
---

I'd rather see one of the following:

---
GET-LATEST-REV /repos
response: revnum
---
or
---
GET /repos/!svn/get-latest-rev
response: revnum
---

(I sort of prefer the latter, but could be convinced the former works too.)

I think that args should be explicit in the URLs rather than then
query strings, so get-dated-rev could be:

---
GET /repos/!svn/get-dated-rev/<ISO date string>
response: revnum
---

Also:
---
  rev-proplist

    GET /repos[/path]?cmd=rev-proplist&rev=num

    response:  proplist
---

could be:

---
GET /repos/!svn/<revnum>/props/rev
response: proplist
---

etc, etc, etc.

My $.02.  -- justin

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org

Re: svn commit: r33398 - trunk/notes

Posted by Justin Erenkrantz <ju...@erenkrantz.com>.
On Thu, Oct 2, 2008 at 1:26 PM, Greg Stein <gs...@gmail.com> wrote:
> For a pure C version, I'd lay odds on that being possible in Thrift
> more than protobuf cuz of the contribution/inclusivity thing.

What he said.  =)  -- justin

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org

Re: svn commit: r33398 - trunk/notes

Posted by Greg Stein <gs...@gmail.com>.
On Thu, Oct 2, 2008 at 1:23 PM, Blair Zajac <bl...@orcaware.com> wrote:
> Mark Phippard wrote:
>>
>> On Thu, Oct 2, 2008 at 4:12 PM, Blair Zajac <bl...@orcaware.com> wrote:
>>>
>>> Justin Erenkrantz wrote:
>>>>
>>>> On Thu, Oct 2, 2008 at 8:06 AM,  <su...@tigris.org> wrote:
>>>>>
>>>>> +  XML is out : there's a huge performance penalty for producing and
>>>>> +  consuming it, which is why companies like Facebook and Google have
>>>>> +  released 'fast wire serialization' libraries like Thrift and
>>>>> +  Protocol Buffers.  Unfortunately, these libraries require entire
>>>>> +  structures to be held in memory in order to serialize/deserialze
>>>>> +  them, and this isn't an option when dealing with something
>>>>> +  (potentially) infinitely large like an editor-drive.
>>>>
>>>> Then perhaps we should look at improving Thrift parsing to be streamy.
>>>>  I'd rather use a library than a custom serialization format - as you
>>>> said, one of the goals should be used to make it simpler to
>>>> understand.   And, one of the ways to do that is to go "Learn Thrift"
>>>> rather than read our custom (de)serialization code.  =)
>>>
>>> Is there a reason why you want to make Thrift parsing be streamy instead
>>> of
>>> protobufs?
>>
>> Neither seems to particularly support C.
>>
>> I notice Thrift compiles in Cygwin and requires things like Boost.
>> These all seem line "non-starters" to me.
>
> There is a C-only protobuf port:
>
> http://code.google.com/p/protobuf-c/
>
> I haven't looked at it.
>
> BTW, I'm using protobuf in our C++ svn server already, so I'm a little
> biased in that direction :)

The Apache Thrift project is going to be more open to external patches
and development. While Google (as an entity) would love for protobufs
to get those kinds of changes, I'm familiar with the developer who
worked on the protobuf code, and he is NOT what I'd call an
"inclusive" Open Source developer.

Also, take a look at:
http://stuartsierra.com/2008/07/10/thrift-vs-protocol-buffers

On balance, Thrift is looking further developed, though with worse docco.

For a pure C version, I'd lay odds on that being possible in Thrift
more than protobuf cuz of the contribution/inclusivity thing.

Cheers,
-g

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org

Re: svn commit: r33398 - trunk/notes

Posted by Blair Zajac <bl...@orcaware.com>.
Mark Phippard wrote:
> On Thu, Oct 2, 2008 at 4:12 PM, Blair Zajac <bl...@orcaware.com> wrote:
>> Justin Erenkrantz wrote:
>>> On Thu, Oct 2, 2008 at 8:06 AM,  <su...@tigris.org> wrote:
>>>> +  XML is out : there's a huge performance penalty for producing and
>>>> +  consuming it, which is why companies like Facebook and Google have
>>>> +  released 'fast wire serialization' libraries like Thrift and
>>>> +  Protocol Buffers.  Unfortunately, these libraries require entire
>>>> +  structures to be held in memory in order to serialize/deserialze
>>>> +  them, and this isn't an option when dealing with something
>>>> +  (potentially) infinitely large like an editor-drive.
>>> Then perhaps we should look at improving Thrift parsing to be streamy.
>>>  I'd rather use a library than a custom serialization format - as you
>>> said, one of the goals should be used to make it simpler to
>>> understand.   And, one of the ways to do that is to go "Learn Thrift"
>>> rather than read our custom (de)serialization code.  =)
>> Is there a reason why you want to make Thrift parsing be streamy instead of
>> protobufs?
> 
> Neither seems to particularly support C.
> 
> I notice Thrift compiles in Cygwin and requires things like Boost.
> These all seem line "non-starters" to me.

There is a C-only protobuf port:

http://code.google.com/p/protobuf-c/

I haven't looked at it.

BTW, I'm using protobuf in our C++ svn server already, so I'm a little biased in 
that direction :)

Blair


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org

Re: svn commit: r33398 - trunk/notes

Posted by Mark Phippard <ma...@gmail.com>.
On Thu, Oct 2, 2008 at 4:12 PM, Blair Zajac <bl...@orcaware.com> wrote:
> Justin Erenkrantz wrote:
>>
>> On Thu, Oct 2, 2008 at 8:06 AM,  <su...@tigris.org> wrote:
>>>
>>> +  XML is out : there's a huge performance penalty for producing and
>>> +  consuming it, which is why companies like Facebook and Google have
>>> +  released 'fast wire serialization' libraries like Thrift and
>>> +  Protocol Buffers.  Unfortunately, these libraries require entire
>>> +  structures to be held in memory in order to serialize/deserialze
>>> +  them, and this isn't an option when dealing with something
>>> +  (potentially) infinitely large like an editor-drive.
>>
>> Then perhaps we should look at improving Thrift parsing to be streamy.
>>  I'd rather use a library than a custom serialization format - as you
>> said, one of the goals should be used to make it simpler to
>> understand.   And, one of the ways to do that is to go "Learn Thrift"
>> rather than read our custom (de)serialization code.  =)
>
> Is there a reason why you want to make Thrift parsing be streamy instead of
> protobufs?

Neither seems to particularly support C.

I notice Thrift compiles in Cygwin and requires things like Boost.
These all seem line "non-starters" to me.

-- 
Thanks

Mark Phippard
http://markphip.blogspot.com/

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org

Re: svn commit: r33398 - trunk/notes

Posted by Blair Zajac <bl...@orcaware.com>.
Justin Erenkrantz wrote:
> On Thu, Oct 2, 2008 at 8:06 AM,  <su...@tigris.org> wrote:
>> +  XML is out : there's a huge performance penalty for producing and
>> +  consuming it, which is why companies like Facebook and Google have
>> +  released 'fast wire serialization' libraries like Thrift and
>> +  Protocol Buffers.  Unfortunately, these libraries require entire
>> +  structures to be held in memory in order to serialize/deserialze
>> +  them, and this isn't an option when dealing with something
>> +  (potentially) infinitely large like an editor-drive.
> 
> Then perhaps we should look at improving Thrift parsing to be streamy.
>  I'd rather use a library than a custom serialization format - as you
> said, one of the goals should be used to make it simpler to
> understand.   And, one of the ways to do that is to go "Learn Thrift"
> rather than read our custom (de)serialization code.  =)

Is there a reason why you want to make Thrift parsing be streamy instead of 
protobufs?

Blair

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org

Re: svn commit: r33398 - trunk/notes

Posted by Ben Collins-Sussman <su...@red-bean.com>.
On Thu, Oct 2, 2008 at 11:58 AM, Justin Erenkrantz
<ju...@erenkrantz.com> wrote:


> Also, from a server admin perspective, if we
> are explicit about verbage in the URL path space, it makes it slightly
> simpler to configure balancing/proxying.  For example, I'd like to
> simplify the pass-through logic for the write-through mirror to just
> look at the URL space - looking at the bodies of the request is lame.
> Ideally, if we stick everything into URL space as paths, you could
> configure the load-balancing/write-through entirely with a cluster of
> ProxyPass directives - ProxyPass won't look at query strings, IIRC.
> -- justin
>

Very interesting.  Maybe I'll flip to using path space instead of
query strings, and see how it looks.

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org

Re: svn commit: r33398 - trunk/notes

Posted by Justin Erenkrantz <ju...@erenkrantz.com>.
On Thu, Oct 2, 2008 at 9:34 AM, Ben Collins-Sussman
<su...@red-bean.com> wrote:
> Curious -- why do you prefer putting the "verbs" and parameters
> directly into the path space, rather than having them as query
> strings?   I had chat about this topic with gstein in IRC, and he
> seemed to prefer query-args.
>
> Is there really any technical advantage to one style over another, or
> is it just a bikeshed?

There's a tiny amount of bikeshed as I think you can make both work,
but I think it's better to avoid query strings as there are at least
some costs involved in supporting them.  But, to me, I think doing
everything as query args makes the module a tiny bit more complex that
I think we could avoid.

Caching is also a factor here - as if we use query args, we have to
explicitly override and specify which responses are cacheable; but
some caches are a little stupid and don't consider query args as ever
being a differentiator.  Also, from a server admin perspective, if we
are explicit about verbage in the URL path space, it makes it slightly
simpler to configure balancing/proxying.  For example, I'd like to
simplify the pass-through logic for the write-through mirror to just
look at the URL space - looking at the bodies of the request is lame.
Ideally, if we stick everything into URL space as paths, you could
configure the load-balancing/write-through entirely with a cluster of
ProxyPass directives - ProxyPass won't look at query strings, IIRC.
-- justin

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org

Re: svn commit: r33398 - trunk/notes

Posted by Ben Collins-Sussman <su...@red-bean.com>.
On Thu, Oct 2, 2008 at 11:16 AM, Justin Erenkrantz
<ju...@erenkrantz.com> wrote:
> On Thu, Oct 2, 2008 at 8:06 AM,  <su...@tigris.org> wrote:
>> +  XML is out : there's a huge performance penalty for producing and
>> +  consuming it, which is why companies like Facebook and Google have
>> +  released 'fast wire serialization' libraries like Thrift and
>> +  Protocol Buffers.  Unfortunately, these libraries require entire
>> +  structures to be held in memory in order to serialize/deserialze
>> +  them, and this isn't an option when dealing with something
>> +  (potentially) infinitely large like an editor-drive.
>
> Then perhaps we should look at improving Thrift parsing to be streamy.
>  I'd rather use a library than a custom serialization format - as you
> said, one of the goals should be used to make it simpler to
> understand.   And, one of the ways to do that is to go "Learn Thrift"
> rather than read our custom (de)serialization code.  =)
>
>>  In the list of commands, all commands are assumed to be attached as
>> -?cmd=command to GET requests on the repository root URI, with the
>> -exception of {commit, ...}, which are attached to POST requests.
>> +?cmd=command to the request URI.  Command parameters are all
>> +query-encoded (&parm=val), and optional parameters are listed in
>> +square brackets.  Server response values are assumed to be in response
>> +bodies.
>
> So, a question is how much we want to push the HTTP envelope.  But,
> doing query parameters is kinda lame from a performance standpoint.
>
> The diff kinda butchered it, but you had:
> ---
>  get-latest-rev
>
>    GET /repos[/path]?cmd=get-latest-rev
>
>    response: revnum
> ---
>
> I'd rather see one of the following:
>
> ---
> GET-LATEST-REV /repos
> response: revnum
> ---
> or
> ---
> GET /repos/!svn/get-latest-rev
> response: revnum
> ---
>
> (I sort of prefer the latter, but could be convinced the former works too.)
>
> I think that args should be explicit in the URLs rather than then
> query strings, so get-dated-rev could be:
>
> ---
> GET /repos/!svn/get-dated-rev/<ISO date string>
> response: revnum
> ---
>
> Also:
> ---
>  rev-proplist
>
>    GET /repos[/path]?cmd=rev-proplist&rev=num
>
>    response:  proplist
> ---
>
> could be:
>
> ---
> GET /repos/!svn/<revnum>/props/rev
> response: proplist
> ---
>
> etc, etc, etc.
>
> My $.02.  -- justin
>

Curious -- why do you prefer putting the "verbs" and parameters
directly into the path space, rather than having them as query
strings?   I had chat about this topic with gstein in IRC, and he
seemed to prefer query-args.

Is there really any technical advantage to one style over another, or
is it just a bikeshed?

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org