You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@shindig.apache.org by Chris Chabot <ch...@xs4all.nl> on 2008/06/25 02:05:26 UTC
RESTful Batching support in shindig - spec compliance?
Hey Guys,
Gonzalo has been working hard on implementing Batching support to PHP
shindig (and doing a bang up job at it too), and looking at the
resulting code it does work with our restful-container.js
implementation..
Our implementation laces a couple of requests together into one big
json object. However looking at the spec at:
http://sites.google.com/a/opensocial.org/opensocial/Technical-Resources/opensocial-specification----implementation-version-08/restful-api-specification
I see a different approach to this, using multiple form parts in a
post. (and returning the response with the same kind of multi form
part headers too).
Are we aware of this, and whats the future plan here? Doing 2 types of
batching request support (wire format and spec format) or is the
future plan to merge these 2 and this is just code to make the basics
work as we are working towards the real implementation? (It feels to
me that that's the case here, or should be, but i don't remember
reading an email to that effect).
-- Chris
Re: RESTful Batching support in shindig - spec compliance?
Posted by Chris Chabot <ch...@xs4all.nl>.
Thanks for clarifying that, that helps a lot :)
It might be an idea indeed to consider adding it to the spec if we
keep it in.
Server sided it shouldn't be -to hard- to parse the multipart and use
the same internal request & response structures, i'll go see what i
can do with that.
If you could find that javascript that be great.
-- Chris
On Jun 25, 2008, at 3:13 AM, John Panzer wrote:
> Yeah, this was talked about on Friday but I guess Cassie didn't have
> time to
> mail the list before going on vacation.
>
> It's possible to implement the spec (mime/multipart) in Javascript,
> in fact
> we did it a few months back as a sanity check, I'll need to dig up
> the code
> to send around.
>
> The discussion so far has been that it's faster to just implement the
> JSON-only batch format for now. The spec doesn't disallow this as the
> /batch-proxy URL can accept any MIME type you care to throw at it (and
> define), if we decide to standardize the JSON one we should probably
> write
> up some spec text though :). application/json batch can co-exist with
> mime/multipart easily enough if necessary.
>
> John Panzer (http://abstractioneer.org)
>
> On Tue, Jun 24, 2008 at 6:04 PM, Kevin Brown <et...@google.com> wrote:
>
>> On Tue, Jun 24, 2008 at 5:54 PM, Chris Chabot <ch...@xs4all.nl>
>> wrote:
>>
>>> Did you forget the apache meme? If it's not in my mailbox, it
>>> doesn't
>> exist
>>> ! :-)
>>
>>
>> Yeah, there's not really anything to send around though -- we
>> talked about
>> it, but nobody really had a good idea about what to do. I assume that
>> Cassie
>> was taking care of it since she's been working in that area.
>>
>> Might be nice to keep us who are not privy to the google meetings
>> involved
>>> in it? (both spec and shindig wise), else we'll end up having people
>>> spinning their wheels for nothing, and no one likes to loose time,
>>> even
>> just
>>> some heads up would be useful for the people involved. (we went by
>> cassie's
>>> words "as soon as the php guys implement batching you can flip the
>> switch",
>>> so i had no insight in nor hints to these discussions)
>>
>>
>> Well, there weren't actually any proposals for what to do, so there
>> wasn't
>> anything to share. If someone has a good idea of how to do it and
>> wants to
>> write it, by all means go for it. If you can make it work for PHP
>> I'm sure
>> it can be adapted for Java (there's certainly no reason why the PHP
>> code
>> has
>> to follow the lead of what's done in the Java base, after all).
>>
>>
>>> What's the problem, impossible to make such a request from
>>> javascript
>>> (something i could imagine)? In that case i guess at least we can
>>> keep
>> the
>>> work we did so far on the php side.
>>
>>
>> I don't know; David and Cassie had different opinions on the
>> matter, so
>> they
>> talked to try to sort it out. I don't think they came to any
>> conclusion on
>> the matter, so hopefully someone with better ideas will :).
>>
>>
>>>
>>> -- Chris
>>>
>>>
>>> On Jun 25, 2008, at 2:47 AM, Kevin Brown wrote:
>>>
>>> Several of us have been discussing this problem over at Google,
>>> but I
>>>> don't
>>>> know that there was a clear conclusion
>>>>
>>>
>>>
>>
Re: RESTful Batching support in shindig - spec compliance?
Posted by John Panzer <jp...@google.com>.
FWIW, here's some _very_ quick and dirty Javascript that reads and writes an
(earlier, cruftier version of) mime/multipart we used for texting a while
back. It not what you'd want to use in production but it's a useful sanity
check on implementability complexity I think.
<script type="text/javascript">
function println(txt) {
document.write(txt + '<br>');
}
function parseResponse(req) {
if (!req) {
println("no request");
return null;
}
var response = req.responseText;
var header = req.getResponseHeader('Content-Type');
var date = new Date();
var startTime = date.getMilliseconds();
var responses = [];
if (!response) {
println('no response');
return null;
}
if (!header) {
println('no header');
return null;
}
var boundaryidx = header.indexOf('boundary=');
if (boundaryidx == -1) {
println('header: [' +header + ']');
println('no boundary found');
return null;
}
boundary = header.substring(boundaryidx + 9);
var parts = response.split('--' + boundary);
var l = parts.length;
for (var i = 0; i < l; i++) {
var p = parts[i];
if (p == '--\r\n') continue;
if (p == '') continue;
var bodystartidx = p.indexOf('\r\n\r\n');
var rawheaders = p.substring(2,bodystartidx);
var rawbody = p.substring(bodystartidx+4,p.length);
var statuscode = null;
var statustext = null;
var headers = rawheaders.split('\r\n');
var status = headers[0];
var statuscodeindex = status.indexOf(':');
statuscode = status.substring(statuscodeindex+2);
var statustextindex = statuscode.indexOf(' ');
statustext = statuscode.substring(statustextindex+1);
statuscode = statuscode.substring(0, statustextindex);
var lh = headers.length;
var headerobjects = [];
for (var j = 0; j < lh; j++) {
var text = headers[j];
var delimidx = text.indexOf(':');
headerobjects.push({'name' : text.substring(0, delimidx),
'value' : text.substring(delimidx + 2)});
}
responses.push({'headers' : headerobjects, 'body' : rawbody,
'statuscode' : statuscode, 'statustext' : statustext});
}
date = new Date();
var endTime = date.getMilliseconds();
println('PARSE TIME (in milliseconds): ' + (endTime - startTime));
return responses;
}
function printResponses(responses) {
println('Dump Response Array');
if (!responses) {
println("no responses");
return;
}
var l = responses.length;
for (var i = 0; i < l; i++) {
var r = responses[i];
for (var j = 0; j < r.headers.length; j++) {
var ho = r.headers[j];
println('header[' + ho.name + '] = [' + ho.value + ']');
}
println('statuscode: [' + r.statuscode + ']');
println('statustext: [' + r.statustext + ']');
println('body: [' + r.body + ']');
println('------------------------------------');
}
}
function EXPECT_EQ(value, result, idx) {
if (idx >= result.length) {
return 'FAIL EXPECT_EQ. Index[' + idx + '] exceeds length[' +
result.length + ']';
}
if (value == result[idx]) {
return 'PASS EXPECT_EQ. Idx[' + idx + '] Expected[' + value + ']
Actual[' + result[idx] + ']';
}
return 'FAIL EXPECT_EQ. Idx[' + idx + '] Expected[' + value + ']
Actual[' + result[idx] + ']';
}
function saveXMLDoc(url, text, boundaryString) {
var req = null;
if(window.ActiveXObject) {
try {
req = new ActiveXObject('Msxml2.XMLHTTP');
} catch(e) {
try {
req = new ActiveXObject('Microsoft.XMLHTTP');
} catch(e) {
req = null;
}
}
} else if(window.XMLHttpRequest) {
try {
req = new XMLHttpRequest();
} catch(e) {
req = null;
}
}
if(req) {
req.open('POST', url, false);
req.setRequestHeader('Content-Type', 'multipart/mixed; boundary='
+ boundaryString);
req.send(text);
if (req.readyState == 4 && req.status == 200) {
return req;
}
return null;
}
function sendRequest() {
var boundaryString = 'batch-a73hdj3dy3mm347ddjjdf';
var boundary = '--' + boundaryString;
var text = '';
text += boundary + '\r\n';
text += 'X-Batch-Operation: POST /people/friends HTTP/1.1' +
'\r\n';
text += 'Host: container.org' + '\r\n';
text += '\r\n';
text += '...representation of new Person elided...' + '\r\n';
text += '\r\n';
text += boundary + '\r\n';
text += 'X-Batch-Operation: GET /people/friends/page=5&num=10'
+ '\r\n';
text += 'Host: container.org' + '\r\n';
text += '-batch-a73hdj3dy3mm347ddjjdf' + '\r\n';
text += 'If-None-Match: "837dyfkdi39df"' + '\r\n';
text += '\r\n';
text += boundary + '--\r\n';
return
saveXMLDoc('http://example.org/batch-proxy<http://warpgundy.corp.google.com:8000/rnr>',
text, boundaryString);
}
var req = sendRequest();
var resp = parseResponse(req);
printResponses(resp);
</script>
John Panzer (http://abstractioneer.org)
On Tue, Jun 24, 2008 at 6:13 PM, John Panzer <jp...@google.com> wrote:
> Yeah, this was talked about on Friday but I guess Cassie didn't have time
> to mail the list before going on vacation.
>
> It's possible to implement the spec (mime/multipart) in Javascript, in fact
> we did it a few months back as a sanity check, I'll need to dig up the code
> to send around.
>
> The discussion so far has been that it's faster to just implement the
> JSON-only batch format for now. The spec doesn't disallow this as the
> /batch-proxy URL can accept any MIME type you care to throw at it (and
> define), if we decide to standardize the JSON one we should probably write
> up some spec text though :). application/json batch can co-exist with
> mime/multipart easily enough if necessary.
>
> John Panzer (http://abstractioneer.org)
>
> On Tue, Jun 24, 2008 at 6:04 PM, Kevin Brown <et...@google.com> wrote:
>
>> On Tue, Jun 24, 2008 at 5:54 PM, Chris Chabot <ch...@xs4all.nl> wrote:
>>
>> > Did you forget the apache meme? If it's not in my mailbox, it doesn't
>> exist
>> > ! :-)
>>
>>
>> Yeah, there's not really anything to send around though -- we talked about
>> it, but nobody really had a good idea about what to do. I assume that
>> Cassie
>> was taking care of it since she's been working in that area.
>>
>> Might be nice to keep us who are not privy to the google meetings involved
>> > in it? (both spec and shindig wise), else we'll end up having people
>> > spinning their wheels for nothing, and no one likes to loose time, even
>> just
>> > some heads up would be useful for the people involved. (we went by
>> cassie's
>> > words "as soon as the php guys implement batching you can flip the
>> switch",
>> > so i had no insight in nor hints to these discussions)
>>
>>
>> Well, there weren't actually any proposals for what to do, so there wasn't
>> anything to share. If someone has a good idea of how to do it and wants to
>> write it, by all means go for it. If you can make it work for PHP I'm sure
>> it can be adapted for Java (there's certainly no reason why the PHP code
>> has
>> to follow the lead of what's done in the Java base, after all).
>>
>>
>> > What's the problem, impossible to make such a request from javascript
>> > (something i could imagine)? In that case i guess at least we can keep
>> the
>> > work we did so far on the php side.
>>
>>
>> I don't know; David and Cassie had different opinions on the matter, so
>> they
>> talked to try to sort it out. I don't think they came to any conclusion on
>> the matter, so hopefully someone with better ideas will :).
>>
>>
>> >
>> > -- Chris
>> >
>> >
>> > On Jun 25, 2008, at 2:47 AM, Kevin Brown wrote:
>> >
>> > Several of us have been discussing this problem over at Google, but I
>> >> don't
>> >> know that there was a clear conclusion
>> >>
>> >
>> >
>>
>
>
Re: RESTful Batching support in shindig - spec compliance?
Posted by John Panzer <jp...@google.com>.
Yeah, this was talked about on Friday but I guess Cassie didn't have time to
mail the list before going on vacation.
It's possible to implement the spec (mime/multipart) in Javascript, in fact
we did it a few months back as a sanity check, I'll need to dig up the code
to send around.
The discussion so far has been that it's faster to just implement the
JSON-only batch format for now. The spec doesn't disallow this as the
/batch-proxy URL can accept any MIME type you care to throw at it (and
define), if we decide to standardize the JSON one we should probably write
up some spec text though :). application/json batch can co-exist with
mime/multipart easily enough if necessary.
John Panzer (http://abstractioneer.org)
On Tue, Jun 24, 2008 at 6:04 PM, Kevin Brown <et...@google.com> wrote:
> On Tue, Jun 24, 2008 at 5:54 PM, Chris Chabot <ch...@xs4all.nl> wrote:
>
> > Did you forget the apache meme? If it's not in my mailbox, it doesn't
> exist
> > ! :-)
>
>
> Yeah, there's not really anything to send around though -- we talked about
> it, but nobody really had a good idea about what to do. I assume that
> Cassie
> was taking care of it since she's been working in that area.
>
> Might be nice to keep us who are not privy to the google meetings involved
> > in it? (both spec and shindig wise), else we'll end up having people
> > spinning their wheels for nothing, and no one likes to loose time, even
> just
> > some heads up would be useful for the people involved. (we went by
> cassie's
> > words "as soon as the php guys implement batching you can flip the
> switch",
> > so i had no insight in nor hints to these discussions)
>
>
> Well, there weren't actually any proposals for what to do, so there wasn't
> anything to share. If someone has a good idea of how to do it and wants to
> write it, by all means go for it. If you can make it work for PHP I'm sure
> it can be adapted for Java (there's certainly no reason why the PHP code
> has
> to follow the lead of what's done in the Java base, after all).
>
>
> > What's the problem, impossible to make such a request from javascript
> > (something i could imagine)? In that case i guess at least we can keep
> the
> > work we did so far on the php side.
>
>
> I don't know; David and Cassie had different opinions on the matter, so
> they
> talked to try to sort it out. I don't think they came to any conclusion on
> the matter, so hopefully someone with better ideas will :).
>
>
> >
> > -- Chris
> >
> >
> > On Jun 25, 2008, at 2:47 AM, Kevin Brown wrote:
> >
> > Several of us have been discussing this problem over at Google, but I
> >> don't
> >> know that there was a clear conclusion
> >>
> >
> >
>
Re: RESTful Batching support in shindig - spec compliance?
Posted by Cassie <do...@google.com>.
Oh my - now this was an interesting thread in my absence. Now don't worry -
I haven't been making any decisions without you Chris!
My thought process went like this:
we have to implement the spec
we also need to get the opensocial js library working on top of restful
we need it done yesterday
the spec says that its batching is optional
multipart stuff looks like it would take some work
the current shindig wire format already has a simpler batching format and we
need this for prod opensocial js libraries
the spec says extensions are fine
okay, i'll just port the current stuff over - same js code, minor java
changes.
we can change this to match the spec format later, or always keep this just
for the js library or do something else, who knows - lets discuss.
vacation.
and there you go! custom simple batching thingy whipped together!
this doesn't have to stay in shindig and can be replaced with the multipart
stuff, or it could be added to the spec at a later date. it all depends on
what we want to do.
on that note, has anyone made any progress on the spec consistent batching?
- cassie
On Tue, Jun 24, 2008 at 6:29 PM, Chris Chabot <ch...@xs4all.nl> wrote:
> My main concern was if this would impact our spec support or not. Having
> some code in place that does not follow the spec, is something we usually
> try to avoid right? :)
>
> John's answer clarified that for me though, seems we have the
> 'alternate/new' json format for now, which possibly might be added to the
> spec if it sticks around, and hopefully later also add code that deals with
> the way it's currently documented in the spec.
>
>
>
>
> On Jun 25, 2008, at 3:21 AM, Kevin Brown wrote:
>
> I dunno, I usually don't post threads saying I can't figure out how to do
>> something unless someone was expecting that I was doing it. Were you
>> actually waiting on Cassie on this? If so, yeah -- she probably should
>> have
>> given you a heads up that there were some issues. I don't see why you'd
>> wait
>> on her though if you've got something that works.
>>
>
>
Re: RESTful Batching support in shindig - spec compliance?
Posted by Chris Chabot <ch...@xs4all.nl>.
My main concern was if this would impact our spec support or not.
Having some code in place that does not follow the spec, is something
we usually try to avoid right? :)
John's answer clarified that for me though, seems we have the
'alternate/new' json format for now, which possibly might be added to
the spec if it sticks around, and hopefully later also add code that
deals with the way it's currently documented in the spec.
On Jun 25, 2008, at 3:21 AM, Kevin Brown wrote:
> I dunno, I usually don't post threads saying I can't figure out how
> to do
> something unless someone was expecting that I was doing it. Were you
> actually waiting on Cassie on this? If so, yeah -- she probably
> should have
> given you a heads up that there were some issues. I don't see why
> you'd wait
> on her though if you've got something that works.
Re: RESTful Batching support in shindig - spec compliance?
Posted by Kevin Brown <et...@google.com>.
On Tue, Jun 24, 2008 at 6:10 PM, Chris Chabot <ch...@xs4all.nl> wrote:
> If only we had heard their sides and ideas, it would've been a lot easier
> to expand upon, and if we collectively can't come up with something it might
> be a good idea to take it to the spec list otherwise? There's a lot of smart
> people there too from various parties that can provide some useful input
> too.
>
> Sorry but i would've really preferred if we had at least some idea that
> some people were at least aware that there was a problem, and what it was,
> so that people could contribute to it where possible, and otherwise at least
> know what's up you know :)
I dunno, I usually don't post threads saying I can't figure out how to do
something unless someone was expecting that I was doing it. Were you
actually waiting on Cassie on this? If so, yeah -- she probably should have
given you a heads up that there were some issues. I don't see why you'd wait
on her though if you've got something that works.
So what exactly is the problem? The multi part construction from javascript
> or the server side interpretation of it?
Frankly, I don't even know, but I haven't really looked at the REST code
thoroughly enough to have an opinion on whether there's a problem at all.
>
>
> On Jun 25, 2008, at 3:04 AM, Kevin Brown wrote:
>
>
>> I don't know; David and Cassie had different opinions on the matter, so
>> they
>> talked to try to sort it out. I don't think they came to any conclusion on
>> the matter, so hopefully someone with better ideas will :).
>>
>
>
Re: RESTful Batching support in shindig - spec compliance?
Posted by Chris Chabot <ch...@xs4all.nl>.
If only we had heard their sides and ideas, it would've been a lot
easier to expand upon, and if we collectively can't come up with
something it might be a good idea to take it to the spec list
otherwise? There's a lot of smart people there too from various
parties that can provide some useful input too.
Sorry but i would've really preferred if we had at least some idea
that some people were at least aware that there was a problem, and
what it was, so that people could contribute to it where possible, and
otherwise at least know what's up you know :)
So what exactly is the problem? The multi part construction from
javascript or the server side interpretation of it?
On Jun 25, 2008, at 3:04 AM, Kevin Brown wrote:
>
> I don't know; David and Cassie had different opinions on the matter,
> so they
> talked to try to sort it out. I don't think they came to any
> conclusion on
> the matter, so hopefully someone with better ideas will :).
Re: RESTful Batching support in shindig - spec compliance?
Posted by Kevin Brown <et...@google.com>.
On Tue, Jun 24, 2008 at 5:54 PM, Chris Chabot <ch...@xs4all.nl> wrote:
> Did you forget the apache meme? If it's not in my mailbox, it doesn't exist
> ! :-)
Yeah, there's not really anything to send around though -- we talked about
it, but nobody really had a good idea about what to do. I assume that Cassie
was taking care of it since she's been working in that area.
Might be nice to keep us who are not privy to the google meetings involved
> in it? (both spec and shindig wise), else we'll end up having people
> spinning their wheels for nothing, and no one likes to loose time, even just
> some heads up would be useful for the people involved. (we went by cassie's
> words "as soon as the php guys implement batching you can flip the switch",
> so i had no insight in nor hints to these discussions)
Well, there weren't actually any proposals for what to do, so there wasn't
anything to share. If someone has a good idea of how to do it and wants to
write it, by all means go for it. If you can make it work for PHP I'm sure
it can be adapted for Java (there's certainly no reason why the PHP code has
to follow the lead of what's done in the Java base, after all).
> What's the problem, impossible to make such a request from javascript
> (something i could imagine)? In that case i guess at least we can keep the
> work we did so far on the php side.
I don't know; David and Cassie had different opinions on the matter, so they
talked to try to sort it out. I don't think they came to any conclusion on
the matter, so hopefully someone with better ideas will :).
>
> -- Chris
>
>
> On Jun 25, 2008, at 2:47 AM, Kevin Brown wrote:
>
> Several of us have been discussing this problem over at Google, but I
>> don't
>> know that there was a clear conclusion
>>
>
>
Re: RESTful Batching support in shindig - spec compliance?
Posted by Chris Chabot <ch...@xs4all.nl>.
Did you forget the apache meme? If it's not in my mailbox, it doesn't
exist ! :-)
Might be nice to keep us who are not privy to the google meetings
involved in it? (both spec and shindig wise), else we'll end up having
people spinning their wheels for nothing, and no one likes to loose
time, even just some heads up would be useful for the people involved.
(we went by cassie's words "as soon as the php guys implement batching
you can flip the switch", so i had no insight in nor hints to these
discussions)
What's the problem, impossible to make such a request from javascript
(something i could imagine)? In that case i guess at least we can keep
the work we did so far on the php side.
-- Chris
On Jun 25, 2008, at 2:47 AM, Kevin Brown wrote:
> Several of us have been discussing this problem over at Google, but
> I don't
> know that there was a clear conclusion
Re: RESTful Batching support in shindig - spec compliance?
Posted by Kevin Brown <et...@google.com>.
On Tue, Jun 24, 2008 at 5:05 PM, Chris Chabot <ch...@xs4all.nl> wrote:
> Hey Guys,
>
> Gonzalo has been working hard on implementing Batching support to PHP
> shindig (and doing a bang up job at it too), and looking at the resulting
> code it does work with our restful-container.js implementation..
>
> Our implementation laces a couple of requests together into one big json
> object. However looking at the spec at:
>
> http://sites.google.com/a/opensocial.org/opensocial/Technical-Resources/opensocial-specification----implementation-version-08/restful-api-specification
>
> I see a different approach to this, using multiple form parts in a post.
> (and returning the response with the same kind of multi form part headers
> too).
>
> Are we aware of this, and whats the future plan here? Doing 2 types of
> batching request support (wire format and spec format) or is the future plan
> to merge these 2 and this is just code to make the basics work as we are
> working towards the real implementation? (It feels to me that that's the
> case here, or should be, but i don't remember reading an email to that
> effect).
Several of us have been discussing this problem over at Google, but I don't
know that there was a clear conclusion. Cassie was going to come up with a
proposal for doing it, but she's on vacation for the next two weeks. Dave
Primmer might know.
>
> -- Chris
>