Current REST-API - Sessions

[Ethan Jewett <es...@gmail.com>]

Hi all,

Last week I said I was going to start working on a YQL-wrapper for the
ESME API, hoping that my work in that area would start to drive API
development again. (After the whole RPC vs. REST uproar we ended up
with one blog post, one page on the wiki at
http://incubator.apache.org/esme/restapi.html and that was it, which
was mostly my fault for losing interest for various reasons.)

Today I started and almost immediately ran into the requirement of the
current API to sustain a session in the client. I know it would be
possible to simulate this by manually creating the appropriate headers
in Javascript, however I don't think this is a reasonable approach
with YQL as YQL itself has no mechanism to allow storing a session key
across requests, so session key storage would have to be managed by
the YQL client, and then the session key passed in the YQL request.

I'd like to revisit the use of sessions in the API. I do not know
Lift, but my understanding is that we gain some ease of use in the
scenario of interfaces built on top of the API using Lift because of
its automatic handling of sessions. Are there other reasons?

I'd like to understand all the reasons for this approach so that we
can figure out if there is an alternate way to handle this that is
more in line with the way web APIs are programmed these days (and
subsequently will hopefully be more useful with web API interaction
tools like YQL work).

Thanks,
Ethan

Re: Current REST-API - Sessions

[Vassil Dichev <vd...@apache.org>]

> I'd be interested to see whatever we have. Nothing comes up when I
> google the old esme-dev google group, but maybe someone has something
> somewhere. I checked the old REST API documentation page and didn't
> see anything. Do you think I would be able to figure it out from
> looking at the source code?

Source code is how I figured it out :) and the description's probably
on an even older mailing list ;-) Currently I see only a mapping of
current API to a recommended REST API in the wiki, is this the best
place I should document it?

> That guy may have been me. I'm slowly having it explained to me that
> the design principles of ESME don't necessarily match up to the design
> principles of HTTP and to a certain extent REST. You'll have to bear
> with me as I work through the cognitive dissonance of an HTTP
> (document-focused) API to a stream-focused tool. :-)

It definitely wasn't you :) I was starting to recognize the principles
of pure REST at about that time myself, so you probably know more than
me already.

> I think what is most helpful is that exceptions have justifications
> that are documented unemotionally somewhere accessible. I had not
> understood the idea of ESME as stream-based until today due to this
> conversation with you and another offline conversation with David. I
> now realize that I had seen these very same justifications written
> about multiple times before, but never in a way I personally could
> understand. Hopefully on my next flight I'll be able to write a blog
> and/or wiki page on the topic that may help out others with this
> problem in the future.

As a rule of thumb, we shouldn't convert the long poll calls into
REST-compliant methods.

> Thanks for taking the time to work with me on this!

You're welcome! As a result of this there's a chance that even I start
understanding this REST stuff :)

Vassil

Re: Current REST-API - Sessions

[Ethan Jewett <es...@gmail.com>]

On Mon, Aug 3, 2009 at 5:20 PM, Vassil Dichev<vd...@apache.org> wrote:
> Noted. Maybe it's not even on the old page, but in a mail somewhere in
> the old mailing list, which I agree is next to useless.

I'd be interested to see whatever we have. Nothing comes up when I
google the old esme-dev google group, but maybe someone has something
somewhere. I checked the old REST API documentation page and didn't
see anything. Do you think I would be able to figure it out from
looking at the source code?

>
> I'm sorry if it sounded like I'm disparaging your efforts to stick to
> the REST principles- I didn't mean to. I just recalled that the "whole
> RPC vs. REST uproar" was caused by a guy who had some valid points but
> in the end wasn't aware of how ESME works and the reasons it's
> designed this way. So if I was a bit bitter, it was because of this
> case when someone jumps to conclusions and condemns ESME as being all
> wrong because we called it REST instead of RESTful or REST-like.
> Sheesh...

That guy may have been me. I'm slowly having it explained to me that
the design principles of ESME don't necessarily match up to the design
principles of HTTP and to a certain extent REST. You'll have to bear
with me as I work through the cognitive dissonance of an HTTP
(document-focused) API to a stream-focused tool. :-)

At root I think the issue is that we are talking about an HTTP
programming interface to a tool that is conceived in a way that is
totally different from HTTP. An interesting problem, and one that I
had not realized existed until now. I think it would be nice to
provide a way to expose all ESME objects as traditional HTTP resources
in a RESTful manner, but I need to go back to the drawing board to
think about whether this is even reasonable.

> Let us know about the use case (what do you want to achieve?), so we
> are better prepared to help. If it's not in the Twitter API, then it
> must be pretty specific to ESME. What are you trying to use- actions,
> pools, tagclouds, tracking...?

What I'd like to do is expose as much of the ESME API as possible
through YQL. I'm not sure this is always going to be reasonable, but
I'd like to do it anyway, both as an exercise, and to make ESME more
mashable and exposed to possible use-cases we had not thought of
before. I'm specifically looking to make the ESME API accessible
through Yahoo! Pipes and through Webhook interfaces. A YQL wrapper
should make this possible with relatively little effort.

> For me the takeaway point of the rant was that we shouldn't try to fit
> everything into a mold. Not all apps should conform completely to a
> design pattern at all costs. One should understand the underlying
> reasons for sticking to a best practice before trying to apply it
> everywhere, otherwise one might be trying to fit square pegs into
> round holes. What this means for ESME- it's fine to use some of the
> advice regarding REST, but the prescription that all interaction is
> stateless doesn't fit with ESME, where state is important and is
> central to the application design.
>
> Maybe as a result of this old discussion we should have outlined the
> things that we can do to improve things and stuff that we are not
> going to change, because it doesn't make sense. This way there should
> be no false expectations regarding what we wanted to do.

I think what is most helpful is that exceptions have justifications
that are documented unemotionally somewhere accessible. I had not
understood the idea of ESME as stream-based until today due to this
conversation with you and another offline conversation with David. I
now realize that I had seen these very same justifications written
about multiple times before, but never in a way I personally could
understand. Hopefully on my next flight I'll be able to write a blog
and/or wiki page on the topic that may help out others with this
problem in the future.

Thanks for taking the time to work with me on this!

Ethan

Re: Current REST-API - Sessions

[Vassil Dichev <vd...@apache.org>]

> Yup, I'm aware of the Twitter API. I'd like to be able to use an API
> that exposes all of the functionality of ESME, which definitely goes
> way beyond what Twitter and the Twitter API provide.

I wasn't sure what you're trying to do, but this sounds like a pretty
complicated use of YQL.

> I wasn't aware of the existence of two versions of this API method.
> From the old REST API documentation it looked to me like I had to have
> a session set up to use any of the API methods. I may have missed this
> though. Is it on the old REST API page on Google Code?

Noted. Maybe it's not even on the old page, but in a mail somewhere in
the old mailing list, which I agree is next to useless.

> Right, got it. It was a bad choice of words. I don't claim more
> expertise on this than anyone here. Perhaps somewhat different
> expertise, but not more.

I'm sorry if it sounded like I'm disparaging your efforts to stick to
the REST principles- I didn't mean to. I just recalled that the "whole
RPC vs. REST uproar" was caused by a guy who had some valid points but
in the end wasn't aware of how ESME works and the reasons it's
designed this way. So if I was a bit bitter, it was because of this
case when someone jumps to conclusions and condemns ESME as being all
wrong because we called it REST instead of RESTful or REST-like.
Sheesh...

> What I meant was this: I think for programming interfaces there is
> something to be said for sticking to a convention unless there are
> good technical or usability reasons to break with the convention. The
> fact is that few existing APIs require the use of sessions and because
> of this very few of the environments and libraries with which API
> clients are programmed have robust session support. My impression is
> that in this case (YQL) and in other cases (dynamic languages,
> server-based programming environments, etc), interoperability would be
> improved by removing sessions.

Let us know about the use case (what do you want to achieve?), so we
are better prepared to help. If it's not in the Twitter API, then it
must be pretty specific to ESME. What are you trying to use- actions,
pools, tagclouds, tracking...?

> I'm wondering if there are specific reasons this choice was made, as I
> still don't understand the trade-off. Apologies if I'm being thick,
> but David's earlier rant didn't really clarify things for me if it is
> the one I'm thinking of (I'm on a plane as I write this, so I don't
> have immediate access to it). I assure you, I carefully read all rants
> I receive, though I don't usually find it very useful to respond. :-)
>

For me the takeaway point of the rant was that we shouldn't try to fit
everything into a mold. Not all apps should conform completely to a
design pattern at all costs. One should understand the underlying
reasons for sticking to a best practice before trying to apply it
everywhere, otherwise one might be trying to fit square pegs into
round holes. What this means for ESME- it's fine to use some of the
advice regarding REST, but the prescription that all interaction is
stateless doesn't fit with ESME, where state is important and is
central to the application design.

Maybe as a result of this old discussion we should have outlined the
things that we can do to improve things and stuff that we are not
going to change, because it doesn't make sense. This way there should
be no false expectations regarding what we wanted to do.

Vassil

Re: Current REST-API - Sessions

[Ethan Jewett <es...@gmail.com>]

Hi Vassil,

Replies inline below.

Thanks,
Ethan

2009/8/3 Vassil Dichev <vd...@apache.org>:
>
> For a start, you could use the Twitter API, where nothing is stateful,
> and you don't need to be logged in.

Yup, I'm aware of the Twitter API. I'd like to be able to use an API
that exposes all of the functionality of ESME, which definitely goes
way beyond what Twitter and the Twitter API provide.

> I don't think I understand the question. We *can* use stateful *and*
> stateless calls in Lift, too. For instance, the post_msg has two
> versions, and for one of these, you don't need to be logged in. Surely
> you don't want ESME to reimplement its own version of a RESTful API
> creation library, when there's a convenient one in Lift already?

I wasn't aware of the existence of two versions of this API method.
>From the old REST API documentation it looked to me like I had to have
a session set up to use any of the API methods. I may have missed this
though. Is it on the old REST API page on Google Code?

>> I'd like to understand all the reasons for this approach so that we
>> can figure out if there is an alternate way to handle this that is
>> more in line with the way web APIs are programmed these days (and
>> subsequently will hopefully be more useful with web API interaction
>> tools like YQL work).
>
> Ethan, "the way web APIs are programmed these days" may not always be
> the most appropriate thing to do for any project. I don't think you
> need another rant from David, there is an old one on the topic already
> ;-)
>
> http://mail-archives.apache.org/mod_mbox/incubator-esme-dev/200812.mbox/%3Ccdbebedf0812212111k7d618729r5d3a10a5844b0a74@mail.gmail.com%3E
>

Right, got it. It was a bad choice of words. I don't claim more
expertise on this than anyone here. Perhaps somewhat different
expertise, but not more.

What I meant was this: I think for programming interfaces there is
something to be said for sticking to a convention unless there are
good technical or usability reasons to break with the convention. The
fact is that few existing APIs require the use of sessions and because
of this very few of the environments and libraries with which API
clients are programmed have robust session support. My impression is
that in this case (YQL) and in other cases (dynamic languages,
server-based programming environments, etc), interoperability would be
improved by removing sessions.

I'm wondering if there are specific reasons this choice was made, as I
still don't understand the trade-off. Apologies if I'm being thick,
but David's earlier rant didn't really clarify things for me if it is
the one I'm thinking of (I'm on a plane as I write this, so I don't
have immediate access to it). I assure you, I carefully read all rants
I receive, though I don't usually find it very useful to respond. :-)

Re: Current REST-API - Sessions

[Vassil Dichev <vd...@apache.org>]

> Today I started and almost immediately ran into the requirement of the
> current API to sustain a session in the client. I know it would be
> possible to simulate this by manually creating the appropriate headers
> in Javascript, however I don't think this is a reasonable approach
> with YQL as YQL itself has no mechanism to allow storing a session key
> across requests, so session key storage would have to be managed by
> the YQL client, and then the session key passed in the YQL request.

For a start, you could use the Twitter API, where nothing is stateful,
and you don't need to be logged in.

> I'd like to revisit the use of sessions in the API. I do not know
> Lift, but my understanding is that we gain some ease of use in the
> scenario of interfaces built on top of the API using Lift because of
> its automatic handling of sessions. Are there other reasons?

I don't think I understand the question. We *can* use stateful *and*
stateless calls in Lift, too. For instance, the post_msg has two
versions, and for one of these, you don't need to be logged in. Surely
you don't want ESME to reimplement its own version of a RESTful API
creation library, when there's a convenient one in Lift already?

> I'd like to understand all the reasons for this approach so that we
> can figure out if there is an alternate way to handle this that is
> more in line with the way web APIs are programmed these days (and
> subsequently will hopefully be more useful with web API interaction
> tools like YQL work).

Ethan, "the way web APIs are programmed these days" may not always be
the most appropriate thing to do for any project. I don't think you
need another rant from David, there is an old one on the topic already
;-)

http://mail-archives.apache.org/mod_mbox/incubator-esme-dev/200812.mbox/%3Ccdbebedf0812212111k7d618729r5d3a10a5844b0a74@mail.gmail.com%3E

Re: Current REST-API - Sessions

[Ethan Jewett <es...@gmail.com>]

On Mon, Aug 3, 2009 at 3:39 PM, Darren Hague<dh...@fortybeans.com> wrote:
> I wonder if a possible solution would be to allow a timestamp to be passed
> into the API calls?

This is something I'd like to explore and is built into HTTP as the
If-Modified-Since and Last-Modified headers. I'm not sure it gets us
as far as the current implementation with sessions does and I'm not
sure how long-polling could be worked into this scheme, so there's
definitely a lot to consider still.

Ethan

Re: Current REST-API - Sessions

[Darren Hague <dh...@fortybeans.com>]

I wonder if a possible solution would be to allow a timestamp to be 
passed into the API calls? Those calls which would appear to require 
state could be made stateless if called as "get me the data since this 
timestamp", and would return the timestamp of the last item in the data 
returned. This would mean that the client could guarantee to receive all 
the data from the server, but a session ID would not be required.

Just a thought,
Darren

Ethan Jewett wrote:
> Hi David,
>
> Thanks for looking into this. I definitely agree that we need state on
> some calls. Partly I'm curious as to if sessions are the best way to
> do this, but I'm good with whatever we have and I'm aware of the
> resource constraints.
>
> I've been wondering if there is a good way to provide optional state
> through sessions as well as a stateless mode for some calls. Is this
> what you mean by filters?
>
> I'll see if I can get in touch with Sam on this topic as well, as he's
> offered help to me on other YQL interfaces I've worked on.
>
> Ethan
>
> On Mon, Aug 3, 2009 at 9:02 AM, David
> Pollak<fe...@gmail.com> wrote:
>   
>> On Sun, Aug 2, 2009 at 8:45 AM, Ethan Jewett <es...@gmail.com> wrote:
>>
>>     
>>> Hi all,
>>>
>>> Last week I said I was going to start working on a YQL-wrapper for the
>>> ESME API, hoping that my work in that area would start to drive API
>>> development again. (After the whole RPC vs. REST uproar we ended up
>>> with one blog post, one page on the wiki at
>>> http://incubator.apache.org/esme/restapi.html and that was it, which
>>> was mostly my fault for losing interest for various reasons.)
>>>
>>> Today I started and almost immediately ran into the requirement of the
>>> current API to sustain a session in the client. I know it would be
>>> possible to simulate this by manually creating the appropriate headers
>>> in Javascript, however I don't think this is a reasonable approach
>>> with YQL as YQL itself has no mechanism to allow storing a session key
>>> across requests, so session key storage would have to be managed by
>>> the YQL client, and then the session key passed in the YQL request.
>>>
>>> I'd like to revisit the use of sessions in the API. I do not know
>>> Lift, but my understanding is that we gain some ease of use in the
>>> scenario of interfaces built on top of the API using Lift because of
>>> its automatic handling of sessions. Are there other reasons?
>>>
>>> I'd like to understand all the reasons for this approach so that we
>>> can figure out if there is an alternate way to handle this that is
>>> more in line with the way web APIs are programmed these days (and
>>> subsequently will hopefully be more useful with web API interaction
>>> tools like YQL work).
>>>       
>> There are some ESME APIs for which stateless/"RESTful" is appropriate
>> (things that refer to entities that are stable... basically things that map
>> to RDBMS calls) and things where stateless is not appropriate (things that
>> refer to event streams.)  There are clients for which stateless is
>> appropriate (in this case YQL) and clients for which stateless is not as
>> appropriate (i.e., UI clients, web based or desktop based.)
>>
>> We have limited resources and I don't want to have duplication API
>> implementations (or even filters) for stateful and stateless because that's
>> just more work for us, more stuff for us to maintain, and this is the first
>> use case for actually having stateless that's appeared on this list.
>>
>> So, I'll work on a way to address your issue, but it's not going to be "all
>> stateless."  There will be things in the stateful APIs that will not exist
>> in the stateless ones.  I'll also ping Sam Pullara about wether YQL is
>> totally stateless or whether it's possible to have session cookies as part
>> of a request set.
>>
>> Thanks,
>>
>> David
>>
>>
>>     
>>> Thanks,
>>> Ethan
>>>
>>>       
>>
>> --
>> Lift, the simply functional web framework http://liftweb.net
>> Beginning Scala http://www.apress.com/book/view/1430219890
>> Follow me: http://twitter.com/dpp
>> Git some: http://github.com/dpp
>>
>>     

Re: Current REST-API - Sessions

[Ethan Jewett <es...@gmail.com>]

Hi David,

Thanks for looking into this. I definitely agree that we need state on
some calls. Partly I'm curious as to if sessions are the best way to
do this, but I'm good with whatever we have and I'm aware of the
resource constraints.

I've been wondering if there is a good way to provide optional state
through sessions as well as a stateless mode for some calls. Is this
what you mean by filters?

I'll see if I can get in touch with Sam on this topic as well, as he's
offered help to me on other YQL interfaces I've worked on.

Ethan

On Mon, Aug 3, 2009 at 9:02 AM, David
Pollak<fe...@gmail.com> wrote:
> On Sun, Aug 2, 2009 at 8:45 AM, Ethan Jewett <es...@gmail.com> wrote:
>
>> Hi all,
>>
>> Last week I said I was going to start working on a YQL-wrapper for the
>> ESME API, hoping that my work in that area would start to drive API
>> development again. (After the whole RPC vs. REST uproar we ended up
>> with one blog post, one page on the wiki at
>> http://incubator.apache.org/esme/restapi.html and that was it, which
>> was mostly my fault for losing interest for various reasons.)
>>
>> Today I started and almost immediately ran into the requirement of the
>> current API to sustain a session in the client. I know it would be
>> possible to simulate this by manually creating the appropriate headers
>> in Javascript, however I don't think this is a reasonable approach
>> with YQL as YQL itself has no mechanism to allow storing a session key
>> across requests, so session key storage would have to be managed by
>> the YQL client, and then the session key passed in the YQL request.
>>
>> I'd like to revisit the use of sessions in the API. I do not know
>> Lift, but my understanding is that we gain some ease of use in the
>> scenario of interfaces built on top of the API using Lift because of
>> its automatic handling of sessions. Are there other reasons?
>>
>> I'd like to understand all the reasons for this approach so that we
>> can figure out if there is an alternate way to handle this that is
>> more in line with the way web APIs are programmed these days (and
>> subsequently will hopefully be more useful with web API interaction
>> tools like YQL work).
>
>
> There are some ESME APIs for which stateless/"RESTful" is appropriate
> (things that refer to entities that are stable... basically things that map
> to RDBMS calls) and things where stateless is not appropriate (things that
> refer to event streams.)  There are clients for which stateless is
> appropriate (in this case YQL) and clients for which stateless is not as
> appropriate (i.e., UI clients, web based or desktop based.)
>
> We have limited resources and I don't want to have duplication API
> implementations (or even filters) for stateful and stateless because that's
> just more work for us, more stuff for us to maintain, and this is the first
> use case for actually having stateless that's appeared on this list.
>
> So, I'll work on a way to address your issue, but it's not going to be "all
> stateless."  There will be things in the stateful APIs that will not exist
> in the stateless ones.  I'll also ping Sam Pullara about wether YQL is
> totally stateless or whether it's possible to have session cookies as part
> of a request set.
>
> Thanks,
>
> David
>
>
>>
>>
>> Thanks,
>> Ethan
>>
>
>
>
> --
> Lift, the simply functional web framework http://liftweb.net
> Beginning Scala http://www.apress.com/book/view/1430219890
> Follow me: http://twitter.com/dpp
> Git some: http://github.com/dpp
>

Re: Current REST-API - Sessions

[David Pollak <fe...@gmail.com>]

On Sun, Aug 2, 2009 at 8:45 AM, Ethan Jewett <es...@gmail.com> wrote:

> Hi all,
>
> Last week I said I was going to start working on a YQL-wrapper for the
> ESME API, hoping that my work in that area would start to drive API
> development again. (After the whole RPC vs. REST uproar we ended up
> with one blog post, one page on the wiki at
> http://incubator.apache.org/esme/restapi.html and that was it, which
> was mostly my fault for losing interest for various reasons.)
>
> Today I started and almost immediately ran into the requirement of the
> current API to sustain a session in the client. I know it would be
> possible to simulate this by manually creating the appropriate headers
> in Javascript, however I don't think this is a reasonable approach
> with YQL as YQL itself has no mechanism to allow storing a session key
> across requests, so session key storage would have to be managed by
> the YQL client, and then the session key passed in the YQL request.
>
> I'd like to revisit the use of sessions in the API. I do not know
> Lift, but my understanding is that we gain some ease of use in the
> scenario of interfaces built on top of the API using Lift because of
> its automatic handling of sessions. Are there other reasons?
>
> I'd like to understand all the reasons for this approach so that we
> can figure out if there is an alternate way to handle this that is
> more in line with the way web APIs are programmed these days (and
> subsequently will hopefully be more useful with web API interaction
> tools like YQL work).


There are some ESME APIs for which stateless/"RESTful" is appropriate
(things that refer to entities that are stable... basically things that map
to RDBMS calls) and things where stateless is not appropriate (things that
refer to event streams.)  There are clients for which stateless is
appropriate (in this case YQL) and clients for which stateless is not as
appropriate (i.e., UI clients, web based or desktop based.)

We have limited resources and I don't want to have duplication API
implementations (or even filters) for stateful and stateless because that's
just more work for us, more stuff for us to maintain, and this is the first
use case for actually having stateless that's appeared on this list.

So, I'll work on a way to address your issue, but it's not going to be "all
stateless."  There will be things in the stateful APIs that will not exist
in the stateless ones.  I'll also ping Sam Pullara about wether YQL is
totally stateless or whether it's possible to have session cookies as part
of a request set.

Thanks,

David


>
>
> Thanks,
> Ethan
>



-- 
Lift, the simply functional web framework http://liftweb.net
Beginning Scala http://www.apress.com/book/view/1430219890
Follow me: http://twitter.com/dpp
Git some: http://github.com/dpp