API 2.0 update and design questions

[Ethan Jewett <es...@gmail.com>]

Hi all,
This email is serving two purposes. One is to ping the list to let you all
know that I'm making more updates to the API 2.0 design page on the cwiki
at http://cwiki.apache.org/confluence/display/ESME/REST+API+2.0+-+Design  It
now includes reference to the need for real streaming interfaces to parts of
the API.

The second purpose is to surface some of the questions that we're facing at
the moment. Questions are outlined below.

I do first want to note that this API is strictly vapor-ware/-design at the
moment, so it's quite possible that the next step for me here is to learn
Scala and Lift. If anyone else with a few more Scala chops is interested in
starting work on this with me immediately though, then by all means let's
get started. :-)

In the meantime, there are open items that I think need to be resolved to
move ahead. I'm listing them here along with my *opinion* to hopefully
facilitate some discussion.

1. REST - I'm thinking of just dropping "REST" from the name of the API. It
should still be designed with RESTful principles in mind where appropriate
but it's probably time we move past this issue as a group and dropping the
term from the API title might be a good step in this direction.

2. Authorization - I think the current authorization scheme using tokens is
insecure over non-encrypted connections since the token is passed in the
clear. It also uses sessions, which are nice in some contexts and not in
others. Food for thought. I'm not changing anything here at the moment, but
it's on my mind and I don't think it's currently satisfactory. At the risk
of getting my head bitten off, my thought process here still tends towards
the message-signing half of the OAuth spec to guide our authorization
approach, but I'm not particularly devoted to that idea if there are better
ones, especially since OAuth is not proven on non-native-HTTP protocols (see
streaming below).

3. Streaming - I think there is a major question around how we stream the
parts of the API that require streaming (messages specifically). Does ESME
do this currently? My research seems to indicate that AMQP is probably the
most performant and interoperable way to accomplish streaming (though JMS
should be reasonable as well, if not as inter-operable in non-enterprise
environments). I also seem to see that Lift has built-in AMQP support. Does
ESME already support AMQP pub/sub? If so, then I think we can probably just
document this support and call it a day. Other options include XMPP, and
various streaming and pub/sub approaches over HTTP.

4. HTTP fallback for clients that do not support streaming - For clients
that don't interoperate with our chosen streaming mechanism (whatever that
may be), will we offer one or more HTTP-based fallbacks? The simplest
fallback would be a poll-able Atom feed. I think we all agree this is not
good, but it could work. More advanced fallbacks might be long-polling,
Reverse HTTP, or PubSubHubBub. Please see the wiki page for links. I think
we need to offer some HTTP-based capability that is widely-compatible for
those clients that don't have access to network resources below the HTTP
level, but I'd like to know what you think and if you have a preference.

That's probably enough for now. I'd really like to know what you all think
and how you'd like me to proceed if you have a preference.

Thanks,
Ethan

Re: API 2.0 update and design questions

[Daniel Koller <da...@googlemail.com>]

Vassil Dichev schrieb:
>> @Daniel - maybe Vassil can help you there. What exactly was your problem?
>>     
>
> @Daniel- I realize now we intended to make the message parser
> pluggable, but then there was always something which seemed more
> important. I think this is the right time to get back to this issue.
> The problem here is that many of the higher-level parser combinators
> use common low-level combinators, i.e. email address, identifier
> characters (for usernames, pool names, etc.). In order to reuse these,
> it's a good idea to separate and document the low-level parser
> combinators we can rely on (like an API).
>   
The issue was to find whether/how I could reuse the patterns used to
make it recognizable through the pattern.

Another way would be (in Java terms) to let each plugin via a specific
check on its own how it wants to consume message content from esme.

Do we see a flexible MsgParser solution any time soon or should I check
with Vassil on how to hardwire amqp-support?

Kind regards,

Daniel

Re: API 2.0 update and design questions

[Ethan Jewett <es...@gmail.com>]

On Wed, Aug 26, 2009 at 2:25 PM, Vassil Dichev <vd...@apache.org> wrote:
>
>
> I see, those technologies you were listing were just possible options,
> and not a list of things we should implement.
>
> Lift does offer integration with AMQP and we were discussing on the
> ESME mailing list about using that. I also think there might be value
> in Atom since it offers the easiest way for a client to check which
> messages are new (couple of lines of Scala code). As for real-time
> updates over HTTP, the way to go is to use an abstraction like the
> Lift CometActor, so whenever there's a better technology than long
> polling out there, the CometActor can be reimplemented in it and no
> changes in ESME are necessary whatsoever. In fact, if memory serves me
> right, David has mentioned that Comet would be reimplemented in
> Reverse HTTP when it matures a bit.
>

 That's very interesting and could be either good or bad depending on the
situation. If we are building an integrated web application based on Lift
(which we are in the case of ESME) then I really like the auto-upgrade
capability delivered with the CometActor. If we are building an API (which
we are also doing), then I would think we definitely *don't* want the API to
auto-upgrade like you describe. Something like switching from the current
long-polling to reverse-http would break all existing clients and would
require something like a new version of the API and a period where the two
APIs ran in parallel. Would that be possible using the Lift CometActor?

Ethan

Re: API 2.0 update and design questions

[Vassil Dichev <vd...@apache.org>]

> Maybe I should have grouped questions 3 & 4 together. What I'm interested in
> are points of view on what streaming approach we should be using for the
> parts of the API that need to be streamed.
>
> I do think that all information in ESME should be accessible via HTTP, and I
> think long-polling is a good way to accomplish that for things like message
> queues. It's especially good since it already exists. In my understanding,
> long-polling is essentially a short-lived subscription that requires an open
> connection. I also threw out a few other options for pub/sub over HTTP, to
> fill out the field.
>
> My concern, and suspicion, is just that HTTP is not the right protocol to be
> using for this type of streaming of message queues. I'm thinking that a
> protocol like AMQP may be better suited. If we go with something like AMQP
> (for example), then I want to make sure that an HTTP method of access to the
> same messages (and hopefully the same queue, if possible) is still
> available.
>
> Thoughts?

I see, those technologies you were listing were just possible options,
and not a list of things we should implement.

Lift does offer integration with AMQP and we were discussing on the
ESME mailing list about using that. I also think there might be value
in Atom since it offers the easiest way for a client to check which
messages are new (couple of lines of Scala code). As for real-time
updates over HTTP, the way to go is to use an abstraction like the
Lift CometActor, so whenever there's a better technology than long
polling out there, the CometActor can be reimplemented in it and no
changes in ESME are necessary whatsoever. In fact, if memory serves me
right, David has mentioned that Comet would be reimplemented in
Reverse HTTP when it matures a bit.

Vassil

Re: API 2.0 update and design questions

[Ethan Jewett <es...@gmail.com>]

Hi Vassil,
Maybe I should have grouped questions 3 & 4 together. What I'm interested in
are points of view on what streaming approach we should be using for the
parts of the API that need to be streamed.

I do think that all information in ESME should be accessible via HTTP, and I
think long-polling is a good way to accomplish that for things like message
queues. It's especially good since it already exists. In my understanding,
long-polling is essentially a short-lived subscription that requires an open
connection. I also threw out a few other options for pub/sub over HTTP, to
fill out the field.

My concern, and suspicion, is just that HTTP is not the right protocol to be
using for this type of streaming of message queues. I'm thinking that a
protocol like AMQP may be better suited. If we go with something like AMQP
(for example), then I want to make sure that an HTTP method of access to the
same messages (and hopefully the same queue, if possible) is still
available.

Thoughts?

Ethan

On Wed, Aug 26, 2009 at 12:19 PM, Vassil Dichev <vd...@apache.org> wrote:

> > @Daniel - maybe Vassil can help you there. What exactly was your problem?
>
> @Daniel- I realize now we intended to make the message parser
> pluggable, but then there was always something which seemed more
> important. I think this is the right time to get back to this issue.
> The problem here is that many of the higher-level parser combinators
> use common low-level combinators, i.e. email address, identifier
> characters (for usernames, pool names, etc.). In order to reuse these,
> it's a good idea to separate and document the low-level parser
> combinators we can rely on (like an API).
>
> @Ethan- Regarding "streaming", we already have long-polling that's
> built in Lift with the Comet actors. Using Atom is OK, but it's
> certainly serves a different purpose, since it's not real-time. For
> reverse HTTP you'll have to wait a while before we can even discuss
> it, since it's just in the specification draft phase. PubSubHubBub is
> also quite new, and I don't see what more it offers to what we already
> have with the comet actors. All in all, it's a bit confusing for me to
> see all those different protocols and technologies together- they
> serve slightly different purposes, and are in varying stages of
> maturity. So, again- what is the use case we want that we don't
> already have?
>
> I'll be more than happy to help with whatever I've learned about
> Scala- and Lift.
>
> Vassil
>

Re: API 2.0 update and design questions

[Vassil Dichev <vd...@apache.org>]

> @Daniel - maybe Vassil can help you there. What exactly was your problem?

@Daniel- I realize now we intended to make the message parser
pluggable, but then there was always something which seemed more
important. I think this is the right time to get back to this issue.
The problem here is that many of the higher-level parser combinators
use common low-level combinators, i.e. email address, identifier
characters (for usernames, pool names, etc.). In order to reuse these,
it's a good idea to separate and document the low-level parser
combinators we can rely on (like an API).

@Ethan- Regarding "streaming", we already have long-polling that's
built in Lift with the Comet actors. Using Atom is OK, but it's
certainly serves a different purpose, since it's not real-time. For
reverse HTTP you'll have to wait a while before we can even discuss
it, since it's just in the specification draft phase. PubSubHubBub is
also quite new, and I don't see what more it offers to what we already
have with the comet actors. All in all, it's a bit confusing for me to
see all those different protocols and technologies together- they
serve slightly different purposes, and are in varying stages of
maturity. So, again- what is the use case we want that we don't
already have?

I'll be more than happy to help with whatever I've learned about
Scala- and Lift.

Vassil

Re: API 2.0 update and design questions

[Richard Hirsch <hi...@gmail.com>]

@Daniel - maybe Vassil can help you there. What exactly was your problem?

D.

On Mon, Aug 24, 2009 at 10:02 PM, Daniel Koller<da...@googlemail.com> wrote:
> Hi all,
>
> please see my feedback below,
>
> On Mon, Aug 24, 2009 at 1:52 AM, Ethan Jewett <es...@gmail.com> wrote:
>>
>>
>> In the meantime, there are open items that I think need to be resolved to
>> move ahead. I'm listing them here along with my *opinion* to hopefully
>> facilitate some discussion.
>>
>> 1. REST - I'm thinking of just dropping "REST" from the name of the API. It
>> should still be designed with RESTful principles in mind where appropriate
>> but it's probably time we move past this issue as a group and dropping the
>> term from the API title might be a good step in this direction.
>
> +1
>
>>
>> 3. Streaming - I think there is a major question around how we stream the
>> parts of the API that require streaming (messages specifically). Does ESME
>> do this currently? My research seems to indicate that AMQP is probably the
>> most performant and interoperable way to accomplish streaming (though JMS
>> should be reasonable as well, if not as inter-operable in non-enterprise
>> environments). I also seem to see that Lift has built-in AMQP support. Does
>> ESME already support AMQP pub/sub? If so, then I think we can probably just
>> document this support and call it a day. Other options include XMPP, and
>> various streaming and pub/sub approaches over HTTP.
>
>
> I created test-wise an actor some weeks ago, which sends out messages to an
> AMQP infrastructure as a part of an action: it was somehow complicated to
> insert this new Sender in the MsgParser.scala, so integration was hardcoded
> and not very flexible.
> --> if somebody could lend a hand with MsgParser-Integration and
> parametrization, this could get part of the ESME code.
>
> Thanks for putting these topics together,
>
> Daniel
>

Re: API 2.0 update and design questions

[Daniel Koller <da...@googlemail.com>]

Hi all,

please see my feedback below,

On Mon, Aug 24, 2009 at 1:52 AM, Ethan Jewett <es...@gmail.com> wrote:
>
>
> In the meantime, there are open items that I think need to be resolved to
> move ahead. I'm listing them here along with my *opinion* to hopefully
> facilitate some discussion.
>
> 1. REST - I'm thinking of just dropping "REST" from the name of the API. It
> should still be designed with RESTful principles in mind where appropriate
> but it's probably time we move past this issue as a group and dropping the
> term from the API title might be a good step in this direction.

+1

>
> 3. Streaming - I think there is a major question around how we stream the
> parts of the API that require streaming (messages specifically). Does ESME
> do this currently? My research seems to indicate that AMQP is probably the
> most performant and interoperable way to accomplish streaming (though JMS
> should be reasonable as well, if not as inter-operable in non-enterprise
> environments). I also seem to see that Lift has built-in AMQP support. Does
> ESME already support AMQP pub/sub? If so, then I think we can probably just
> document this support and call it a day. Other options include XMPP, and
> various streaming and pub/sub approaches over HTTP.


I created test-wise an actor some weeks ago, which sends out messages to an
AMQP infrastructure as a part of an action: it was somehow complicated to
insert this new Sender in the MsgParser.scala, so integration was hardcoded
and not very flexible.
--> if somebody could lend a hand with MsgParser-Integration and
parametrization, this could get part of the ESME code.

Thanks for putting these topics together,

Daniel