[DISCUSS] Switching Streams from Camel deployment to .war deployment

[Danny Sullivan <ds...@hotmail.com>]

The discussion thread for switching Streams from Camel/osgi/Servicemix to a single .war deployment 		 	   		  

Re: [DISCUSS] Switching Streams from Camel deployment to .war deployment

[Matt Franklin <m....@gmail.com>]

On Thu, Oct 31, 2013 at 5:00 PM, Chris Geer <ch...@cxtsoftware.com> wrote:

> On Thu, Oct 31, 2013 at 1:47 PM, Matt Franklin <m.ben.franklin@gmail.com
> >wrote:
>
> > >
> > > Matt, I understand what you are trying to do here but again, big
> picture
> > > I'm sure sure it's this clean cut. Maybe it is for simple
> implementations
> > > but I would think larger system would require a combination of things
> > like
> > > Storm and Camel working together.
> > >
> > >
> > Possibly.  The main point I was trying to get across is that we need a
> core
> > way to define:
> >
> > * An over-arching workflow for processing
> > * Boundary message formats
> > * Utilities and common models
> > * Interfaces for workflow steps that can be implemented via whatever
> > mechanism suites the solution best
> >
> > In the case where you are processing millions of items a day, this
> > orchestration layer is the single biggest value add of the project IMO.
> >
>
> I agree. My opinion is that the message formats (xml/json - not java
> interface) are the most important piece. If that can be nailed down the
> rest will come together.
>
> For example we know that the system should be able to ingest and output in
> the activitystrea.ms format for starters. But we also wanted to handle
> things like aggregation, so what does an aggregate message look like? What
> other message formats do we need to consider?
>

I think a lot can be handled by the AS format using upstream duplicates or
custom extensions.  Beyond that, we need a few schemas for formats of
metrics, etc.


>
> Chris
>

Re: [DISCUSS] Switching Streams from Camel deployment to .war deployment

[Chris Geer <ch...@cxtsoftware.com>]

On Thu, Oct 31, 2013 at 1:47 PM, Matt Franklin <m....@gmail.com>wrote:

> >
> > Matt, I understand what you are trying to do here but again, big picture
> > I'm sure sure it's this clean cut. Maybe it is for simple implementations
> > but I would think larger system would require a combination of things
> like
> > Storm and Camel working together.
> >
> >
> Possibly.  The main point I was trying to get across is that we need a core
> way to define:
>
> * An over-arching workflow for processing
> * Boundary message formats
> * Utilities and common models
> * Interfaces for workflow steps that can be implemented via whatever
> mechanism suites the solution best
>
> In the case where you are processing millions of items a day, this
> orchestration layer is the single biggest value add of the project IMO.
>

I agree. My opinion is that the message formats (xml/json - not java
interface) are the most important piece. If that can be nailed down the
rest will come together.

For example we know that the system should be able to ingest and output in
the activitystrea.ms format for starters. But we also wanted to handle
things like aggregation, so what does an aggregate message look like? What
other message formats do we need to consider?

Chris

Re: [DISCUSS] Switching Streams from Camel deployment to .war deployment

[Matt Franklin <m....@gmail.com>]

>
> Matt, I understand what you are trying to do here but again, big picture
> I'm sure sure it's this clean cut. Maybe it is for simple implementations
> but I would think larger system would require a combination of things like
> Storm and Camel working together.
>
>
Possibly.  The main point I was trying to get across is that we need a core
way to define:

* An over-arching workflow for processing
* Boundary message formats
* Utilities and common models
* Interfaces for workflow steps that can be implemented via whatever
mechanism suites the solution best

In the case where you are processing millions of items a day, this
orchestration layer is the single biggest value add of the project IMO.

RE: [DISCUSS] Switching Streams from Camel deployment to .war deployment

[Danny Sullivan <ds...@hotmail.com>]

I don't believe that urls are able to be reused when restarting the application, at least in my experience, though I could be wrong. Do you have steps to reproduce starting and stoping a server while maintaining the same url? Also note, that stopping the .war would only lose activities/publishers/subscriptions currently being processed, not the ones successfully stored in the database.

I'm not sure I follow you on focusing on message formats as opposed to interfaces. Could you explain a little more?

I agree that it will not involve something such as a Camel implementation, a Storm implementation, and a Web implementation. I originally imagined the Spring controller acting as an entry point to Storm, but perhaps that remains to be seen.
> Date: Wed, 30 Oct 2013 09:50:38 -0700
> Subject: Re: [DISCUSS] Switching Streams from Camel deployment to .war deployment
> From: chris@cxtsoftware.com
> To: dev@streams.incubator.apache.org
> 
> Ok, lots of thoughts in-line.
> 
> On Wed, Oct 30, 2013 at 8:37 AM, Danny Sullivan <ds...@hotmail.com>wrote:
> 
> > My argument is not for the IoC pattern as that can be (and has been)
> > implemented alongside Camel. My main argument is that the syntax at the
> > entry point is not only familiar but much simpler. This wouldn't be a very
> > strong argument if the Camel implementation wasn't much more complicated
> > but I feel that it is the case. Also, looking toward the future, if the
> > server is restarted, in-routes are lost in Camel.
> 
> 
> Yes, just like if you restarted a server your in-memory processing in a web
> app would be lost. The way to solve this is to use a storage mechanism to
> store state/messages. I believe the current implementation does that with
> ActiveMQ which provides both disk and DB storage of messages.
> 
> 
> > The way to curb this is to persist the dynamic routes that Camel creates,
> > and then on start up pull every one of these routes and recreate a dynamic
> > route for each one. Not only is this much easier to implement using the
> > Spring web implementation, but it already has been implemented and you can
> > try it by checking out the webservice branch, registering a subscriber,
> > restarting tomcat, and using the same url you had before. This will allow
> > subscribers to hang on to their urls once they register. (the same is true
> > for publishers: you can post via the same url after restarting tomcat)
> >
> > > Date: Wed, 30 Oct 2013 11:00:21 -0400
> > > Subject: Re: [DISCUSS] Switching Streams from Camel deployment to .war
> > deployment
> > > From: jletourneau80@gmail.com
> > > To: dev@streams.incubator.apache.org
> > >
> > > To be fair, while the current implementation is heavily camel-based,
> > > all of the interfaces related to Streams functionality are not.  The
> > > current model maps to what Matt has outlined in my opinion, though
> > > packing names etc. probably don't follow that exact pattern.
> >
> 
> One of the problems I have with the current implementation (both of them)
> are the interfaces. In my opinion, to make this work at scale we need to
> forget about Java interfaces and focus on message formats. Sure there needs
> to be an API to get data in and data out but in between you can scale a
> message based system (Storm or Camel/ActiveMQ) easier than an RPC system
> (i.e. Java method calls).
> 
> > >
> > > With regards to the complexity and different components in the
> > > registration process, this was a cut at the abstraction based on the
> > > assumption that different implementations may be plugged in and in
> > > fact may live on different processor space (ie. a polling publisher vs
> > > a push publisher may be instantiated on different servers but the
> > > registration URL is staticly defined).
> > >
> > > Is the main argument I am seeing for Spring  the familiarity of its
> > > IoC pattern implementation and syntax at the entry point?
> > >
> > > On Wed, Oct 30, 2013 at 10:53 AM, Danny Sullivan <ds...@hotmail.com>
> > wrote:
> > > > Could you clarify whether the same entry points would exist for the
> > camel implementation of the core (implementing the "process" method/ using
> > a DynammicRouteBuilder) or would the webservice be the sole entry point to
> > Streams and after it enters would it hand it off to Camel? And what would
> > be the entry point for the Storm implementation?
> > > > -Danny
> > > >
> > > >> Date: Wed, 30 Oct 2013 10:13:04 -0400
> > > >> Subject: Re: [DISCUSS] Switching Streams from Camel deployment to
> > .war deployment
> > > >> From: m.ben.franklin@gmail.com
> > > >> To: dev@streams.incubator.apache.org
> > > >>
> > > >> On Tue, Oct 29, 2013 at 2:55 PM, Danny Sullivan <
> > dsullivan7@hotmail.com>wrote:
> > > >>
> > > >> > My case for switching from OSGi is for simplicity in design. To
> > follow the
> > > >> > path of an activity through Streams in the webservice, there is one
> > main
> > > >> > things the developer needs to understand:
> > > >> > The @RequestMapping annotation specifies the HTTP entry point
> > > >> > There are 4 @RequestMapping annotations that correspond to each of
> > the 4
> > > >> > ways a user enters the application: registering a publisher,
> > registering a
> > > >> > subscriber, publishing activity, and getting an activity stream.
> > Where
> > > >> > these are located in the source code can be found by searching for
> > the
> > > >> > paths specified in the documentation (search for
> > "/publisherRegister",
> > > >> > "/subscriberRegister", "/publishActivity", "/getActivity" which
> > will all
> > > >> > lead you to StreamsWebController.java). From the methods that
> > process
> > > >> > requests, the flow through the application is through methods which
> > can be
> > > >> > understood by most Java programmers.
> > > >> > The flow of activities through the current trunk branch is
> > understood as
> > > >> > follows:
> > > >> > The string "/publisher/register" (the entry point to register a
> > publisher
> > > >> > specified in the documentation) is the value of the
> > > >> > consumer.registrationEndpoint property defined in
> > streams.propertiesThe
> > > >> > camelContext.xml specifies an endpoint with the id
> > > >> > consumerRegistrationEndpoint to have a uri equal to the propertyThe
> > > >> > consumerRegistrationEndpoint routes from uri direct:publisher
> > register with
> > > >> > the bean activityRegistrationProcessor nested in between the
> > routeThe
> > > >> > streams-eip-applicationContext contains a bean with the id
> > > >> > activityRegistrationProcessor created for the class
> > > >> > ActivityPublisherRegistrationProcessorThe exchange will enter the
> > "process"
> > > >> > method of the class ActivityPublisherRegisitrationProcessor and
> > that this
> > > >> > is because this class implements the "Processor" interface provided
> > by
> > > >> > CamelThe direct:add-publisher-route takes the exchange output from
> > the
> > > >> > "process" method and routes it to the activityRegistrationProcessor
> > > >> > "register" method. The bean activityRegistrationProcessor is
> > defined in the
> > > >> > streams-eip-osgi-component-import.xmlThe output from this method is
> > then
> > > >> > sent to the "createNewRouteForConsumer" method of
> > activityConsumerRouter.
> > > >> > This method creates a new route for the newly registered publisher
> > using
> > > >> > the private static final class DynamicConsumerRouteBuilder which is
> > > >> > required to extend RouteBuilder which is provided by Camel. This
> > > >> > DynamicConsumerRouteBuilder contains several methods:
> > > >> > getConsumerReceiveMethod() (which corresponds to @Value
> > > >> > ${consumer.receiveMethod} which corresponds to "receive"),
> > > >> > getConsumerSplitMethod() (@Value ${consumer.splitMethod},"split"),
> > and
> > > >> > getConsumerActivityQUri() (@Value ${consumerActivityQUri},
> > > >> > "direct:activityQ"). This is different than the camelContext.xml in
> > that
> > > >> > the route is being created programmatically. What this is doing is
> > routing
> > > >> > input from the inroute url (which Camel does automatically through
> > the
> > > >> > configure method which is required to be overridden), to the
> > "receive"
> > > >> > method of ActivityConsumer, then to the "split" method of
> > ActivityConsumer,
> > > >> > and then to the "direct:activityQ" which if you look back in the
> > > >> > camelContext.xml routes to the "activemq:queue:activities" which
> > then
> > > >> > routes to "receiveExchange"
> > > >> > This is the process to register a publisher. The process for
> > registering a
> > > >> > subscriber is relatively the same though it involves separate
> > classes with
> > > >> > their own private static final RouteBuilder class. From my
> > perspective, the
> > > >> > two most difficult things with setting this project up were
> > understanding
> > > >> > that the "process" method of the class that implements "Processor"
> > is the
> > > >> > entry point and the DynamicConsumerRouteBuilder creates the second
> > entry
> > > >> > point (The 5th and last points). This made the project very, VERY
> > hard to
> > > >> > understand.
> > > >> > In addition to simplicity of design, the mvn clean install of the
> > web
> > > >> > service project is much faster and small scale activity publishing
> > is also
> > > >> > faster (see my email about load testing). These are minor points
> > though as
> > > >> > compilation has no effect on deployment. OSGi does add the benefit
> > of
> > > >> > modularized programming which is valuable, though I think the added
> > > >> > complexity of Camel merits moving the project away from this
> > paradigm.
> >
> 
> Camel != OSGI. Camel can run just fine inside a WAR and you can have an
> OSGI based application without Camel. OSGI is a deployment model where you
> have well defined interfaces between components and isolation. It's more of
> a competitor with Spring although you can run Spring components in an OSGI
> server. In fact Spring DM is now Eclipse Gemini which is an OSGI Blueprint
> implementation.
> 
> 
> > > >> >
> > > >>
> > > >> I agree that the project is pretty difficult to understand ATM.  I
> > think
> > > >> what we need to do is think about what the responsibilities of the
> > code are
> > > >> and allow for different implementations that are not so tightly
> > coupled as
> > > >> they are now.  For instance, having worked with Storm to ingest
> > millions of
> > > >> activities a day, I personally would like to see streams be
> > responsible for
> > > >> defining an over-arching orchestration model that can be implemented
> > within
> > > >> a single war or on top of a distributed system.  This would look
> > something
> > > >> like the follows
> >
> > >>
> > > >> |__ Streams-Core  (Base classes, interfaces, utilities, extensions,
> > etc)
> > > >> |
> > > >> |__ Streams-Storm (Storm implementation of the core)
> > > >> |
> > > >> |__ Streams-Camel (Camel implementation of the core)
> > > >> |
> > > >> |__ Streams-WS (Web service implementation)
> >
> 
> Matt, I understand what you are trying to do here but again, big picture
> I'm sure sure it's this clean cut. Maybe it is for simple implementations
> but I would think larger system would require a combination of things like
> Storm and Camel working together.
> 
> 
> > > >>
> > > >>
> > > >> > Danny
> > > >> >
> > > >> >
> > > >> > > From: dsullivan7@hotmail.com
> > > >> > > To: dev@streams.incubator.apache.org
> > > >> > > Subject: [DISCUSS] Switching Streams from Camel deployment to .war
> > > >> > deployment
> > > >> > > Date: Tue, 29 Oct 2013 11:07:39 -0400
> > > >> > >
> > > >> > > The discussion thread for switching Streams from
> > Camel/osgi/Servicemix
> > > >> > to a single .war deployment
> > > >> >
> > > >> >
> > > >
> >
> >
 		 	   		  

Re: [DISCUSS] Switching Streams from Camel deployment to .war deployment

[Chris Geer <ch...@cxtsoftware.com>]

Ok, lots of thoughts in-line.

On Wed, Oct 30, 2013 at 8:37 AM, Danny Sullivan <ds...@hotmail.com>wrote:

> My argument is not for the IoC pattern as that can be (and has been)
> implemented alongside Camel. My main argument is that the syntax at the
> entry point is not only familiar but much simpler. This wouldn't be a very
> strong argument if the Camel implementation wasn't much more complicated
> but I feel that it is the case. Also, looking toward the future, if the
> server is restarted, in-routes are lost in Camel.


Yes, just like if you restarted a server your in-memory processing in a web
app would be lost. The way to solve this is to use a storage mechanism to
store state/messages. I believe the current implementation does that with
ActiveMQ which provides both disk and DB storage of messages.


> The way to curb this is to persist the dynamic routes that Camel creates,
> and then on start up pull every one of these routes and recreate a dynamic
> route for each one. Not only is this much easier to implement using the
> Spring web implementation, but it already has been implemented and you can
> try it by checking out the webservice branch, registering a subscriber,
> restarting tomcat, and using the same url you had before. This will allow
> subscribers to hang on to their urls once they register. (the same is true
> for publishers: you can post via the same url after restarting tomcat)
>
> > Date: Wed, 30 Oct 2013 11:00:21 -0400
> > Subject: Re: [DISCUSS] Switching Streams from Camel deployment to .war
> deployment
> > From: jletourneau80@gmail.com
> > To: dev@streams.incubator.apache.org
> >
> > To be fair, while the current implementation is heavily camel-based,
> > all of the interfaces related to Streams functionality are not.  The
> > current model maps to what Matt has outlined in my opinion, though
> > packing names etc. probably don't follow that exact pattern.
>

One of the problems I have with the current implementation (both of them)
are the interfaces. In my opinion, to make this work at scale we need to
forget about Java interfaces and focus on message formats. Sure there needs
to be an API to get data in and data out but in between you can scale a
message based system (Storm or Camel/ActiveMQ) easier than an RPC system
(i.e. Java method calls).

> >
> > With regards to the complexity and different components in the
> > registration process, this was a cut at the abstraction based on the
> > assumption that different implementations may be plugged in and in
> > fact may live on different processor space (ie. a polling publisher vs
> > a push publisher may be instantiated on different servers but the
> > registration URL is staticly defined).
> >
> > Is the main argument I am seeing for Spring  the familiarity of its
> > IoC pattern implementation and syntax at the entry point?
> >
> > On Wed, Oct 30, 2013 at 10:53 AM, Danny Sullivan <ds...@hotmail.com>
> wrote:
> > > Could you clarify whether the same entry points would exist for the
> camel implementation of the core (implementing the "process" method/ using
> a DynammicRouteBuilder) or would the webservice be the sole entry point to
> Streams and after it enters would it hand it off to Camel? And what would
> be the entry point for the Storm implementation?
> > > -Danny
> > >
> > >> Date: Wed, 30 Oct 2013 10:13:04 -0400
> > >> Subject: Re: [DISCUSS] Switching Streams from Camel deployment to
> .war deployment
> > >> From: m.ben.franklin@gmail.com
> > >> To: dev@streams.incubator.apache.org
> > >>
> > >> On Tue, Oct 29, 2013 at 2:55 PM, Danny Sullivan <
> dsullivan7@hotmail.com>wrote:
> > >>
> > >> > My case for switching from OSGi is for simplicity in design. To
> follow the
> > >> > path of an activity through Streams in the webservice, there is one
> main
> > >> > things the developer needs to understand:
> > >> > The @RequestMapping annotation specifies the HTTP entry point
> > >> > There are 4 @RequestMapping annotations that correspond to each of
> the 4
> > >> > ways a user enters the application: registering a publisher,
> registering a
> > >> > subscriber, publishing activity, and getting an activity stream.
> Where
> > >> > these are located in the source code can be found by searching for
> the
> > >> > paths specified in the documentation (search for
> "/publisherRegister",
> > >> > "/subscriberRegister", "/publishActivity", "/getActivity" which
> will all
> > >> > lead you to StreamsWebController.java). From the methods that
> process
> > >> > requests, the flow through the application is through methods which
> can be
> > >> > understood by most Java programmers.
> > >> > The flow of activities through the current trunk branch is
> understood as
> > >> > follows:
> > >> > The string "/publisher/register" (the entry point to register a
> publisher
> > >> > specified in the documentation) is the value of the
> > >> > consumer.registrationEndpoint property defined in
> streams.propertiesThe
> > >> > camelContext.xml specifies an endpoint with the id
> > >> > consumerRegistrationEndpoint to have a uri equal to the propertyThe
> > >> > consumerRegistrationEndpoint routes from uri direct:publisher
> register with
> > >> > the bean activityRegistrationProcessor nested in between the
> routeThe
> > >> > streams-eip-applicationContext contains a bean with the id
> > >> > activityRegistrationProcessor created for the class
> > >> > ActivityPublisherRegistrationProcessorThe exchange will enter the
> "process"
> > >> > method of the class ActivityPublisherRegisitrationProcessor and
> that this
> > >> > is because this class implements the "Processor" interface provided
> by
> > >> > CamelThe direct:add-publisher-route takes the exchange output from
> the
> > >> > "process" method and routes it to the activityRegistrationProcessor
> > >> > "register" method. The bean activityRegistrationProcessor is
> defined in the
> > >> > streams-eip-osgi-component-import.xmlThe output from this method is
> then
> > >> > sent to the "createNewRouteForConsumer" method of
> activityConsumerRouter.
> > >> > This method creates a new route for the newly registered publisher
> using
> > >> > the private static final class DynamicConsumerRouteBuilder which is
> > >> > required to extend RouteBuilder which is provided by Camel. This
> > >> > DynamicConsumerRouteBuilder contains several methods:
> > >> > getConsumerReceiveMethod() (which corresponds to @Value
> > >> > ${consumer.receiveMethod} which corresponds to "receive"),
> > >> > getConsumerSplitMethod() (@Value ${consumer.splitMethod},"split"),
> and
> > >> > getConsumerActivityQUri() (@Value ${consumerActivityQUri},
> > >> > "direct:activityQ"). This is different than the camelContext.xml in
> that
> > >> > the route is being created programmatically. What this is doing is
> routing
> > >> > input from the inroute url (which Camel does automatically through
> the
> > >> > configure method which is required to be overridden), to the
> "receive"
> > >> > method of ActivityConsumer, then to the "split" method of
> ActivityConsumer,
> > >> > and then to the "direct:activityQ" which if you look back in the
> > >> > camelContext.xml routes to the "activemq:queue:activities" which
> then
> > >> > routes to "receiveExchange"
> > >> > This is the process to register a publisher. The process for
> registering a
> > >> > subscriber is relatively the same though it involves separate
> classes with
> > >> > their own private static final RouteBuilder class. From my
> perspective, the
> > >> > two most difficult things with setting this project up were
> understanding
> > >> > that the "process" method of the class that implements "Processor"
> is the
> > >> > entry point and the DynamicConsumerRouteBuilder creates the second
> entry
> > >> > point (The 5th and last points). This made the project very, VERY
> hard to
> > >> > understand.
> > >> > In addition to simplicity of design, the mvn clean install of the
> web
> > >> > service project is much faster and small scale activity publishing
> is also
> > >> > faster (see my email about load testing). These are minor points
> though as
> > >> > compilation has no effect on deployment. OSGi does add the benefit
> of
> > >> > modularized programming which is valuable, though I think the added
> > >> > complexity of Camel merits moving the project away from this
> paradigm.
>

Camel != OSGI. Camel can run just fine inside a WAR and you can have an
OSGI based application without Camel. OSGI is a deployment model where you
have well defined interfaces between components and isolation. It's more of
a competitor with Spring although you can run Spring components in an OSGI
server. In fact Spring DM is now Eclipse Gemini which is an OSGI Blueprint
implementation.


> > >> >
> > >>
> > >> I agree that the project is pretty difficult to understand ATM.  I
> think
> > >> what we need to do is think about what the responsibilities of the
> code are
> > >> and allow for different implementations that are not so tightly
> coupled as
> > >> they are now.  For instance, having worked with Storm to ingest
> millions of
> > >> activities a day, I personally would like to see streams be
> responsible for
> > >> defining an over-arching orchestration model that can be implemented
> within
> > >> a single war or on top of a distributed system.  This would look
> something
> > >> like the follows
>
> >>
> > >> |__ Streams-Core  (Base classes, interfaces, utilities, extensions,
> etc)
> > >> |
> > >> |__ Streams-Storm (Storm implementation of the core)
> > >> |
> > >> |__ Streams-Camel (Camel implementation of the core)
> > >> |
> > >> |__ Streams-WS (Web service implementation)
>

Matt, I understand what you are trying to do here but again, big picture
I'm sure sure it's this clean cut. Maybe it is for simple implementations
but I would think larger system would require a combination of things like
Storm and Camel working together.


> > >>
> > >>
> > >> > Danny
> > >> >
> > >> >
> > >> > > From: dsullivan7@hotmail.com
> > >> > > To: dev@streams.incubator.apache.org
> > >> > > Subject: [DISCUSS] Switching Streams from Camel deployment to .war
> > >> > deployment
> > >> > > Date: Tue, 29 Oct 2013 11:07:39 -0400
> > >> > >
> > >> > > The discussion thread for switching Streams from
> Camel/osgi/Servicemix
> > >> > to a single .war deployment
> > >> >
> > >> >
> > >
>
>

Re: [DISCUSS] Switching Streams from Camel deployment to .war deployment

[Chris Geer <ch...@cxtsoftware.com>]

On Fri, Nov 1, 2013 at 11:06 AM, Danny Sullivan <ds...@hotmail.com>wrote:

> Excellent, thanks very much for the info. This does help me understand why
> you would want to use EIP for development of certain systems. Because I
> understand the value of this design I think it would be a good idea to keep
> the current EIP implementation as the trunk branch and retract my request
> to merge the webservice branch with trunk at this time. However, I think it
> would also still be valuable to draw out the webservice branch and explore
> Streams as an "application" rather than a "system", at least until there is
> a crystal clear vision of what the direction should be.
>
> I think both of these application/systems will move in the same direction
> and ultimately fulfill the initial design goal of a "lightweight (yet
> scalable) server for ActivityStreams". To keep this goal in mind, I'd like
> to keep the discussion going by posing a scenario. Consider that Streams is
> completely developed. What are the answers to these questions in this
> scenario:
>
> "What does Streams do?"
>

My 2 cents - Streams is a collection of components that can be used to add
activity processing capabilities to your system/application.

>
> "How do I enable my application to support Streams?"
>
> I'm not sure it's really about enabling your application to support
Streams, it's about using Streams to enable your system to support
processing activities. Streams should give you the components which you can
configure together to give you the right processing steps to get activity
data in the format you need to power your system.

>
> Thanks very much for all the feedback, this has been a very good
> discussion!
> Danny
>
> > Date: Fri, 1 Nov 2013 12:53:28 -0400
> > Subject: Re: [DISCUSS] Switching Streams from Camel deployment to .war
> deployment
> > From: jletourneau80@gmail.com
> > To: dev@streams.incubator.apache.org
> >
> > Nice Chris - well put.  Streams isn't an application (or shouldn't be
> > thought of as one) it is a system that potentially is part of a system
> > of systems.
> >
> > To punctuate your example, I also like to recall the classic EIP
> > banking/loan example [3]
> >
> > [3] http://www.eaipatterns.com/ComposedMessagingExample.html
> >
> > On Fri, Nov 1, 2013 at 12:48 PM, Chris Geer <ch...@cxtsoftware.com>
> wrote:
> > > On Fri, Nov 1, 2013 at 6:56 AM, Danny Sullivan <dsullivan7@hotmail.com
> >wrote:
> > >
> > >> Perhaps it would be best for me to take a step back and ask what
> problem
> > >> are we trying to solve by using Camel? My understanding is that it is
> to
> > >> allow distribution of Streams to multiple servers. It would be
> helpful to
> > >> me to see another open source application that has this deployment
> > >> structure (it doesn't have to use Camel, just an application that has
> > >> multiple jars communicating with one another). Does anyone have any
> > >> suggestions of software for me to look at?
> > >>
> > >
> > > Danny, I wish I could just show you my system we are building.
> Honestly I
> > > think you are looking at this the wrong way. You are looking at it
> from a
> > > software development perspective (jars, protocols) and not from a
> > > business/system perspective (messages, components). So let me use an
> > > example.
> > >
> > > Let's say I have a business that takes order for widgets. I want the
> order
> > > taking to be streamlined for the user. To make that happen my order
> entry
> > > system does some basic checks when the user is placing the order and if
> > > those pass it saves the order to the database. To kick off the other
> > > processing, whenever I save an order to the database I sent a message
> to a
> > > Topic (pub/sub) saying either an order was created or an order was
> updated.
> > > Now at this point there is a lot more work I have to do to get the
> order
> > > shipped to the customer but it's an async action to the order taking.
> At
> > > this point the order has been placed and the customer can go about
> their
> > > business.
> > >
> > > I also have a component I wrote that listens to the Topic for "Order
> > > Created" messages. It takes each message, retrieves the order does some
> > > inventory processing to get the order ready to ship. At this point the
> > > order is complete and shipped to the customer.
> > >
> > > Now, in this simple example you could argue that the order entry system
> > > could just call the order processing component directly but that would
> > > preclude some extensibility in the future.
> > >
> > > A few months down the road I get a requirement to be able to notify the
> > > sales team whenever an order is placed by a certain set of customers.
> In a
> > > traditional model I'd have to modify the order entry component and put
> in
> > > logic to handle that. In this architecture all I have to do is build a
> > > different component that listens to that same Topic for order by
> certain
> > > customers. When it gets a notification it can then do the logic to
> send the
> > > message. The beauty of this is the order entry system doesn't' have to
> be
> > > modified and there is no risk of inserting a bug into that core
> process.
> > >
> > > So you ask, where does camel come in here. It could be used in lots of
> > > places but the natural fit is for the shipment processing component to
> be a
> > > camel route that listens to the Topic and calls a java object when
> messages
> > > come in. Also for the notification components, the whole thing could
> > > probably be a camel route (0 Java code).
> > >
> > > This is a pretty simple example and there are lots of other benefits I
> > > didn't talk about like redundancy and deployment flexibility.
> > >
> > > Does that help?
> > >
> > >>
> > >> > Date: Thu, 31 Oct 2013 20:23:26 -0400
> > >> > Subject: Re: [DISCUSS] Switching Streams from Camel deployment to
> .war
> > >> deployment
> > >> > From: jletourneau80@gmail.com
> > >> > To: dev@streams.incubator.apache.org
> > >> >
> > >> > So Danny in case it was lost in the ensuing transport discussion,
> > >> > Camel abstracts the transport, so you don't have to care if its http
> > >> > or whatever, only about the message format.  This is helpful if you
> > >> > care about plugging in components that DO care about the transport
> of
> > >> > course.  To answer your other questions, the downside to having the
> > >> > components all work via http (and not using messaging/Camel) is that
> > >> > everything immediately becomes point to point, which isn't a
> > >> > performance issue in its own right but limits scaling to more than
> one
> > >> > component being at the receiving end/fulfillment for instance as
> we've
> > >> > been discussing as being optimal.
> > >> >
> > >> > On Thu, Oct 31, 2013 at 5:38 PM, Matt Franklin <
> m.ben.franklin@gmail.com>
> > >> wrote:
> > >> > > On Thu, Oct 31, 2013 at 5:34 PM, Chris Geer <
> chris@cxtsoftware.com>
> > >> wrote:
> > >> > >
> > >> > >> On Thu, Oct 31, 2013 at 2:32 PM, Matt Franklin <
> > >> m.ben.franklin@gmail.com
> > >> > >> >wrote:
> > >> > >>
> > >> > >> > On Thu, Oct 31, 2013 at 5:28 PM, Chris Geer <
> chris@cxtsoftware.com>
> > >> > >> wrote:
> > >> > >> >
> > >> > >> > > On Thu, Oct 31, 2013 at 2:23 PM, Matt Franklin <
> > >> > >> m.ben.franklin@gmail.com
> > >> > >> > > >wrote:
> > >> > >> > >
> > >> > >> > > > On Thu, Oct 31, 2013 at 5:04 PM, Danny Sullivan <
> > >> > >> > dsullivan7@hotmail.com
> > >> > >> > > > >wrote:
> > >> > >> > > >
> > >> > >> > > > > I'm not quite following this, so I apologize. What I'm
> trying
> > >> to do
> > >> > >> > is
> > >> > >> > > > > programmatically make a request to a jar running on a
> > >> separate jvm
> > >> > >> > and
> > >> > >> > > > get
> > >> > >> > > > > the response from that call all within the same method.
> > >> Similar to
> > >> > >> > this
> > >> > >> > > > > http request:
> > >> > >> > > > >
> > >> > >> > > > > HttpGet httpget = new HttpGet();
> > >> > >> > > > >
> > >> > >> > > > > httpget.setURI(new URI("www.streams-persistence.com"));
> > >> > >> > > > >
> > >> > >> > > > > CloseableHttpResponse response =
> httpclient.execute(httpget);
> > >> > >> > > > >
> > >> > >> > > > > //do stuff with the response...
> > >> > >> > > > >
> > >> > >> > > > >
> > >> > >> > > > >
> > >> > >> > > > >
> > >> > >> > > > > I imagine that this would translate to something in Camel
> > >> similar
> > >> > >> to
> > >> > >> > > > this:
> > >> > >> > > > >
> > >> > >> > > > >
> > >> > >> > > > > <route>
> > >> > >> > > > >
> > >> > >> > > > > <from
> uri="bean:subscriberService?method=getAllSubscribers"/>
> > >> > >> > > > >
> > >> > >> > > > > <inOut
> > >> > >> > > > >
> > >> > >> > > >
> > >> > >> > >
> > >> > >> >
> > >> > >>
> > >>
> uri="addressToStreamsPersistenceJar?method=selectAllSubscribersFromDatabase"/>
> > >> > >> > > > >
> > >> > >> > > > > </route>
> > >> > >> > > > >
> > >> > >> > > >
> > >> > >> > > > Camel uses AMQP as a messaging system so that you don't
> have to
> > >> > >> create
> > >> > >> > > HTTP
> > >> > >> > > > requests between sources, though it is possible to do so.
>  In
> > >> Storm,
> > >> > >> we
> > >> > >> > > > usually try to use a buffer system like Kafka to do the
> same.
> > >> > >> > > >
> > >> > >> > >
> > >> > >> > > Camel can use AMQP through ActiveMQ.
> > >> > >> > >
> > >> > >> > >
> > >> > >> > Can being the operative word.  I haven't done much with Camel.
>  What
> > >> > >> other
> > >> > >> > transports are available.
> > >> > >> >
> > >> > >> > OpenWire (native ActiveMQ)
> > >> > >> AMQP
> > >> > >> STOMP
> > >> > >> HTTP (can be a server or client)
> > >> > >> Mail
> > >> > >> Amazon SQS
> > >> > >> XMPP
> > >> > >> ....hundred or so more
> > >> > >>
> > >> > >> http://camel.apache.org/components.html
> > >> > >
> > >> > >
> > >> > >
> > >> > > Nice.  I was expecting RTFM, so thanks for catering to my
> laziness.
> > >> > >
> > >> > >
> > >> > >>
> > >> > >>
> > >> > >> >
> > >> > >> > > >
> > >> > >> > > >
> > >> > >> > > > >
> > >> > >> > > > >
> > >> > >> > > > > But it is unclear what the actual implementation would
> be.
> > >> > >> > > > >
> > >> > >> > > > >
> > >> > >> > > > > This actually brings me to another suggestion. Would
> there be
> > >> a big
> > >> > >> > > > > performance impact to have communication between the
> software
> > >> > >> > > components
> > >> > >> > > > > occur between http? Say the 5 software components I
> outlined
> > >> > >> earlier
> > >> > >> > > were
> > >> > >> > > > > packaged as 5 separate wars. These wars could communicate
> > >> with each
> > >> > >> > > other
> > >> > >> > > > > via get a post requests. This sounds unconventional
> offhand
> > >> so I'd
> > >> > >> > like
> > >> > >> > > > to
> > >> > >> > > > > hear some thoughts on it.
> > >> > >> > > > >
> > >> > >> > > >
> > >> > >> > > > It is most certainly possible, but IMO probably not the
> best
> > >> option
> > >> > >> for
> > >> > >> > > > success.  Protocols like PubSubHubbub use HTTP for a
> message
> > >> > >> transport.
> > >> > >> > > >
> > >> > >> > > >
> > >> > >> > > > > -Danny
> > >> > >> > > > >
> > >> > >> > > > > > Date: Thu, 31 Oct 2013 13:28:41 -0700
> > >> > >> > > > > > Subject: Re: [DISCUSS] Switching Streams from Camel
> > >> deployment to
> > >> > >> > > .war
> > >> > >> > > > > deployment
> > >> > >> > > > > > From: chris@cxtsoftware.com
> > >> > >> > > > > > To: dev@streams.incubator.apache.org
> > >> > >> > > > > >
> > >> > >> > > > > > Or Content Enricher [2]
> > >> > >> > > > > >
> > >> > >> > > > > > [2] http://camel.apache.org/content-enricher.html
> > >> > >> > > > > >
> > >> > >> > > > > >
> > >> > >> > > > > > On Thu, Oct 31, 2013 at 12:37 PM, Jason Letourneau
> > >> > >> > > > > > <jl...@gmail.com>wrote:
> > >> > >> > > > > >
> > >> > >> > > > > > > check out the link here[1]
> > >> > >> > > > > > >
> > >> > >> > > > > > >
> > >> > >> > > > > > > [1]http://camel.apache.org/request-reply.html
> > >> > >> > > > > > >
> > >> > >> > > > > > > On Thu, Oct 31, 2013 at 2:32 PM, Danny Sullivan <
> > >> > >> > > > > dsullivan7@hotmail.com>
> > >> > >> > > > > > > wrote:
> > >> > >> > > > > > > > I have a quick Camel question that I arrived at in
> the
> > >> > >> > > > > implementation of
> > >> > >> > > > > > > these new components:
> > >> > >> > > > > > > >
> > >> > >> > > > > > > > Lets say I have a method in streams-activity.jar
> that
> > >> needs
> > >> > >> all
> > >> > >> > > > > > > subscribers in the database. This would require a
> call to
> > >> the
> > >> > >> > > > > > > streams-persistence.jar. So far, I've seen camel used
> > >> mostly
> > >> > >> for
> > >> > >> > > > > passing
> > >> > >> > > > > > > data through the application, but not for making a
> single
> > >> > >> > > > > request-reponse
> > >> > >> > > > > > > from within a method. How can I use Camel to get a
> list
> > >> of all
> > >> > >> > > > > subscribers
> > >> > >> > > > > > > in the streams-persistence.jar from the
> > >> streams-activity.jar?
> > >> > >> > > > > > > >
> > >> > >> > > > > > > >> From: dsullivan7@hotmail.com
> > >> > >> > > > > > > >> To: dev@streams.incubator.apache.org
> > >> > >> > > > > > > >> Subject: RE: [DISCUSS] Switching Streams from
> Camel
> > >> > >> deployment
> > >> > >> > > to
> > >> > >> > > > > .war
> > >> > >> > > > > > > deployment
> > >> > >> > > > > > > >> Date: Thu, 31 Oct 2013 08:40:44 -0400
> > >> > >> > > > > > > >>
> > >> > >> > > > > > > >> Excellent, I'll write up something as a proof of
> > >> concept and
> > >> > >> > we
> > >> > >> > > > can
> > >> > >> > > > > > > discuss further to make sure everything is vanilla.
> > >> > >> > > > > > > >>
> > >> > >> > > > > > > >> > Date: Wed, 30 Oct 2013 17:01:28 -0400
> > >> > >> > > > > > > >> > Subject: Re: [DISCUSS] Switching Streams from
> Camel
> > >> > >> > deployment
> > >> > >> > > > to
> > >> > >> > > > > > > .war deployment
> > >> > >> > > > > > > >> > From: jletourneau80@gmail.com
> > >> > >> > > > > > > >> > To: dev@streams.incubator.apache.org
> > >> > >> > > > > > > >> >
> > >> > >> > > > > > > >> > That sounds pretty promising to me.
> > >> > >> > > > > > > >> >
> > >> > >> > > > > > > >> > On Wed, Oct 30, 2013 at 2:38 PM, Danny Sullivan
> <
> > >> > >> > > > > > > dsullivan7@hotmail.com> wrote:
> > >> > >> > > > > > > >> > > Thanks for the feedback. You have an
> interesting
> > >> point
> > >> > >> > about
> > >> > >> > > > the
> > >> > >> > > > > > > url linking to a separate processing space. Let me
> tie my
> > >> > >> answer
> > >> > >> > > into
> > >> > >> > > > > your
> > >> > >> > > > > > > last question about "advocating for the simplicity at
> > >> > >> > registration
> > >> > >> > > to
> > >> > >> > > > > give
> > >> > >> > > > > > > up flexibility at registration, but retaining the
> inner
> > >> "guts"
> > >> > >> of
> > >> > >> > > > > > > EIP/messaging". Consider a new architecture:
> > >> > >> > > > > > > >> > >
> > >> > >> > > > > > > >> > > streams-web.war: single entry point to
> > >> application, but
> > >> > >> > > > > functions
> > >> > >> > > > > > > ONLY as an entry point. From here Camel routes the
> > >> incoming
> > >> > >> > > requests
> > >> > >> > > > > to 4
> > >> > >> > > > > > > separate jarssubscriber-registration.jar: subscriber
> > >> > >> registration
> > >> > >> > > > > > > publisher-registration.jar: publisher
> > >> registrationactivity.jar:
> > >> > >> > > > returns
> > >> > >> > > > > > > activity (also contains subscriber warehouse and
> storm
> > >> activity
> > >> > >> > > > > > > aggregator)publish.jar: publishes
> > >> > >> activitystreams-cassandra.jar:
> > >> > >> > > the
> > >> > >> > > > > above
> > >> > >> > > > > > > 4 jars would all have a hook into this jar which
> would
> > >> function
> > >> > >> > as
> > >> > >> > > a
> > >> > >> > > > > hook
> > >> > >> > > > > > > onto the database. Each jar would have camel route
> output
> > >> to
> > >> > >> this
> > >> > >> > > > jar.
> > >> > >> > > > > > > >> > >
> > >> > >> > > > > > > >> > > In this implementation, Camel would no longer
> be
> > >> the
> > >> > >> entry
> > >> > >> > > and
> > >> > >> > > > > exit
> > >> > >> > > > > > > point of a client to the application, but would
> handle the
> > >> > >> > > > > communication
> > >> > >> > > > > > > between components. The flow of activity through the
> > >> > >> application
> > >> > >> > > > would
> > >> > >> > > > > be
> > >> > >> > > > > > > method based in each jar. This would allow
> deployment on
> > >> up to
> > >> > >> 6
> > >> > >> > > > > different
> > >> > >> > > > > > > process spaces. However, this does not address that
> there
> > >> is a
> > >> > >> > > single
> > >> > >> > > > > > > server entry point, but I'm not sure if it was a
> concern
> > >> in the
> > >> > >> > > first
> > >> > >> > > > > place.
> > >> > >> > > > > > > >> > >
> > >> > >> > > > > > > >> > > My argument, at its basis, is that we should
> move
> > >> away
> > >> > >> > from
> > >> > >> > > > > using
> > >> > >> > > > > > > Camel as the entry point to the application. I would
> be
> > >> happy
> > >> > >> to
> > >> > >> > > > > maintain
> > >> > >> > > > > > > messaging between components.
> > >> > >> > > > > > > >> > >
> > >> > >> > > > > > > >> > >> Date: Wed, 30 Oct 2013 12:32:55 -0400
> > >> > >> > > > > > > >> > >> Subject: Re: [DISCUSS] Switching Streams from
> > >> Camel
> > >> > >> > > > deployment
> > >> > >> > > > > to
> > >> > >> > > > > > > .war deployment
> > >> > >> > > > > > > >> > >> From: jletourneau80@gmail.com
> > >> > >> > > > > > > >> > >> To: dev@streams.incubator.apache.org
> > >> > >> > > > > > > >> > >>
> > >> > >> > > > > > > >> > >> An interesting use case that I am holding
> onto is
> > >> the
> > >> > >> > > ability
> > >> > >> > > > > for
> > >> > >> > > > > > > >> > >> publishers to register via a single URL
> > >> (registration
> > >> > >> > > > > endpoint),
> > >> > >> > > > > > > but
> > >> > >> > > > > > > >> > >> be sent a URL back to post to a different
> process
> > >> space
> > >> > >> > for
> > >> > >> > > > > actual
> > >> > >> > > > > > > >> > >> publishing.  The same is true on the
> subscriber
> > >> front.
> > >> > >> > > > >  Currently,
> > >> > >> > > > > > > the
> > >> > >> > > > > > > >> > >> Camel/EIP infrastructure abstracts this
> because
> > >> > >> different
> > >> > >> > > > > > > components
> > >> > >> > > > > > > >> > >> deployed in different process spaces
> handling the
> > >> route
> > >> > >> > > > > creation
> > >> > >> > > > > > > can
> > >> > >> > > > > > > >> > >> just be bolted onto a running Streams
> instance
> > >> without
> > >> > >> > new
> > >> > >> > > > > > > subs/pubs
> > >> > >> > > > > > > >> > >> behaving any differently than existing.  This
> > >> seems to
> > >> > >> > be a
> > >> > >> > > > > > > >> > >> potentially critical scaling point.  Is
> there a
> > >> way to
> > >> > >> do
> > >> > >> > > > this
> > >> > >> > > > > with
> > >> > >> > > > > > > >> > >> the Spring solution?
> > >> > >> > > > > > > >> > >>
> > >> > >> > > > > > > >> > >> The persistence point is a good one, though I
> > >> would
> > >> > >> > > classify
> > >> > >> > > > > that
> > >> > >> > > > > > > as
> > >> > >> > > > > > > >> > >> "not implemented" vs "not possible" (not
> that you
> > >> > >> were).
> > >> > >> > > > > > > >> > >>
> > >> > >> > > > > > > >> > >> I'm not married to Camel, I just like the EIP
> > >> approach
> > >> > >> to
> > >> > >> > > > > building
> > >> > >> > > > > > > >> > >> something that is ultimately a messaging
> system.
> > >>  There
> > >> > >> > are
> > >> > >> > > > > known
> > >> > >> > > > > > > >> > >> patterns that solve at least a subset of the
> > >> problems
> > >> > >> > > Streams
> > >> > >> > > > > is
> > >> > >> > > > > > > >> > >> trying to solve and implementations that can
> > >> handle the
> > >> > >> > > load
> > >> > >> > > > > and
> > >> > >> > > > > > > I'll
> > >> > >> > > > > > > >> > >> reiterate flexibility == complexity almost
> always.
> > >> > >> > > > > > > >> > >>
> > >> > >> > > > > > > >> > >> It comes right back to the central question:
> Do
> > >> you
> > >> > >> want
> > >> > >> > > > > > > flexibility
> > >> > >> > > > > > > >> > >> or simplicity?  It doesn't have to be black
> and
> > >> white
> > >> > >> > > either
> > >> > >> > > > I
> > >> > >> > > > > > > don't
> > >> > >> > > > > > > >> > >> think...
> > >> > >> > > > > > > >> > >>
> > >> > >> > > > > > > >> > >> More pointedly: Where should we give up
> > >> flexibility for
> > >> > >> > > > > > > simplicity?  I
> > >> > >> > > > > > > >> > >> read that Danny is advocating for the
> simplicity
> > >> at
> > >> > >> > > > > registration to
> > >> > >> > > > > > > >> > >> give up flexibility at registration, but
> > >> retaining the
> > >> > >> > > inner
> > >> > >> > > > > > > "guts" of
> > >> > >> > > > > > > >> > >> EIP/messaging?  Thoughts?
> > >> > >> > > > > > > >> > >>
> > >> > >> > > > > > > >> > >> On Wed, Oct 30, 2013 at 11:37 AM, Danny
> Sullivan <
> > >> > >> > > > > > > dsullivan7@hotmail.com> wrote:
> > >> > >> > > > > > > >> > >> > My argument is not for the IoC pattern as
> that
> > >> can be
> > >> > >> > > (and
> > >> > >> > > > > has
> > >> > >> > > > > > > been) implemented alongside Camel. My main argument
> is
> > >> that the
> > >> > >> > > > syntax
> > >> > >> > > > > at
> > >> > >> > > > > > > the entry point is not only familiar but much
> simpler.
> > >> This
> > >> > >> > > wouldn't
> > >> > >> > > > > be a
> > >> > >> > > > > > > very strong argument if the Camel implementation
> wasn't
> > >> much
> > >> > >> more
> > >> > >> > > > > > > complicated but I feel that it is the case. Also,
> looking
> > >> > >> toward
> > >> > >> > > the
> > >> > >> > > > > > > future, if the server is restarted, in-routes are
> lost in
> > >> > >> Camel.
> > >> > >> > > The
> > >> > >> > > > > way to
> > >> > >> > > > > > > curb this is to persist the dynamic routes that Camel
> > >> creates,
> > >> > >> > and
> > >> > >> > > > > then on
> > >> > >> > > > > > > start up pull every one of these routes and recreate
> a
> > >> dynamic
> > >> > >> > > route
> > >> > >> > > > > for
> > >> > >> > > > > > > each one. Not only is this much easier to implement
> using
> > >> the
> > >> > >> > > Spring
> > >> > >> > > > > web
> > >> > >> > > > > > > implementation, but it already has been implemented
> and
> > >> you can
> > >> > >> > try
> > >> > >> > > > it
> > >> > >> > > > > by
> > >> > >> > > > > > > checking out the webservice branch, registering a
> > >> subscriber,
> > >> > >> > > > > restarting
> > >> > >> > > > > > > tomcat, and using the same url you had before. This
> will
> > >> allow
> > >> > >> > > > > subscribers
> > >> > >> > > > > > > to hang on to their urls once they register. (the
> same is
> > >> true
> > >> > >> > for
> > >> > >> > > > > > > publishers: you can post via the same url after
> restarting
> > >> > >> > tomcat)
> > >> > >> > > > > > > >> > >> >
> > >> > >> > > > > > > >> > >> >> Date: Wed, 30 Oct 2013 11:00:21 -0400
> > >> > >> > > > > > > >> > >> >> Subject: Re: [DISCUSS] Switching Streams
> from
> > >> Camel
> > >> > >> > > > > deployment
> > >> > >> > > > > > > to .war deployment
> > >> > >> > > > > > > >> > >> >> From: jletourneau80@gmail.com
> > >> > >> > > > > > > >> > >> >> To: dev@streams.incubator.apache.org
> > >> > >> > > > > > > >> > >> >>
> > >> > >> > > > > > > >> > >> >> To be fair, while the current
> implementation is
> > >> > >> > heavily
> > >> > >> > > > > > > camel-based,
> > >> > >> > > > > > > >> > >> >> all of the interfaces related to Streams
> > >> > >> functionality
> > >> > >> > > are
> > >> > >> > > > > not.
> > >> > >> > > > > > >  The
> > >> > >> > > > > > > >> > >> >> current model maps to what Matt has
> outlined
> > >> in my
> > >> > >> > > > opinion,
> > >> > >> > > > > > > though
> > >> > >> > > > > > > >> > >> >> packing names etc. probably don't follow
> that
> > >> exact
> > >> > >> > > > pattern.
> > >> > >> > > > > > > >> > >> >>
> > >> > >> > > > > > > >> > >> >> With regards to the complexity and
> different
> > >> > >> > components
> > >> > >> > > in
> > >> > >> > > > > the
> > >> > >> > > > > > > >> > >> >> registration process, this was a cut at
> the
> > >> > >> > abstraction
> > >> > >> > > > > based
> > >> > >> > > > > > > on the
> > >> > >> > > > > > > >> > >> >> assumption that different implementations
> may
> > >> be
> > >> > >> > plugged
> > >> > >> > > > in
> > >> > >> > > > > and
> > >> > >> > > > > > > in
> > >> > >> > > > > > > >> > >> >> fact may live on different processor space
> > >> (ie. a
> > >> > >> > > polling
> > >> > >> > > > > > > publisher vs
> > >> > >> > > > > > > >> > >> >> a push publisher may be instantiated on
> > >> different
> > >> > >> > > servers
> > >> > >> > > > > but
> > >> > >> > > > > > > the
> > >> > >> > > > > > > >> > >> >> registration URL is staticly defined).
> > >> > >> > > > > > > >> > >> >>
> > >> > >> > > > > > > >> > >> >> Is the main argument I am seeing for
> Spring
> > >>  the
> > >> > >> > > > > familiarity of
> > >> > >> > > > > > > its
> > >> > >> > > > > > > >> > >> >> IoC pattern implementation and syntax at
> the
> > >> entry
> > >> > >> > > point?
> > >> > >> > > > > > > >> > >> >>
> > >> > >> > > > > > > >> > >> >> On Wed, Oct 30, 2013 at 10:53 AM, Danny
> > >> Sullivan <
> > >> > >> > > > > > > dsullivan7@hotmail.com> wrote:
> > >> > >> > > > > > > >> > >> >> > Could you clarify whether the same entry
> > >> points
> > >> > >> > would
> > >> > >> > > > > exist
> > >> > >> > > > > > > for the camel implementation of the core
> (implementing the
> > >> > >> > > "process"
> > >> > >> > > > > > > method/ using a DynammicRouteBuilder) or would the
> > >> webservice
> > >> > >> be
> > >> > >> > > the
> > >> > >> > > > > sole
> > >> > >> > > > > > > entry point to Streams and after it enters would it
> hand
> > >> it off
> > >> > >> > to
> > >> > >> > > > > Camel?
> > >> > >> > > > > > > And what would be the entry point for the Storm
> > >> implementation?
> > >> > >> > > > > > > >> > >> >> > -Danny
> > >> > >> > > > > > > >> > >> >> >
> > >> > >> > > > > > > >> > >> >> >> Date: Wed, 30 Oct 2013 10:13:04 -0400
> > >> > >> > > > > > > >> > >> >> >> Subject: Re: [DISCUSS] Switching
> Streams
> > >> from
> > >> > >> Camel
> > >> > >> > > > > > > deployment to .war deployment
> > >> > >> > > > > > > >> > >> >> >> From: m.ben.franklin@gmail.com
> > >> > >> > > > > > > >> > >> >> >> To: dev@streams.incubator.apache.org
> > >> > >> > > > > > > >> > >> >> >>
> > >> > >> > > > > > > >> > >> >> >> On Tue, Oct 29, 2013 at 2:55 PM, Danny
> > >> Sullivan <
> > >> > >> > > > > > > dsullivan7@hotmail.com>wrote:
> > >> > >> > > > > > > >> > >> >> >>
> > >> > >> > > > > > > >> > >> >> >> > My case for switching from OSGi is
> for
> > >> > >> simplicity
> > >> > >> > > in
> > >> > >> > > > > > > design. To follow the
> > >> > >> > > > > > > >> > >> >> >> > path of an activity through Streams
> in the
> > >> > >> > > > webservice,
> > >> > >> > > > > > > there is one main
> > >> > >> > > > > > > >> > >> >> >> > things the developer needs to
> understand:
> > >> > >> > > > > > > >> > >> >> >> > The @RequestMapping annotation
> specifies
> > >> the
> > >> > >> HTTP
> > >> > >> > > > entry
> > >> > >> > > > > > > point
> > >> > >> > > > > > > >> > >> >> >> > There are 4 @RequestMapping
> annotations
> > >> that
> > >> > >> > > > > correspond to
> > >> > >> > > > > > > each of the 4
> > >> > >> > > > > > > >> > >> >> >> > ways a user enters the application:
> > >> > >> registering a
> > >> > >> > > > > > > publisher, registering a
> > >> > >> > > > > > > >> > >> >> >> > subscriber, publishing activity, and
> > >> getting an
> > >> > >> > > > > activity
> > >> > >> > > > > > > stream. Where
> > >> > >> > > > > > > >> > >> >> >> > these are located in the source code
> can
> > >> be
> > >> > >> found
> > >> > >> > > by
> > >> > >> > > > > > > searching for the
> > >> > >> > > > > > > >> > >> >> >> > paths specified in the documentation
> > >> (search
> > >> > >> for
> > >> > >> > > > > > > "/publisherRegister",
> > >> > >> > > > > > > >> > >> >> >> > "/subscriberRegister",
> "/publishActivity",
> > >> > >> > > > > "/getActivity"
> > >> > >> > > > > > > which will all
> > >> > >> > > > > > > >> > >> >> >> > lead you to
> StreamsWebController.java).
> > >> From
> > >> > >> the
> > >> > >> > > > > methods
> > >> > >> > > > > > > that process
> > >> > >> > > > > > > >> > >> >> >> > requests, the flow through the
> > >> application is
> > >> > >> > > through
> > >> > >> > > > > > > methods which can be
> > >> > >> > > > > > > >> > >> >> >> > understood by most Java programmers.
> > >> > >> > > > > > > >> > >> >> >> > The flow of activities through the
> current
> > >> > >> trunk
> > >> > >> > > > > branch is
> > >> > >> > > > > > > understood as
> > >> > >> > > > > > > >> > >> >> >> > follows:
> > >> > >> > > > > > > >> > >> >> >> > The string "/publisher/register" (the
> > >> entry
> > >> > >> point
> > >> > >> > > to
> > >> > >> > > > > > > register a publisher
> > >> > >> > > > > > > >> > >> >> >> > specified in the documentation) is
> the
> > >> value of
> > >> > >> > the
> > >> > >> > > > > > > >> > >> >> >> > consumer.registrationEndpoint
> property
> > >> defined
> > >> > >> in
> > >> > >> > > > > > > streams.propertiesThe
> > >> > >> > > > > > > >> > >> >> >> > camelContext.xml specifies an
> endpoint
> > >> with the
> > >> > >> > id
> > >> > >> > > > > > > >> > >> >> >> > consumerRegistrationEndpoint to have
> a uri
> > >> > >> equal
> > >> > >> > to
> > >> > >> > > > the
> > >> > >> > > > > > > propertyThe
> > >> > >> > > > > > > >> > >> >> >> > consumerRegistrationEndpoint routes
> from
> > >> uri
> > >> > >> > > > > > > direct:publisher register with
> > >> > >> > > > > > > >> > >> >> >> > the bean
> activityRegistrationProcessor
> > >> nested
> > >> > >> in
> > >> > >> > > > > between
> > >> > >> > > > > > > the routeThe
> > >> > >> > > > > > > >> > >> >> >> > streams-eip-applicationContext
> contains a
> > >> bean
> > >> > >> > with
> > >> > >> > > > > the id
> > >> > >> > > > > > > >> > >> >> >> > activityRegistrationProcessor
> created for
> > >> the
> > >> > >> > class
> > >> > >> > > > > > > >> > >> >> >> >
> ActivityPublisherRegistrationProcessorThe
> > >> > >> > exchange
> > >> > >> > > > will
> > >> > >> > > > > > > enter the "process"
> > >> > >> > > > > > > >> > >> >> >> > method of the class
> > >> > >> > > > > > > ActivityPublisherRegisitrationProcessor and that this
> > >> > >> > > > > > > >> > >> >> >> > is because this class implements the
> > >> > >> "Processor"
> > >> > >> > > > > interface
> > >> > >> > > > > > > provided by
> > >> > >> > > > > > > >> > >> >> >> > CamelThe direct:add-publisher-route
> takes
> > >> the
> > >> > >> > > > exchange
> > >> > >> > > > > > > output from the
> > >> > >> > > > > > > >> > >> >> >> > "process" method and routes it to the
> > >> > >> > > > > > > activityRegistrationProcessor
> > >> > >> > > > > > > >> > >> >> >> > "register" method. The bean
> > >> > >> > > > > activityRegistrationProcessor
> > >> > >> > > > > > > is defined in the
> > >> > >> > > > > > > >> > >> >> >> >
> streams-eip-osgi-component-import.xmlThe
> > >> output
> > >> > >> > > from
> > >> > >> > > > > this
> > >> > >> > > > > > > method is then
> > >> > >> > > > > > > >> > >> >> >> > sent to the
> "createNewRouteForConsumer"
> > >> method
> > >> > >> of
> > >> > >> > > > > > > activityConsumerRouter.
> > >> > >> > > > > > > >> > >> >> >> > This method creates a new route for
> the
> > >> newly
> > >> > >> > > > > registered
> > >> > >> > > > > > > publisher using
> > >> > >> > > > > > > >> > >> >> >> > the private static final class
> > >> > >> > > > > DynamicConsumerRouteBuilder
> > >> > >> > > > > > > which is
> > >> > >> > > > > > > >> > >> >> >> > required to extend RouteBuilder
> which is
> > >> > >> provided
> > >> > >> > > by
> > >> > >> > > > > > > Camel. This
> > >> > >> > > > > > > >> > >> >> >> > DynamicConsumerRouteBuilder contains
> > >> several
> > >> > >> > > methods:
> > >> > >> > > > > > > >> > >> >> >> > getConsumerReceiveMethod() (which
> > >> corresponds
> > >> > >> to
> > >> > >> > > > @Value
> > >> > >> > > > > > > >> > >> >> >> > ${consumer.receiveMethod} which
> > >> corresponds to
> > >> > >> > > > > "receive"),
> > >> > >> > > > > > > >> > >> >> >> > getConsumerSplitMethod() (@Value
> > >> > >> > > > > > > ${consumer.splitMethod},"split"), and
> > >> > >> > > > > > > >> > >> >> >> > getConsumerActivityQUri() (@Value
> > >> > >> > > > > ${consumerActivityQUri},
> > >> > >> > > > > > > >> > >> >> >> > "direct:activityQ"). This is
> different
> > >> than the
> > >> > >> > > > > > > camelContext.xml in that
> > >> > >> > > > > > > >> > >> >> >> > the route is being created
> > >> programmatically.
> > >> > >> What
> > >> > >> > > > this
> > >> > >> > > > > is
> > >> > >> > > > > > > doing is routing
> > >> > >> > > > > > > >> > >> >> >> > input from the inroute url (which
> Camel
> > >> does
> > >> > >> > > > > automatically
> > >> > >> > > > > > > through the
> > >> > >> > > > > > > >> > >> >> >> > configure method which is required
> to be
> > >> > >> > > overridden),
> > >> > >> > > > > to
> > >> > >> > > > > > > the "receive"
> > >> > >> > > > > > > >> > >> >> >> > method of ActivityConsumer, then to
> the
> > >> "split"
> > >> > >> > > > method
> > >> > >> > > > > of
> > >> > >> > > > > > > ActivityConsumer,
> > >> > >> > > > > > > >> > >> >> >> > and then to the "direct:activityQ"
> which
> > >> if you
> > >> > >> > > look
> > >> > >> > > > > back
> > >> > >> > > > > > > in the
> > >> > >> > > > > > > >> > >> >> >> > camelContext.xml routes to the
> > >> > >> > > > > "activemq:queue:activities"
> > >> > >> > > > > > > which then
> > >> > >> > > > > > > >> > >> >> >> > routes to "receiveExchange"
> > >> > >> > > > > > > >> > >> >> >> > This is the process to register a
> > >> publisher.
> > >> > >> The
> > >> > >> > > > > process
> > >> > >> > > > > > > for registering a
> > >> > >> > > > > > > >> > >> >> >> > subscriber is relatively the same
> though
> > >> it
> > >> > >> > > involves
> > >> > >> > > > > > > separate classes with
> > >> > >> > > > > > > >> > >> >> >> > their own private static final
> > >> RouteBuilder
> > >> > >> > class.
> > >> > >> > > > > From my
> > >> > >> > > > > > > perspective, the
> > >> > >> > > > > > > >> > >> >> >> > two most difficult things with
> setting
> > >> this
> > >> > >> > project
> > >> > >> > > > up
> > >> > >> > > > > > > were understanding
> > >> > >> > > > > > > >> > >> >> >> > that the "process" method of the
> class
> > >> that
> > >> > >> > > > implements
> > >> > >> > > > > > > "Processor" is the
> > >> > >> > > > > > > >> > >> >> >> > entry point and the
> > >> DynamicConsumerRouteBuilder
> > >> > >> > > > creates
> > >> > >> > > > > > > the second entry
> > >> > >> > > > > > > >> > >> >> >> > point (The 5th and last points). This
> > >> made the
> > >> > >> > > > project
> > >> > >> > > > > > > very, VERY hard to
> > >> > >> > > > > > > >> > >> >> >> > understand.
> > >> > >> > > > > > > >> > >> >> >> > In addition to simplicity of design,
> the
> > >> mvn
> > >> > >> > clean
> > >> > >> > > > > install
> > >> > >> > > > > > > of the web
> > >> > >> > > > > > > >> > >> >> >> > service project is much faster and
> small
> > >> scale
> > >> > >> > > > activity
> > >> > >> > > > > > > publishing is also
> > >> > >> > > > > > > >> > >> >> >> > faster (see my email about load
> testing).
> > >> These
> > >> > >> > are
> > >> > >> > > > > minor
> > >> > >> > > > > > > points though as
> > >> > >> > > > > > > >> > >> >> >> > compilation has no effect on
> deployment.
> > >> OSGi
> > >> > >> > does
> > >> > >> > > > add
> > >> > >> > > > > the
> > >> > >> > > > > > > benefit of
> > >> > >> > > > > > > >> > >> >> >> > modularized programming which is
> valuable,
> > >> > >> > though I
> > >> > >> > > > > think
> > >> > >> > > > > > > the added
> > >> > >> > > > > > > >> > >> >> >> > complexity of Camel merits moving the
> > >> project
> > >> > >> > away
> > >> > >> > > > from
> > >> > >> > > > > > > this paradigm.
> > >> > >> > > > > > > >> > >> >> >> >
> > >> > >> > > > > > > >> > >> >> >>
> > >> > >> > > > > > > >> > >> >> >> I agree that the project is pretty
> > >> difficult to
> > >> > >> > > > > understand
> > >> > >> > > > > > > ATM.  I think
> > >> > >> > > > > > > >> > >> >> >> what we need to do is think about what
> the
> > >> > >> > > > > responsibilities
> > >> > >> > > > > > > of the code are
> > >> > >> > > > > > > >> > >> >> >> and allow for different implementations
> > >> that are
> > >> > >> > not
> > >> > >> > > so
> > >> > >> > > > > > > tightly coupled as
> > >> > >> > > > > > > >> > >> >> >> they are now.  For instance, having
> worked
> > >> with
> > >> > >> > Storm
> > >> > >> > > > to
> > >> > >> > > > > > > ingest millions of
> > >> > >> > > > > > > >> > >> >> >> activities a day, I personally would
> like
> > >> to see
> > >> > >> > > > streams
> > >> > >> > > > > be
> > >> > >> > > > > > > responsible for
> > >> > >> > > > > > > >> > >> >> >> defining an over-arching orchestration
> > >> model that
> > >> > >> > can
> > >> > >> > > > be
> > >> > >> > > > > > > implemented within
> > >> > >> > > > > > > >> > >> >> >> a single war or on top of a distributed
> > >> system.
> > >> > >> >  This
> > >> > >> > > > > would
> > >> > >> > > > > > > look something
> > >> > >> > > > > > > >> > >> >> >> like the follows:
> > >> > >> > > > > > > >> > >> >> >>
> > >> > >> > > > > > > >> > >> >> >> |__ Streams-Core  (Base classes,
> interfaces,
> > >> > >> > > utilities,
> > >> > >> > > > > > > extensions, etc)
> > >> > >> > > > > > > >> > >> >> >> |
> > >> > >> > > > > > > >> > >> >> >> |__ Streams-Storm (Storm
> implementation of
> > >> the
> > >> > >> > core)
> > >> > >> > > > > > > >> > >> >> >> |
> > >> > >> > > > > > > >> > >> >> >> |__ Streams-Camel (Camel
> implementation of
> > >> the
> > >> > >> > core)
> > >> > >> > > > > > > >> > >> >> >> |
> > >> > >> > > > > > > >> > >> >> >> |__ Streams-WS (Web service
> implementation)
> > >> > >> > > > > > > >> > >> >> >>
> > >> > >> > > > > > > >> > >> >> >>
> > >> > >> > > > > > > >> > >> >> >> > Danny
> > >> > >> > > > > > > >> > >> >> >> >
> > >> > >> > > > > > > >> > >> >> >> >
> > >> > >> > > > > > > >> > >> >> >> > > From: dsullivan7@hotmail.com
> > >> > >> > > > > > > >> > >> >> >> > > To:
> dev@streams.incubator.apache.org
> > >> > >> > > > > > > >> > >> >> >> > > Subject: [DISCUSS] Switching
> Streams
> > >> from
> > >> > >> Camel
> > >> > >> > > > > > > deployment to .war
> > >> > >> > > > > > > >> > >> >> >> > deployment
> > >> > >> > > > > > > >> > >> >> >> > > Date: Tue, 29 Oct 2013 11:07:39
> -0400
> > >> > >> > > > > > > >> > >> >> >> > >
> > >> > >> > > > > > > >> > >> >> >> > > The discussion thread for switching
> > >> Streams
> > >> > >> > from
> > >> > >> > > > > > > Camel/osgi/Servicemix
> > >> > >> > > > > > > >> > >> >> >> > to a single .war deployment
> > >> > >> > > > > > > >> > >> >> >> >
> > >> > >> > > > > > > >> > >> >> >> >
> > >> > >> > > > > > > >> > >> >> >
> > >> > >> > > > > > > >> > >> >
> > >> > >> > > > > > > >> > >
> > >> > >> > > > > > > >>
> > >> > >> > > > > > > >
> > >> > >> > > > > > >
> > >> > >> > > > >
> > >> > >> > > > >
> > >> > >> > > >
> > >> > >> > >
> > >> > >> >
> > >> > >>
> > >>
> > >>
>
>

RE: [DISCUSS] Switching Streams from Camel deployment to .war deployment

[Danny Sullivan <ds...@hotmail.com>]

Excellent, thanks very much for the info. This does help me understand why you would want to use EIP for development of certain systems. Because I understand the value of this design I think it would be a good idea to keep the current EIP implementation as the trunk branch and retract my request to merge the webservice branch with trunk at this time. However, I think it would also still be valuable to draw out the webservice branch and explore Streams as an "application" rather than a "system", at least until there is a crystal clear vision of what the direction should be.

I think both of these application/systems will move in the same direction and ultimately fulfill the initial design goal of a "lightweight (yet scalable) server for ActivityStreams". To keep this goal in mind, I'd like to keep the discussion going by posing a scenario. Consider that Streams is completely developed. What are the answers to these questions in this scenario:

"What does Streams do?"

"How do I enable my application to support Streams?" 


Thanks very much for all the feedback, this has been a very good discussion!
Danny

> Date: Fri, 1 Nov 2013 12:53:28 -0400
> Subject: Re: [DISCUSS] Switching Streams from Camel deployment to .war deployment
> From: jletourneau80@gmail.com
> To: dev@streams.incubator.apache.org
> 
> Nice Chris - well put.  Streams isn't an application (or shouldn't be
> thought of as one) it is a system that potentially is part of a system
> of systems.
> 
> To punctuate your example, I also like to recall the classic EIP
> banking/loan example [3]
> 
> [3] http://www.eaipatterns.com/ComposedMessagingExample.html
> 
> On Fri, Nov 1, 2013 at 12:48 PM, Chris Geer <ch...@cxtsoftware.com> wrote:
> > On Fri, Nov 1, 2013 at 6:56 AM, Danny Sullivan <ds...@hotmail.com>wrote:
> >
> >> Perhaps it would be best for me to take a step back and ask what problem
> >> are we trying to solve by using Camel? My understanding is that it is to
> >> allow distribution of Streams to multiple servers. It would be helpful to
> >> me to see another open source application that has this deployment
> >> structure (it doesn't have to use Camel, just an application that has
> >> multiple jars communicating with one another). Does anyone have any
> >> suggestions of software for me to look at?
> >>
> >
> > Danny, I wish I could just show you my system we are building. Honestly I
> > think you are looking at this the wrong way. You are looking at it from a
> > software development perspective (jars, protocols) and not from a
> > business/system perspective (messages, components). So let me use an
> > example.
> >
> > Let's say I have a business that takes order for widgets. I want the order
> > taking to be streamlined for the user. To make that happen my order entry
> > system does some basic checks when the user is placing the order and if
> > those pass it saves the order to the database. To kick off the other
> > processing, whenever I save an order to the database I sent a message to a
> > Topic (pub/sub) saying either an order was created or an order was updated.
> > Now at this point there is a lot more work I have to do to get the order
> > shipped to the customer but it's an async action to the order taking. At
> > this point the order has been placed and the customer can go about their
> > business.
> >
> > I also have a component I wrote that listens to the Topic for "Order
> > Created" messages. It takes each message, retrieves the order does some
> > inventory processing to get the order ready to ship. At this point the
> > order is complete and shipped to the customer.
> >
> > Now, in this simple example you could argue that the order entry system
> > could just call the order processing component directly but that would
> > preclude some extensibility in the future.
> >
> > A few months down the road I get a requirement to be able to notify the
> > sales team whenever an order is placed by a certain set of customers. In a
> > traditional model I'd have to modify the order entry component and put in
> > logic to handle that. In this architecture all I have to do is build a
> > different component that listens to that same Topic for order by certain
> > customers. When it gets a notification it can then do the logic to send the
> > message. The beauty of this is the order entry system doesn't' have to be
> > modified and there is no risk of inserting a bug into that core process.
> >
> > So you ask, where does camel come in here. It could be used in lots of
> > places but the natural fit is for the shipment processing component to be a
> > camel route that listens to the Topic and calls a java object when messages
> > come in. Also for the notification components, the whole thing could
> > probably be a camel route (0 Java code).
> >
> > This is a pretty simple example and there are lots of other benefits I
> > didn't talk about like redundancy and deployment flexibility.
> >
> > Does that help?
> >
> >>
> >> > Date: Thu, 31 Oct 2013 20:23:26 -0400
> >> > Subject: Re: [DISCUSS] Switching Streams from Camel deployment to .war
> >> deployment
> >> > From: jletourneau80@gmail.com
> >> > To: dev@streams.incubator.apache.org
> >> >
> >> > So Danny in case it was lost in the ensuing transport discussion,
> >> > Camel abstracts the transport, so you don't have to care if its http
> >> > or whatever, only about the message format.  This is helpful if you
> >> > care about plugging in components that DO care about the transport of
> >> > course.  To answer your other questions, the downside to having the
> >> > components all work via http (and not using messaging/Camel) is that
> >> > everything immediately becomes point to point, which isn't a
> >> > performance issue in its own right but limits scaling to more than one
> >> > component being at the receiving end/fulfillment for instance as we've
> >> > been discussing as being optimal.
> >> >
> >> > On Thu, Oct 31, 2013 at 5:38 PM, Matt Franklin <m....@gmail.com>
> >> wrote:
> >> > > On Thu, Oct 31, 2013 at 5:34 PM, Chris Geer <ch...@cxtsoftware.com>
> >> wrote:
> >> > >
> >> > >> On Thu, Oct 31, 2013 at 2:32 PM, Matt Franklin <
> >> m.ben.franklin@gmail.com
> >> > >> >wrote:
> >> > >>
> >> > >> > On Thu, Oct 31, 2013 at 5:28 PM, Chris Geer <ch...@cxtsoftware.com>
> >> > >> wrote:
> >> > >> >
> >> > >> > > On Thu, Oct 31, 2013 at 2:23 PM, Matt Franklin <
> >> > >> m.ben.franklin@gmail.com
> >> > >> > > >wrote:
> >> > >> > >
> >> > >> > > > On Thu, Oct 31, 2013 at 5:04 PM, Danny Sullivan <
> >> > >> > dsullivan7@hotmail.com
> >> > >> > > > >wrote:
> >> > >> > > >
> >> > >> > > > > I'm not quite following this, so I apologize. What I'm trying
> >> to do
> >> > >> > is
> >> > >> > > > > programmatically make a request to a jar running on a
> >> separate jvm
> >> > >> > and
> >> > >> > > > get
> >> > >> > > > > the response from that call all within the same method.
> >> Similar to
> >> > >> > this
> >> > >> > > > > http request:
> >> > >> > > > >
> >> > >> > > > > HttpGet httpget = new HttpGet();
> >> > >> > > > >
> >> > >> > > > > httpget.setURI(new URI("www.streams-persistence.com"));
> >> > >> > > > >
> >> > >> > > > > CloseableHttpResponse response = httpclient.execute(httpget);
> >> > >> > > > >
> >> > >> > > > > //do stuff with the response...
> >> > >> > > > >
> >> > >> > > > >
> >> > >> > > > >
> >> > >> > > > >
> >> > >> > > > > I imagine that this would translate to something in Camel
> >> similar
> >> > >> to
> >> > >> > > > this:
> >> > >> > > > >
> >> > >> > > > >
> >> > >> > > > > <route>
> >> > >> > > > >
> >> > >> > > > > <from uri="bean:subscriberService?method=getAllSubscribers"/>
> >> > >> > > > >
> >> > >> > > > > <inOut
> >> > >> > > > >
> >> > >> > > >
> >> > >> > >
> >> > >> >
> >> > >>
> >> uri="addressToStreamsPersistenceJar?method=selectAllSubscribersFromDatabase"/>
> >> > >> > > > >
> >> > >> > > > > </route>
> >> > >> > > > >
> >> > >> > > >
> >> > >> > > > Camel uses AMQP as a messaging system so that you don't have to
> >> > >> create
> >> > >> > > HTTP
> >> > >> > > > requests between sources, though it is possible to do so.  In
> >> Storm,
> >> > >> we
> >> > >> > > > usually try to use a buffer system like Kafka to do the same.
> >> > >> > > >
> >> > >> > >
> >> > >> > > Camel can use AMQP through ActiveMQ.
> >> > >> > >
> >> > >> > >
> >> > >> > Can being the operative word.  I haven't done much with Camel.  What
> >> > >> other
> >> > >> > transports are available.
> >> > >> >
> >> > >> > OpenWire (native ActiveMQ)
> >> > >> AMQP
> >> > >> STOMP
> >> > >> HTTP (can be a server or client)
> >> > >> Mail
> >> > >> Amazon SQS
> >> > >> XMPP
> >> > >> ....hundred or so more
> >> > >>
> >> > >> http://camel.apache.org/components.html
> >> > >
> >> > >
> >> > >
> >> > > Nice.  I was expecting RTFM, so thanks for catering to my laziness.
> >> > >
> >> > >
> >> > >>
> >> > >>
> >> > >> >
> >> > >> > > >
> >> > >> > > >
> >> > >> > > > >
> >> > >> > > > >
> >> > >> > > > > But it is unclear what the actual implementation would be.
> >> > >> > > > >
> >> > >> > > > >
> >> > >> > > > > This actually brings me to another suggestion. Would there be
> >> a big
> >> > >> > > > > performance impact to have communication between the software
> >> > >> > > components
> >> > >> > > > > occur between http? Say the 5 software components I outlined
> >> > >> earlier
> >> > >> > > were
> >> > >> > > > > packaged as 5 separate wars. These wars could communicate
> >> with each
> >> > >> > > other
> >> > >> > > > > via get a post requests. This sounds unconventional offhand
> >> so I'd
> >> > >> > like
> >> > >> > > > to
> >> > >> > > > > hear some thoughts on it.
> >> > >> > > > >
> >> > >> > > >
> >> > >> > > > It is most certainly possible, but IMO probably not the best
> >> option
> >> > >> for
> >> > >> > > > success.  Protocols like PubSubHubbub use HTTP for a message
> >> > >> transport.
> >> > >> > > >
> >> > >> > > >
> >> > >> > > > > -Danny
> >> > >> > > > >
> >> > >> > > > > > Date: Thu, 31 Oct 2013 13:28:41 -0700
> >> > >> > > > > > Subject: Re: [DISCUSS] Switching Streams from Camel
> >> deployment to
> >> > >> > > .war
> >> > >> > > > > deployment
> >> > >> > > > > > From: chris@cxtsoftware.com
> >> > >> > > > > > To: dev@streams.incubator.apache.org
> >> > >> > > > > >
> >> > >> > > > > > Or Content Enricher [2]
> >> > >> > > > > >
> >> > >> > > > > > [2] http://camel.apache.org/content-enricher.html
> >> > >> > > > > >
> >> > >> > > > > >
> >> > >> > > > > > On Thu, Oct 31, 2013 at 12:37 PM, Jason Letourneau
> >> > >> > > > > > <jl...@gmail.com>wrote:
> >> > >> > > > > >
> >> > >> > > > > > > check out the link here[1]
> >> > >> > > > > > >
> >> > >> > > > > > >
> >> > >> > > > > > > [1]http://camel.apache.org/request-reply.html
> >> > >> > > > > > >
> >> > >> > > > > > > On Thu, Oct 31, 2013 at 2:32 PM, Danny Sullivan <
> >> > >> > > > > dsullivan7@hotmail.com>
> >> > >> > > > > > > wrote:
> >> > >> > > > > > > > I have a quick Camel question that I arrived at in the
> >> > >> > > > > implementation of
> >> > >> > > > > > > these new components:
> >> > >> > > > > > > >
> >> > >> > > > > > > > Lets say I have a method in streams-activity.jar that
> >> needs
> >> > >> all
> >> > >> > > > > > > subscribers in the database. This would require a call to
> >> the
> >> > >> > > > > > > streams-persistence.jar. So far, I've seen camel used
> >> mostly
> >> > >> for
> >> > >> > > > > passing
> >> > >> > > > > > > data through the application, but not for making a single
> >> > >> > > > > request-reponse
> >> > >> > > > > > > from within a method. How can I use Camel to get a list
> >> of all
> >> > >> > > > > subscribers
> >> > >> > > > > > > in the streams-persistence.jar from the
> >> streams-activity.jar?
> >> > >> > > > > > > >
> >> > >> > > > > > > >> From: dsullivan7@hotmail.com
> >> > >> > > > > > > >> To: dev@streams.incubator.apache.org
> >> > >> > > > > > > >> Subject: RE: [DISCUSS] Switching Streams from Camel
> >> > >> deployment
> >> > >> > > to
> >> > >> > > > > .war
> >> > >> > > > > > > deployment
> >> > >> > > > > > > >> Date: Thu, 31 Oct 2013 08:40:44 -0400
> >> > >> > > > > > > >>
> >> > >> > > > > > > >> Excellent, I'll write up something as a proof of
> >> concept and
> >> > >> > we
> >> > >> > > > can
> >> > >> > > > > > > discuss further to make sure everything is vanilla.
> >> > >> > > > > > > >>
> >> > >> > > > > > > >> > Date: Wed, 30 Oct 2013 17:01:28 -0400
> >> > >> > > > > > > >> > Subject: Re: [DISCUSS] Switching Streams from Camel
> >> > >> > deployment
> >> > >> > > > to
> >> > >> > > > > > > .war deployment
> >> > >> > > > > > > >> > From: jletourneau80@gmail.com
> >> > >> > > > > > > >> > To: dev@streams.incubator.apache.org
> >> > >> > > > > > > >> >
> >> > >> > > > > > > >> > That sounds pretty promising to me.
> >> > >> > > > > > > >> >
> >> > >> > > > > > > >> > On Wed, Oct 30, 2013 at 2:38 PM, Danny Sullivan <
> >> > >> > > > > > > dsullivan7@hotmail.com> wrote:
> >> > >> > > > > > > >> > > Thanks for the feedback. You have an interesting
> >> point
> >> > >> > about
> >> > >> > > > the
> >> > >> > > > > > > url linking to a separate processing space. Let me tie my
> >> > >> answer
> >> > >> > > into
> >> > >> > > > > your
> >> > >> > > > > > > last question about "advocating for the simplicity at
> >> > >> > registration
> >> > >> > > to
> >> > >> > > > > give
> >> > >> > > > > > > up flexibility at registration, but retaining the inner
> >> "guts"
> >> > >> of
> >> > >> > > > > > > EIP/messaging". Consider a new architecture:
> >> > >> > > > > > > >> > >
> >> > >> > > > > > > >> > > streams-web.war: single entry point to
> >> application, but
> >> > >> > > > > functions
> >> > >> > > > > > > ONLY as an entry point. From here Camel routes the
> >> incoming
> >> > >> > > requests
> >> > >> > > > > to 4
> >> > >> > > > > > > separate jarssubscriber-registration.jar: subscriber
> >> > >> registration
> >> > >> > > > > > > publisher-registration.jar: publisher
> >> registrationactivity.jar:
> >> > >> > > > returns
> >> > >> > > > > > > activity (also contains subscriber warehouse and storm
> >> activity
> >> > >> > > > > > > aggregator)publish.jar: publishes
> >> > >> activitystreams-cassandra.jar:
> >> > >> > > the
> >> > >> > > > > above
> >> > >> > > > > > > 4 jars would all have a hook into this jar which would
> >> function
> >> > >> > as
> >> > >> > > a
> >> > >> > > > > hook
> >> > >> > > > > > > onto the database. Each jar would have camel route output
> >> to
> >> > >> this
> >> > >> > > > jar.
> >> > >> > > > > > > >> > >
> >> > >> > > > > > > >> > > In this implementation, Camel would no longer be
> >> the
> >> > >> entry
> >> > >> > > and
> >> > >> > > > > exit
> >> > >> > > > > > > point of a client to the application, but would handle the
> >> > >> > > > > communication
> >> > >> > > > > > > between components. The flow of activity through the
> >> > >> application
> >> > >> > > > would
> >> > >> > > > > be
> >> > >> > > > > > > method based in each jar. This would allow deployment on
> >> up to
> >> > >> 6
> >> > >> > > > > different
> >> > >> > > > > > > process spaces. However, this does not address that there
> >> is a
> >> > >> > > single
> >> > >> > > > > > > server entry point, but I'm not sure if it was a concern
> >> in the
> >> > >> > > first
> >> > >> > > > > place.
> >> > >> > > > > > > >> > >
> >> > >> > > > > > > >> > > My argument, at its basis, is that we should move
> >> away
> >> > >> > from
> >> > >> > > > > using
> >> > >> > > > > > > Camel as the entry point to the application. I would be
> >> happy
> >> > >> to
> >> > >> > > > > maintain
> >> > >> > > > > > > messaging between components.
> >> > >> > > > > > > >> > >
> >> > >> > > > > > > >> > >> Date: Wed, 30 Oct 2013 12:32:55 -0400
> >> > >> > > > > > > >> > >> Subject: Re: [DISCUSS] Switching Streams from
> >> Camel
> >> > >> > > > deployment
> >> > >> > > > > to
> >> > >> > > > > > > .war deployment
> >> > >> > > > > > > >> > >> From: jletourneau80@gmail.com
> >> > >> > > > > > > >> > >> To: dev@streams.incubator.apache.org
> >> > >> > > > > > > >> > >>
> >> > >> > > > > > > >> > >> An interesting use case that I am holding onto is
> >> the
> >> > >> > > ability
> >> > >> > > > > for
> >> > >> > > > > > > >> > >> publishers to register via a single URL
> >> (registration
> >> > >> > > > > endpoint),
> >> > >> > > > > > > but
> >> > >> > > > > > > >> > >> be sent a URL back to post to a different process
> >> space
> >> > >> > for
> >> > >> > > > > actual
> >> > >> > > > > > > >> > >> publishing.  The same is true on the subscriber
> >> front.
> >> > >> > > > >  Currently,
> >> > >> > > > > > > the
> >> > >> > > > > > > >> > >> Camel/EIP infrastructure abstracts this because
> >> > >> different
> >> > >> > > > > > > components
> >> > >> > > > > > > >> > >> deployed in different process spaces handling the
> >> route
> >> > >> > > > > creation
> >> > >> > > > > > > can
> >> > >> > > > > > > >> > >> just be bolted onto a running Streams instance
> >> without
> >> > >> > new
> >> > >> > > > > > > subs/pubs
> >> > >> > > > > > > >> > >> behaving any differently than existing.  This
> >> seems to
> >> > >> > be a
> >> > >> > > > > > > >> > >> potentially critical scaling point.  Is there a
> >> way to
> >> > >> do
> >> > >> > > > this
> >> > >> > > > > with
> >> > >> > > > > > > >> > >> the Spring solution?
> >> > >> > > > > > > >> > >>
> >> > >> > > > > > > >> > >> The persistence point is a good one, though I
> >> would
> >> > >> > > classify
> >> > >> > > > > that
> >> > >> > > > > > > as
> >> > >> > > > > > > >> > >> "not implemented" vs "not possible" (not that you
> >> > >> were).
> >> > >> > > > > > > >> > >>
> >> > >> > > > > > > >> > >> I'm not married to Camel, I just like the EIP
> >> approach
> >> > >> to
> >> > >> > > > > building
> >> > >> > > > > > > >> > >> something that is ultimately a messaging system.
> >>  There
> >> > >> > are
> >> > >> > > > > known
> >> > >> > > > > > > >> > >> patterns that solve at least a subset of the
> >> problems
> >> > >> > > Streams
> >> > >> > > > > is
> >> > >> > > > > > > >> > >> trying to solve and implementations that can
> >> handle the
> >> > >> > > load
> >> > >> > > > > and
> >> > >> > > > > > > I'll
> >> > >> > > > > > > >> > >> reiterate flexibility == complexity almost always.
> >> > >> > > > > > > >> > >>
> >> > >> > > > > > > >> > >> It comes right back to the central question: Do
> >> you
> >> > >> want
> >> > >> > > > > > > flexibility
> >> > >> > > > > > > >> > >> or simplicity?  It doesn't have to be black and
> >> white
> >> > >> > > either
> >> > >> > > > I
> >> > >> > > > > > > don't
> >> > >> > > > > > > >> > >> think...
> >> > >> > > > > > > >> > >>
> >> > >> > > > > > > >> > >> More pointedly: Where should we give up
> >> flexibility for
> >> > >> > > > > > > simplicity?  I
> >> > >> > > > > > > >> > >> read that Danny is advocating for the simplicity
> >> at
> >> > >> > > > > registration to
> >> > >> > > > > > > >> > >> give up flexibility at registration, but
> >> retaining the
> >> > >> > > inner
> >> > >> > > > > > > "guts" of
> >> > >> > > > > > > >> > >> EIP/messaging?  Thoughts?
> >> > >> > > > > > > >> > >>
> >> > >> > > > > > > >> > >> On Wed, Oct 30, 2013 at 11:37 AM, Danny Sullivan <
> >> > >> > > > > > > dsullivan7@hotmail.com> wrote:
> >> > >> > > > > > > >> > >> > My argument is not for the IoC pattern as that
> >> can be
> >> > >> > > (and
> >> > >> > > > > has
> >> > >> > > > > > > been) implemented alongside Camel. My main argument is
> >> that the
> >> > >> > > > syntax
> >> > >> > > > > at
> >> > >> > > > > > > the entry point is not only familiar but much simpler.
> >> This
> >> > >> > > wouldn't
> >> > >> > > > > be a
> >> > >> > > > > > > very strong argument if the Camel implementation wasn't
> >> much
> >> > >> more
> >> > >> > > > > > > complicated but I feel that it is the case. Also, looking
> >> > >> toward
> >> > >> > > the
> >> > >> > > > > > > future, if the server is restarted, in-routes are lost in
> >> > >> Camel.
> >> > >> > > The
> >> > >> > > > > way to
> >> > >> > > > > > > curb this is to persist the dynamic routes that Camel
> >> creates,
> >> > >> > and
> >> > >> > > > > then on
> >> > >> > > > > > > start up pull every one of these routes and recreate a
> >> dynamic
> >> > >> > > route
> >> > >> > > > > for
> >> > >> > > > > > > each one. Not only is this much easier to implement using
> >> the
> >> > >> > > Spring
> >> > >> > > > > web
> >> > >> > > > > > > implementation, but it already has been implemented and
> >> you can
> >> > >> > try
> >> > >> > > > it
> >> > >> > > > > by
> >> > >> > > > > > > checking out the webservice branch, registering a
> >> subscriber,
> >> > >> > > > > restarting
> >> > >> > > > > > > tomcat, and using the same url you had before. This will
> >> allow
> >> > >> > > > > subscribers
> >> > >> > > > > > > to hang on to their urls once they register. (the same is
> >> true
> >> > >> > for
> >> > >> > > > > > > publishers: you can post via the same url after restarting
> >> > >> > tomcat)
> >> > >> > > > > > > >> > >> >
> >> > >> > > > > > > >> > >> >> Date: Wed, 30 Oct 2013 11:00:21 -0400
> >> > >> > > > > > > >> > >> >> Subject: Re: [DISCUSS] Switching Streams from
> >> Camel
> >> > >> > > > > deployment
> >> > >> > > > > > > to .war deployment
> >> > >> > > > > > > >> > >> >> From: jletourneau80@gmail.com
> >> > >> > > > > > > >> > >> >> To: dev@streams.incubator.apache.org
> >> > >> > > > > > > >> > >> >>
> >> > >> > > > > > > >> > >> >> To be fair, while the current implementation is
> >> > >> > heavily
> >> > >> > > > > > > camel-based,
> >> > >> > > > > > > >> > >> >> all of the interfaces related to Streams
> >> > >> functionality
> >> > >> > > are
> >> > >> > > > > not.
> >> > >> > > > > > >  The
> >> > >> > > > > > > >> > >> >> current model maps to what Matt has outlined
> >> in my
> >> > >> > > > opinion,
> >> > >> > > > > > > though
> >> > >> > > > > > > >> > >> >> packing names etc. probably don't follow that
> >> exact
> >> > >> > > > pattern.
> >> > >> > > > > > > >> > >> >>
> >> > >> > > > > > > >> > >> >> With regards to the complexity and different
> >> > >> > components
> >> > >> > > in
> >> > >> > > > > the
> >> > >> > > > > > > >> > >> >> registration process, this was a cut at the
> >> > >> > abstraction
> >> > >> > > > > based
> >> > >> > > > > > > on the
> >> > >> > > > > > > >> > >> >> assumption that different implementations may
> >> be
> >> > >> > plugged
> >> > >> > > > in
> >> > >> > > > > and
> >> > >> > > > > > > in
> >> > >> > > > > > > >> > >> >> fact may live on different processor space
> >> (ie. a
> >> > >> > > polling
> >> > >> > > > > > > publisher vs
> >> > >> > > > > > > >> > >> >> a push publisher may be instantiated on
> >> different
> >> > >> > > servers
> >> > >> > > > > but
> >> > >> > > > > > > the
> >> > >> > > > > > > >> > >> >> registration URL is staticly defined).
> >> > >> > > > > > > >> > >> >>
> >> > >> > > > > > > >> > >> >> Is the main argument I am seeing for Spring
> >>  the
> >> > >> > > > > familiarity of
> >> > >> > > > > > > its
> >> > >> > > > > > > >> > >> >> IoC pattern implementation and syntax at the
> >> entry
> >> > >> > > point?
> >> > >> > > > > > > >> > >> >>
> >> > >> > > > > > > >> > >> >> On Wed, Oct 30, 2013 at 10:53 AM, Danny
> >> Sullivan <
> >> > >> > > > > > > dsullivan7@hotmail.com> wrote:
> >> > >> > > > > > > >> > >> >> > Could you clarify whether the same entry
> >> points
> >> > >> > would
> >> > >> > > > > exist
> >> > >> > > > > > > for the camel implementation of the core (implementing the
> >> > >> > > "process"
> >> > >> > > > > > > method/ using a DynammicRouteBuilder) or would the
> >> webservice
> >> > >> be
> >> > >> > > the
> >> > >> > > > > sole
> >> > >> > > > > > > entry point to Streams and after it enters would it hand
> >> it off
> >> > >> > to
> >> > >> > > > > Camel?
> >> > >> > > > > > > And what would be the entry point for the Storm
> >> implementation?
> >> > >> > > > > > > >> > >> >> > -Danny
> >> > >> > > > > > > >> > >> >> >
> >> > >> > > > > > > >> > >> >> >> Date: Wed, 30 Oct 2013 10:13:04 -0400
> >> > >> > > > > > > >> > >> >> >> Subject: Re: [DISCUSS] Switching Streams
> >> from
> >> > >> Camel
> >> > >> > > > > > > deployment to .war deployment
> >> > >> > > > > > > >> > >> >> >> From: m.ben.franklin@gmail.com
> >> > >> > > > > > > >> > >> >> >> To: dev@streams.incubator.apache.org
> >> > >> > > > > > > >> > >> >> >>
> >> > >> > > > > > > >> > >> >> >> On Tue, Oct 29, 2013 at 2:55 PM, Danny
> >> Sullivan <
> >> > >> > > > > > > dsullivan7@hotmail.com>wrote:
> >> > >> > > > > > > >> > >> >> >>
> >> > >> > > > > > > >> > >> >> >> > My case for switching from OSGi is for
> >> > >> simplicity
> >> > >> > > in
> >> > >> > > > > > > design. To follow the
> >> > >> > > > > > > >> > >> >> >> > path of an activity through Streams in the
> >> > >> > > > webservice,
> >> > >> > > > > > > there is one main
> >> > >> > > > > > > >> > >> >> >> > things the developer needs to understand:
> >> > >> > > > > > > >> > >> >> >> > The @RequestMapping annotation specifies
> >> the
> >> > >> HTTP
> >> > >> > > > entry
> >> > >> > > > > > > point
> >> > >> > > > > > > >> > >> >> >> > There are 4 @RequestMapping annotations
> >> that
> >> > >> > > > > correspond to
> >> > >> > > > > > > each of the 4
> >> > >> > > > > > > >> > >> >> >> > ways a user enters the application:
> >> > >> registering a
> >> > >> > > > > > > publisher, registering a
> >> > >> > > > > > > >> > >> >> >> > subscriber, publishing activity, and
> >> getting an
> >> > >> > > > > activity
> >> > >> > > > > > > stream. Where
> >> > >> > > > > > > >> > >> >> >> > these are located in the source code can
> >> be
> >> > >> found
> >> > >> > > by
> >> > >> > > > > > > searching for the
> >> > >> > > > > > > >> > >> >> >> > paths specified in the documentation
> >> (search
> >> > >> for
> >> > >> > > > > > > "/publisherRegister",
> >> > >> > > > > > > >> > >> >> >> > "/subscriberRegister", "/publishActivity",
> >> > >> > > > > "/getActivity"
> >> > >> > > > > > > which will all
> >> > >> > > > > > > >> > >> >> >> > lead you to StreamsWebController.java).
> >> From
> >> > >> the
> >> > >> > > > > methods
> >> > >> > > > > > > that process
> >> > >> > > > > > > >> > >> >> >> > requests, the flow through the
> >> application is
> >> > >> > > through
> >> > >> > > > > > > methods which can be
> >> > >> > > > > > > >> > >> >> >> > understood by most Java programmers.
> >> > >> > > > > > > >> > >> >> >> > The flow of activities through the current
> >> > >> trunk
> >> > >> > > > > branch is
> >> > >> > > > > > > understood as
> >> > >> > > > > > > >> > >> >> >> > follows:
> >> > >> > > > > > > >> > >> >> >> > The string "/publisher/register" (the
> >> entry
> >> > >> point
> >> > >> > > to
> >> > >> > > > > > > register a publisher
> >> > >> > > > > > > >> > >> >> >> > specified in the documentation) is the
> >> value of
> >> > >> > the
> >> > >> > > > > > > >> > >> >> >> > consumer.registrationEndpoint property
> >> defined
> >> > >> in
> >> > >> > > > > > > streams.propertiesThe
> >> > >> > > > > > > >> > >> >> >> > camelContext.xml specifies an endpoint
> >> with the
> >> > >> > id
> >> > >> > > > > > > >> > >> >> >> > consumerRegistrationEndpoint to have a uri
> >> > >> equal
> >> > >> > to
> >> > >> > > > the
> >> > >> > > > > > > propertyThe
> >> > >> > > > > > > >> > >> >> >> > consumerRegistrationEndpoint routes from
> >> uri
> >> > >> > > > > > > direct:publisher register with
> >> > >> > > > > > > >> > >> >> >> > the bean activityRegistrationProcessor
> >> nested
> >> > >> in
> >> > >> > > > > between
> >> > >> > > > > > > the routeThe
> >> > >> > > > > > > >> > >> >> >> > streams-eip-applicationContext contains a
> >> bean
> >> > >> > with
> >> > >> > > > > the id
> >> > >> > > > > > > >> > >> >> >> > activityRegistrationProcessor created for
> >> the
> >> > >> > class
> >> > >> > > > > > > >> > >> >> >> > ActivityPublisherRegistrationProcessorThe
> >> > >> > exchange
> >> > >> > > > will
> >> > >> > > > > > > enter the "process"
> >> > >> > > > > > > >> > >> >> >> > method of the class
> >> > >> > > > > > > ActivityPublisherRegisitrationProcessor and that this
> >> > >> > > > > > > >> > >> >> >> > is because this class implements the
> >> > >> "Processor"
> >> > >> > > > > interface
> >> > >> > > > > > > provided by
> >> > >> > > > > > > >> > >> >> >> > CamelThe direct:add-publisher-route takes
> >> the
> >> > >> > > > exchange
> >> > >> > > > > > > output from the
> >> > >> > > > > > > >> > >> >> >> > "process" method and routes it to the
> >> > >> > > > > > > activityRegistrationProcessor
> >> > >> > > > > > > >> > >> >> >> > "register" method. The bean
> >> > >> > > > > activityRegistrationProcessor
> >> > >> > > > > > > is defined in the
> >> > >> > > > > > > >> > >> >> >> > streams-eip-osgi-component-import.xmlThe
> >> output
> >> > >> > > from
> >> > >> > > > > this
> >> > >> > > > > > > method is then
> >> > >> > > > > > > >> > >> >> >> > sent to the "createNewRouteForConsumer"
> >> method
> >> > >> of
> >> > >> > > > > > > activityConsumerRouter.
> >> > >> > > > > > > >> > >> >> >> > This method creates a new route for the
> >> newly
> >> > >> > > > > registered
> >> > >> > > > > > > publisher using
> >> > >> > > > > > > >> > >> >> >> > the private static final class
> >> > >> > > > > DynamicConsumerRouteBuilder
> >> > >> > > > > > > which is
> >> > >> > > > > > > >> > >> >> >> > required to extend RouteBuilder which is
> >> > >> provided
> >> > >> > > by
> >> > >> > > > > > > Camel. This
> >> > >> > > > > > > >> > >> >> >> > DynamicConsumerRouteBuilder contains
> >> several
> >> > >> > > methods:
> >> > >> > > > > > > >> > >> >> >> > getConsumerReceiveMethod() (which
> >> corresponds
> >> > >> to
> >> > >> > > > @Value
> >> > >> > > > > > > >> > >> >> >> > ${consumer.receiveMethod} which
> >> corresponds to
> >> > >> > > > > "receive"),
> >> > >> > > > > > > >> > >> >> >> > getConsumerSplitMethod() (@Value
> >> > >> > > > > > > ${consumer.splitMethod},"split"), and
> >> > >> > > > > > > >> > >> >> >> > getConsumerActivityQUri() (@Value
> >> > >> > > > > ${consumerActivityQUri},
> >> > >> > > > > > > >> > >> >> >> > "direct:activityQ"). This is different
> >> than the
> >> > >> > > > > > > camelContext.xml in that
> >> > >> > > > > > > >> > >> >> >> > the route is being created
> >> programmatically.
> >> > >> What
> >> > >> > > > this
> >> > >> > > > > is
> >> > >> > > > > > > doing is routing
> >> > >> > > > > > > >> > >> >> >> > input from the inroute url (which Camel
> >> does
> >> > >> > > > > automatically
> >> > >> > > > > > > through the
> >> > >> > > > > > > >> > >> >> >> > configure method which is required to be
> >> > >> > > overridden),
> >> > >> > > > > to
> >> > >> > > > > > > the "receive"
> >> > >> > > > > > > >> > >> >> >> > method of ActivityConsumer, then to the
> >> "split"
> >> > >> > > > method
> >> > >> > > > > of
> >> > >> > > > > > > ActivityConsumer,
> >> > >> > > > > > > >> > >> >> >> > and then to the "direct:activityQ" which
> >> if you
> >> > >> > > look
> >> > >> > > > > back
> >> > >> > > > > > > in the
> >> > >> > > > > > > >> > >> >> >> > camelContext.xml routes to the
> >> > >> > > > > "activemq:queue:activities"
> >> > >> > > > > > > which then
> >> > >> > > > > > > >> > >> >> >> > routes to "receiveExchange"
> >> > >> > > > > > > >> > >> >> >> > This is the process to register a
> >> publisher.
> >> > >> The
> >> > >> > > > > process
> >> > >> > > > > > > for registering a
> >> > >> > > > > > > >> > >> >> >> > subscriber is relatively the same though
> >> it
> >> > >> > > involves
> >> > >> > > > > > > separate classes with
> >> > >> > > > > > > >> > >> >> >> > their own private static final
> >> RouteBuilder
> >> > >> > class.
> >> > >> > > > > From my
> >> > >> > > > > > > perspective, the
> >> > >> > > > > > > >> > >> >> >> > two most difficult things with setting
> >> this
> >> > >> > project
> >> > >> > > > up
> >> > >> > > > > > > were understanding
> >> > >> > > > > > > >> > >> >> >> > that the "process" method of the class
> >> that
> >> > >> > > > implements
> >> > >> > > > > > > "Processor" is the
> >> > >> > > > > > > >> > >> >> >> > entry point and the
> >> DynamicConsumerRouteBuilder
> >> > >> > > > creates
> >> > >> > > > > > > the second entry
> >> > >> > > > > > > >> > >> >> >> > point (The 5th and last points). This
> >> made the
> >> > >> > > > project
> >> > >> > > > > > > very, VERY hard to
> >> > >> > > > > > > >> > >> >> >> > understand.
> >> > >> > > > > > > >> > >> >> >> > In addition to simplicity of design, the
> >> mvn
> >> > >> > clean
> >> > >> > > > > install
> >> > >> > > > > > > of the web
> >> > >> > > > > > > >> > >> >> >> > service project is much faster and small
> >> scale
> >> > >> > > > activity
> >> > >> > > > > > > publishing is also
> >> > >> > > > > > > >> > >> >> >> > faster (see my email about load testing).
> >> These
> >> > >> > are
> >> > >> > > > > minor
> >> > >> > > > > > > points though as
> >> > >> > > > > > > >> > >> >> >> > compilation has no effect on deployment.
> >> OSGi
> >> > >> > does
> >> > >> > > > add
> >> > >> > > > > the
> >> > >> > > > > > > benefit of
> >> > >> > > > > > > >> > >> >> >> > modularized programming which is valuable,
> >> > >> > though I
> >> > >> > > > > think
> >> > >> > > > > > > the added
> >> > >> > > > > > > >> > >> >> >> > complexity of Camel merits moving the
> >> project
> >> > >> > away
> >> > >> > > > from
> >> > >> > > > > > > this paradigm.
> >> > >> > > > > > > >> > >> >> >> >
> >> > >> > > > > > > >> > >> >> >>
> >> > >> > > > > > > >> > >> >> >> I agree that the project is pretty
> >> difficult to
> >> > >> > > > > understand
> >> > >> > > > > > > ATM.  I think
> >> > >> > > > > > > >> > >> >> >> what we need to do is think about what the
> >> > >> > > > > responsibilities
> >> > >> > > > > > > of the code are
> >> > >> > > > > > > >> > >> >> >> and allow for different implementations
> >> that are
> >> > >> > not
> >> > >> > > so
> >> > >> > > > > > > tightly coupled as
> >> > >> > > > > > > >> > >> >> >> they are now.  For instance, having worked
> >> with
> >> > >> > Storm
> >> > >> > > > to
> >> > >> > > > > > > ingest millions of
> >> > >> > > > > > > >> > >> >> >> activities a day, I personally would like
> >> to see
> >> > >> > > > streams
> >> > >> > > > > be
> >> > >> > > > > > > responsible for
> >> > >> > > > > > > >> > >> >> >> defining an over-arching orchestration
> >> model that
> >> > >> > can
> >> > >> > > > be
> >> > >> > > > > > > implemented within
> >> > >> > > > > > > >> > >> >> >> a single war or on top of a distributed
> >> system.
> >> > >> >  This
> >> > >> > > > > would
> >> > >> > > > > > > look something
> >> > >> > > > > > > >> > >> >> >> like the follows:
> >> > >> > > > > > > >> > >> >> >>
> >> > >> > > > > > > >> > >> >> >> |__ Streams-Core  (Base classes, interfaces,
> >> > >> > > utilities,
> >> > >> > > > > > > extensions, etc)
> >> > >> > > > > > > >> > >> >> >> |
> >> > >> > > > > > > >> > >> >> >> |__ Streams-Storm (Storm implementation of
> >> the
> >> > >> > core)
> >> > >> > > > > > > >> > >> >> >> |
> >> > >> > > > > > > >> > >> >> >> |__ Streams-Camel (Camel implementation of
> >> the
> >> > >> > core)
> >> > >> > > > > > > >> > >> >> >> |
> >> > >> > > > > > > >> > >> >> >> |__ Streams-WS (Web service implementation)
> >> > >> > > > > > > >> > >> >> >>
> >> > >> > > > > > > >> > >> >> >>
> >> > >> > > > > > > >> > >> >> >> > Danny
> >> > >> > > > > > > >> > >> >> >> >
> >> > >> > > > > > > >> > >> >> >> >
> >> > >> > > > > > > >> > >> >> >> > > From: dsullivan7@hotmail.com
> >> > >> > > > > > > >> > >> >> >> > > To: dev@streams.incubator.apache.org
> >> > >> > > > > > > >> > >> >> >> > > Subject: [DISCUSS] Switching Streams
> >> from
> >> > >> Camel
> >> > >> > > > > > > deployment to .war
> >> > >> > > > > > > >> > >> >> >> > deployment
> >> > >> > > > > > > >> > >> >> >> > > Date: Tue, 29 Oct 2013 11:07:39 -0400
> >> > >> > > > > > > >> > >> >> >> > >
> >> > >> > > > > > > >> > >> >> >> > > The discussion thread for switching
> >> Streams
> >> > >> > from
> >> > >> > > > > > > Camel/osgi/Servicemix
> >> > >> > > > > > > >> > >> >> >> > to a single .war deployment
> >> > >> > > > > > > >> > >> >> >> >
> >> > >> > > > > > > >> > >> >> >> >
> >> > >> > > > > > > >> > >> >> >
> >> > >> > > > > > > >> > >> >
> >> > >> > > > > > > >> > >
> >> > >> > > > > > > >>
> >> > >> > > > > > > >
> >> > >> > > > > > >
> >> > >> > > > >
> >> > >> > > > >
> >> > >> > > >
> >> > >> > >
> >> > >> >
> >> > >>
> >>
> >>
 		 	   		  

Re: [DISCUSS] Switching Streams from Camel deployment to .war deployment

[Jason Letourneau <jl...@gmail.com>]

Nice Chris - well put.  Streams isn't an application (or shouldn't be
thought of as one) it is a system that potentially is part of a system
of systems.

To punctuate your example, I also like to recall the classic EIP
banking/loan example [3]

[3] http://www.eaipatterns.com/ComposedMessagingExample.html

On Fri, Nov 1, 2013 at 12:48 PM, Chris Geer <ch...@cxtsoftware.com> wrote:
> On Fri, Nov 1, 2013 at 6:56 AM, Danny Sullivan <ds...@hotmail.com>wrote:
>
>> Perhaps it would be best for me to take a step back and ask what problem
>> are we trying to solve by using Camel? My understanding is that it is to
>> allow distribution of Streams to multiple servers. It would be helpful to
>> me to see another open source application that has this deployment
>> structure (it doesn't have to use Camel, just an application that has
>> multiple jars communicating with one another). Does anyone have any
>> suggestions of software for me to look at?
>>
>
> Danny, I wish I could just show you my system we are building. Honestly I
> think you are looking at this the wrong way. You are looking at it from a
> software development perspective (jars, protocols) and not from a
> business/system perspective (messages, components). So let me use an
> example.
>
> Let's say I have a business that takes order for widgets. I want the order
> taking to be streamlined for the user. To make that happen my order entry
> system does some basic checks when the user is placing the order and if
> those pass it saves the order to the database. To kick off the other
> processing, whenever I save an order to the database I sent a message to a
> Topic (pub/sub) saying either an order was created or an order was updated.
> Now at this point there is a lot more work I have to do to get the order
> shipped to the customer but it's an async action to the order taking. At
> this point the order has been placed and the customer can go about their
> business.
>
> I also have a component I wrote that listens to the Topic for "Order
> Created" messages. It takes each message, retrieves the order does some
> inventory processing to get the order ready to ship. At this point the
> order is complete and shipped to the customer.
>
> Now, in this simple example you could argue that the order entry system
> could just call the order processing component directly but that would
> preclude some extensibility in the future.
>
> A few months down the road I get a requirement to be able to notify the
> sales team whenever an order is placed by a certain set of customers. In a
> traditional model I'd have to modify the order entry component and put in
> logic to handle that. In this architecture all I have to do is build a
> different component that listens to that same Topic for order by certain
> customers. When it gets a notification it can then do the logic to send the
> message. The beauty of this is the order entry system doesn't' have to be
> modified and there is no risk of inserting a bug into that core process.
>
> So you ask, where does camel come in here. It could be used in lots of
> places but the natural fit is for the shipment processing component to be a
> camel route that listens to the Topic and calls a java object when messages
> come in. Also for the notification components, the whole thing could
> probably be a camel route (0 Java code).
>
> This is a pretty simple example and there are lots of other benefits I
> didn't talk about like redundancy and deployment flexibility.
>
> Does that help?
>
>>
>> > Date: Thu, 31 Oct 2013 20:23:26 -0400
>> > Subject: Re: [DISCUSS] Switching Streams from Camel deployment to .war
>> deployment
>> > From: jletourneau80@gmail.com
>> > To: dev@streams.incubator.apache.org
>> >
>> > So Danny in case it was lost in the ensuing transport discussion,
>> > Camel abstracts the transport, so you don't have to care if its http
>> > or whatever, only about the message format.  This is helpful if you
>> > care about plugging in components that DO care about the transport of
>> > course.  To answer your other questions, the downside to having the
>> > components all work via http (and not using messaging/Camel) is that
>> > everything immediately becomes point to point, which isn't a
>> > performance issue in its own right but limits scaling to more than one
>> > component being at the receiving end/fulfillment for instance as we've
>> > been discussing as being optimal.
>> >
>> > On Thu, Oct 31, 2013 at 5:38 PM, Matt Franklin <m....@gmail.com>
>> wrote:
>> > > On Thu, Oct 31, 2013 at 5:34 PM, Chris Geer <ch...@cxtsoftware.com>
>> wrote:
>> > >
>> > >> On Thu, Oct 31, 2013 at 2:32 PM, Matt Franklin <
>> m.ben.franklin@gmail.com
>> > >> >wrote:
>> > >>
>> > >> > On Thu, Oct 31, 2013 at 5:28 PM, Chris Geer <ch...@cxtsoftware.com>
>> > >> wrote:
>> > >> >
>> > >> > > On Thu, Oct 31, 2013 at 2:23 PM, Matt Franklin <
>> > >> m.ben.franklin@gmail.com
>> > >> > > >wrote:
>> > >> > >
>> > >> > > > On Thu, Oct 31, 2013 at 5:04 PM, Danny Sullivan <
>> > >> > dsullivan7@hotmail.com
>> > >> > > > >wrote:
>> > >> > > >
>> > >> > > > > I'm not quite following this, so I apologize. What I'm trying
>> to do
>> > >> > is
>> > >> > > > > programmatically make a request to a jar running on a
>> separate jvm
>> > >> > and
>> > >> > > > get
>> > >> > > > > the response from that call all within the same method.
>> Similar to
>> > >> > this
>> > >> > > > > http request:
>> > >> > > > >
>> > >> > > > > HttpGet httpget = new HttpGet();
>> > >> > > > >
>> > >> > > > > httpget.setURI(new URI("www.streams-persistence.com"));
>> > >> > > > >
>> > >> > > > > CloseableHttpResponse response = httpclient.execute(httpget);
>> > >> > > > >
>> > >> > > > > //do stuff with the response...
>> > >> > > > >
>> > >> > > > >
>> > >> > > > >
>> > >> > > > >
>> > >> > > > > I imagine that this would translate to something in Camel
>> similar
>> > >> to
>> > >> > > > this:
>> > >> > > > >
>> > >> > > > >
>> > >> > > > > <route>
>> > >> > > > >
>> > >> > > > > <from uri="bean:subscriberService?method=getAllSubscribers"/>
>> > >> > > > >
>> > >> > > > > <inOut
>> > >> > > > >
>> > >> > > >
>> > >> > >
>> > >> >
>> > >>
>> uri="addressToStreamsPersistenceJar?method=selectAllSubscribersFromDatabase"/>
>> > >> > > > >
>> > >> > > > > </route>
>> > >> > > > >
>> > >> > > >
>> > >> > > > Camel uses AMQP as a messaging system so that you don't have to
>> > >> create
>> > >> > > HTTP
>> > >> > > > requests between sources, though it is possible to do so.  In
>> Storm,
>> > >> we
>> > >> > > > usually try to use a buffer system like Kafka to do the same.
>> > >> > > >
>> > >> > >
>> > >> > > Camel can use AMQP through ActiveMQ.
>> > >> > >
>> > >> > >
>> > >> > Can being the operative word.  I haven't done much with Camel.  What
>> > >> other
>> > >> > transports are available.
>> > >> >
>> > >> > OpenWire (native ActiveMQ)
>> > >> AMQP
>> > >> STOMP
>> > >> HTTP (can be a server or client)
>> > >> Mail
>> > >> Amazon SQS
>> > >> XMPP
>> > >> ....hundred or so more
>> > >>
>> > >> http://camel.apache.org/components.html
>> > >
>> > >
>> > >
>> > > Nice.  I was expecting RTFM, so thanks for catering to my laziness.
>> > >
>> > >
>> > >>
>> > >>
>> > >> >
>> > >> > > >
>> > >> > > >
>> > >> > > > >
>> > >> > > > >
>> > >> > > > > But it is unclear what the actual implementation would be.
>> > >> > > > >
>> > >> > > > >
>> > >> > > > > This actually brings me to another suggestion. Would there be
>> a big
>> > >> > > > > performance impact to have communication between the software
>> > >> > > components
>> > >> > > > > occur between http? Say the 5 software components I outlined
>> > >> earlier
>> > >> > > were
>> > >> > > > > packaged as 5 separate wars. These wars could communicate
>> with each
>> > >> > > other
>> > >> > > > > via get a post requests. This sounds unconventional offhand
>> so I'd
>> > >> > like
>> > >> > > > to
>> > >> > > > > hear some thoughts on it.
>> > >> > > > >
>> > >> > > >
>> > >> > > > It is most certainly possible, but IMO probably not the best
>> option
>> > >> for
>> > >> > > > success.  Protocols like PubSubHubbub use HTTP for a message
>> > >> transport.
>> > >> > > >
>> > >> > > >
>> > >> > > > > -Danny
>> > >> > > > >
>> > >> > > > > > Date: Thu, 31 Oct 2013 13:28:41 -0700
>> > >> > > > > > Subject: Re: [DISCUSS] Switching Streams from Camel
>> deployment to
>> > >> > > .war
>> > >> > > > > deployment
>> > >> > > > > > From: chris@cxtsoftware.com
>> > >> > > > > > To: dev@streams.incubator.apache.org
>> > >> > > > > >
>> > >> > > > > > Or Content Enricher [2]
>> > >> > > > > >
>> > >> > > > > > [2] http://camel.apache.org/content-enricher.html
>> > >> > > > > >
>> > >> > > > > >
>> > >> > > > > > On Thu, Oct 31, 2013 at 12:37 PM, Jason Letourneau
>> > >> > > > > > <jl...@gmail.com>wrote:
>> > >> > > > > >
>> > >> > > > > > > check out the link here[1]
>> > >> > > > > > >
>> > >> > > > > > >
>> > >> > > > > > > [1]http://camel.apache.org/request-reply.html
>> > >> > > > > > >
>> > >> > > > > > > On Thu, Oct 31, 2013 at 2:32 PM, Danny Sullivan <
>> > >> > > > > dsullivan7@hotmail.com>
>> > >> > > > > > > wrote:
>> > >> > > > > > > > I have a quick Camel question that I arrived at in the
>> > >> > > > > implementation of
>> > >> > > > > > > these new components:
>> > >> > > > > > > >
>> > >> > > > > > > > Lets say I have a method in streams-activity.jar that
>> needs
>> > >> all
>> > >> > > > > > > subscribers in the database. This would require a call to
>> the
>> > >> > > > > > > streams-persistence.jar. So far, I've seen camel used
>> mostly
>> > >> for
>> > >> > > > > passing
>> > >> > > > > > > data through the application, but not for making a single
>> > >> > > > > request-reponse
>> > >> > > > > > > from within a method. How can I use Camel to get a list
>> of all
>> > >> > > > > subscribers
>> > >> > > > > > > in the streams-persistence.jar from the
>> streams-activity.jar?
>> > >> > > > > > > >
>> > >> > > > > > > >> From: dsullivan7@hotmail.com
>> > >> > > > > > > >> To: dev@streams.incubator.apache.org
>> > >> > > > > > > >> Subject: RE: [DISCUSS] Switching Streams from Camel
>> > >> deployment
>> > >> > > to
>> > >> > > > > .war
>> > >> > > > > > > deployment
>> > >> > > > > > > >> Date: Thu, 31 Oct 2013 08:40:44 -0400
>> > >> > > > > > > >>
>> > >> > > > > > > >> Excellent, I'll write up something as a proof of
>> concept and
>> > >> > we
>> > >> > > > can
>> > >> > > > > > > discuss further to make sure everything is vanilla.
>> > >> > > > > > > >>
>> > >> > > > > > > >> > Date: Wed, 30 Oct 2013 17:01:28 -0400
>> > >> > > > > > > >> > Subject: Re: [DISCUSS] Switching Streams from Camel
>> > >> > deployment
>> > >> > > > to
>> > >> > > > > > > .war deployment
>> > >> > > > > > > >> > From: jletourneau80@gmail.com
>> > >> > > > > > > >> > To: dev@streams.incubator.apache.org
>> > >> > > > > > > >> >
>> > >> > > > > > > >> > That sounds pretty promising to me.
>> > >> > > > > > > >> >
>> > >> > > > > > > >> > On Wed, Oct 30, 2013 at 2:38 PM, Danny Sullivan <
>> > >> > > > > > > dsullivan7@hotmail.com> wrote:
>> > >> > > > > > > >> > > Thanks for the feedback. You have an interesting
>> point
>> > >> > about
>> > >> > > > the
>> > >> > > > > > > url linking to a separate processing space. Let me tie my
>> > >> answer
>> > >> > > into
>> > >> > > > > your
>> > >> > > > > > > last question about "advocating for the simplicity at
>> > >> > registration
>> > >> > > to
>> > >> > > > > give
>> > >> > > > > > > up flexibility at registration, but retaining the inner
>> "guts"
>> > >> of
>> > >> > > > > > > EIP/messaging". Consider a new architecture:
>> > >> > > > > > > >> > >
>> > >> > > > > > > >> > > streams-web.war: single entry point to
>> application, but
>> > >> > > > > functions
>> > >> > > > > > > ONLY as an entry point. From here Camel routes the
>> incoming
>> > >> > > requests
>> > >> > > > > to 4
>> > >> > > > > > > separate jarssubscriber-registration.jar: subscriber
>> > >> registration
>> > >> > > > > > > publisher-registration.jar: publisher
>> registrationactivity.jar:
>> > >> > > > returns
>> > >> > > > > > > activity (also contains subscriber warehouse and storm
>> activity
>> > >> > > > > > > aggregator)publish.jar: publishes
>> > >> activitystreams-cassandra.jar:
>> > >> > > the
>> > >> > > > > above
>> > >> > > > > > > 4 jars would all have a hook into this jar which would
>> function
>> > >> > as
>> > >> > > a
>> > >> > > > > hook
>> > >> > > > > > > onto the database. Each jar would have camel route output
>> to
>> > >> this
>> > >> > > > jar.
>> > >> > > > > > > >> > >
>> > >> > > > > > > >> > > In this implementation, Camel would no longer be
>> the
>> > >> entry
>> > >> > > and
>> > >> > > > > exit
>> > >> > > > > > > point of a client to the application, but would handle the
>> > >> > > > > communication
>> > >> > > > > > > between components. The flow of activity through the
>> > >> application
>> > >> > > > would
>> > >> > > > > be
>> > >> > > > > > > method based in each jar. This would allow deployment on
>> up to
>> > >> 6
>> > >> > > > > different
>> > >> > > > > > > process spaces. However, this does not address that there
>> is a
>> > >> > > single
>> > >> > > > > > > server entry point, but I'm not sure if it was a concern
>> in the
>> > >> > > first
>> > >> > > > > place.
>> > >> > > > > > > >> > >
>> > >> > > > > > > >> > > My argument, at its basis, is that we should move
>> away
>> > >> > from
>> > >> > > > > using
>> > >> > > > > > > Camel as the entry point to the application. I would be
>> happy
>> > >> to
>> > >> > > > > maintain
>> > >> > > > > > > messaging between components.
>> > >> > > > > > > >> > >
>> > >> > > > > > > >> > >> Date: Wed, 30 Oct 2013 12:32:55 -0400
>> > >> > > > > > > >> > >> Subject: Re: [DISCUSS] Switching Streams from
>> Camel
>> > >> > > > deployment
>> > >> > > > > to
>> > >> > > > > > > .war deployment
>> > >> > > > > > > >> > >> From: jletourneau80@gmail.com
>> > >> > > > > > > >> > >> To: dev@streams.incubator.apache.org
>> > >> > > > > > > >> > >>
>> > >> > > > > > > >> > >> An interesting use case that I am holding onto is
>> the
>> > >> > > ability
>> > >> > > > > for
>> > >> > > > > > > >> > >> publishers to register via a single URL
>> (registration
>> > >> > > > > endpoint),
>> > >> > > > > > > but
>> > >> > > > > > > >> > >> be sent a URL back to post to a different process
>> space
>> > >> > for
>> > >> > > > > actual
>> > >> > > > > > > >> > >> publishing.  The same is true on the subscriber
>> front.
>> > >> > > > >  Currently,
>> > >> > > > > > > the
>> > >> > > > > > > >> > >> Camel/EIP infrastructure abstracts this because
>> > >> different
>> > >> > > > > > > components
>> > >> > > > > > > >> > >> deployed in different process spaces handling the
>> route
>> > >> > > > > creation
>> > >> > > > > > > can
>> > >> > > > > > > >> > >> just be bolted onto a running Streams instance
>> without
>> > >> > new
>> > >> > > > > > > subs/pubs
>> > >> > > > > > > >> > >> behaving any differently than existing.  This
>> seems to
>> > >> > be a
>> > >> > > > > > > >> > >> potentially critical scaling point.  Is there a
>> way to
>> > >> do
>> > >> > > > this
>> > >> > > > > with
>> > >> > > > > > > >> > >> the Spring solution?
>> > >> > > > > > > >> > >>
>> > >> > > > > > > >> > >> The persistence point is a good one, though I
>> would
>> > >> > > classify
>> > >> > > > > that
>> > >> > > > > > > as
>> > >> > > > > > > >> > >> "not implemented" vs "not possible" (not that you
>> > >> were).
>> > >> > > > > > > >> > >>
>> > >> > > > > > > >> > >> I'm not married to Camel, I just like the EIP
>> approach
>> > >> to
>> > >> > > > > building
>> > >> > > > > > > >> > >> something that is ultimately a messaging system.
>>  There
>> > >> > are
>> > >> > > > > known
>> > >> > > > > > > >> > >> patterns that solve at least a subset of the
>> problems
>> > >> > > Streams
>> > >> > > > > is
>> > >> > > > > > > >> > >> trying to solve and implementations that can
>> handle the
>> > >> > > load
>> > >> > > > > and
>> > >> > > > > > > I'll
>> > >> > > > > > > >> > >> reiterate flexibility == complexity almost always.
>> > >> > > > > > > >> > >>
>> > >> > > > > > > >> > >> It comes right back to the central question: Do
>> you
>> > >> want
>> > >> > > > > > > flexibility
>> > >> > > > > > > >> > >> or simplicity?  It doesn't have to be black and
>> white
>> > >> > > either
>> > >> > > > I
>> > >> > > > > > > don't
>> > >> > > > > > > >> > >> think...
>> > >> > > > > > > >> > >>
>> > >> > > > > > > >> > >> More pointedly: Where should we give up
>> flexibility for
>> > >> > > > > > > simplicity?  I
>> > >> > > > > > > >> > >> read that Danny is advocating for the simplicity
>> at
>> > >> > > > > registration to
>> > >> > > > > > > >> > >> give up flexibility at registration, but
>> retaining the
>> > >> > > inner
>> > >> > > > > > > "guts" of
>> > >> > > > > > > >> > >> EIP/messaging?  Thoughts?
>> > >> > > > > > > >> > >>
>> > >> > > > > > > >> > >> On Wed, Oct 30, 2013 at 11:37 AM, Danny Sullivan <
>> > >> > > > > > > dsullivan7@hotmail.com> wrote:
>> > >> > > > > > > >> > >> > My argument is not for the IoC pattern as that
>> can be
>> > >> > > (and
>> > >> > > > > has
>> > >> > > > > > > been) implemented alongside Camel. My main argument is
>> that the
>> > >> > > > syntax
>> > >> > > > > at
>> > >> > > > > > > the entry point is not only familiar but much simpler.
>> This
>> > >> > > wouldn't
>> > >> > > > > be a
>> > >> > > > > > > very strong argument if the Camel implementation wasn't
>> much
>> > >> more
>> > >> > > > > > > complicated but I feel that it is the case. Also, looking
>> > >> toward
>> > >> > > the
>> > >> > > > > > > future, if the server is restarted, in-routes are lost in
>> > >> Camel.
>> > >> > > The
>> > >> > > > > way to
>> > >> > > > > > > curb this is to persist the dynamic routes that Camel
>> creates,
>> > >> > and
>> > >> > > > > then on
>> > >> > > > > > > start up pull every one of these routes and recreate a
>> dynamic
>> > >> > > route
>> > >> > > > > for
>> > >> > > > > > > each one. Not only is this much easier to implement using
>> the
>> > >> > > Spring
>> > >> > > > > web
>> > >> > > > > > > implementation, but it already has been implemented and
>> you can
>> > >> > try
>> > >> > > > it
>> > >> > > > > by
>> > >> > > > > > > checking out the webservice branch, registering a
>> subscriber,
>> > >> > > > > restarting
>> > >> > > > > > > tomcat, and using the same url you had before. This will
>> allow
>> > >> > > > > subscribers
>> > >> > > > > > > to hang on to their urls once they register. (the same is
>> true
>> > >> > for
>> > >> > > > > > > publishers: you can post via the same url after restarting
>> > >> > tomcat)
>> > >> > > > > > > >> > >> >
>> > >> > > > > > > >> > >> >> Date: Wed, 30 Oct 2013 11:00:21 -0400
>> > >> > > > > > > >> > >> >> Subject: Re: [DISCUSS] Switching Streams from
>> Camel
>> > >> > > > > deployment
>> > >> > > > > > > to .war deployment
>> > >> > > > > > > >> > >> >> From: jletourneau80@gmail.com
>> > >> > > > > > > >> > >> >> To: dev@streams.incubator.apache.org
>> > >> > > > > > > >> > >> >>
>> > >> > > > > > > >> > >> >> To be fair, while the current implementation is
>> > >> > heavily
>> > >> > > > > > > camel-based,
>> > >> > > > > > > >> > >> >> all of the interfaces related to Streams
>> > >> functionality
>> > >> > > are
>> > >> > > > > not.
>> > >> > > > > > >  The
>> > >> > > > > > > >> > >> >> current model maps to what Matt has outlined
>> in my
>> > >> > > > opinion,
>> > >> > > > > > > though
>> > >> > > > > > > >> > >> >> packing names etc. probably don't follow that
>> exact
>> > >> > > > pattern.
>> > >> > > > > > > >> > >> >>
>> > >> > > > > > > >> > >> >> With regards to the complexity and different
>> > >> > components
>> > >> > > in
>> > >> > > > > the
>> > >> > > > > > > >> > >> >> registration process, this was a cut at the
>> > >> > abstraction
>> > >> > > > > based
>> > >> > > > > > > on the
>> > >> > > > > > > >> > >> >> assumption that different implementations may
>> be
>> > >> > plugged
>> > >> > > > in
>> > >> > > > > and
>> > >> > > > > > > in
>> > >> > > > > > > >> > >> >> fact may live on different processor space
>> (ie. a
>> > >> > > polling
>> > >> > > > > > > publisher vs
>> > >> > > > > > > >> > >> >> a push publisher may be instantiated on
>> different
>> > >> > > servers
>> > >> > > > > but
>> > >> > > > > > > the
>> > >> > > > > > > >> > >> >> registration URL is staticly defined).
>> > >> > > > > > > >> > >> >>
>> > >> > > > > > > >> > >> >> Is the main argument I am seeing for Spring
>>  the
>> > >> > > > > familiarity of
>> > >> > > > > > > its
>> > >> > > > > > > >> > >> >> IoC pattern implementation and syntax at the
>> entry
>> > >> > > point?
>> > >> > > > > > > >> > >> >>
>> > >> > > > > > > >> > >> >> On Wed, Oct 30, 2013 at 10:53 AM, Danny
>> Sullivan <
>> > >> > > > > > > dsullivan7@hotmail.com> wrote:
>> > >> > > > > > > >> > >> >> > Could you clarify whether the same entry
>> points
>> > >> > would
>> > >> > > > > exist
>> > >> > > > > > > for the camel implementation of the core (implementing the
>> > >> > > "process"
>> > >> > > > > > > method/ using a DynammicRouteBuilder) or would the
>> webservice
>> > >> be
>> > >> > > the
>> > >> > > > > sole
>> > >> > > > > > > entry point to Streams and after it enters would it hand
>> it off
>> > >> > to
>> > >> > > > > Camel?
>> > >> > > > > > > And what would be the entry point for the Storm
>> implementation?
>> > >> > > > > > > >> > >> >> > -Danny
>> > >> > > > > > > >> > >> >> >
>> > >> > > > > > > >> > >> >> >> Date: Wed, 30 Oct 2013 10:13:04 -0400
>> > >> > > > > > > >> > >> >> >> Subject: Re: [DISCUSS] Switching Streams
>> from
>> > >> Camel
>> > >> > > > > > > deployment to .war deployment
>> > >> > > > > > > >> > >> >> >> From: m.ben.franklin@gmail.com
>> > >> > > > > > > >> > >> >> >> To: dev@streams.incubator.apache.org
>> > >> > > > > > > >> > >> >> >>
>> > >> > > > > > > >> > >> >> >> On Tue, Oct 29, 2013 at 2:55 PM, Danny
>> Sullivan <
>> > >> > > > > > > dsullivan7@hotmail.com>wrote:
>> > >> > > > > > > >> > >> >> >>
>> > >> > > > > > > >> > >> >> >> > My case for switching from OSGi is for
>> > >> simplicity
>> > >> > > in
>> > >> > > > > > > design. To follow the
>> > >> > > > > > > >> > >> >> >> > path of an activity through Streams in the
>> > >> > > > webservice,
>> > >> > > > > > > there is one main
>> > >> > > > > > > >> > >> >> >> > things the developer needs to understand:
>> > >> > > > > > > >> > >> >> >> > The @RequestMapping annotation specifies
>> the
>> > >> HTTP
>> > >> > > > entry
>> > >> > > > > > > point
>> > >> > > > > > > >> > >> >> >> > There are 4 @RequestMapping annotations
>> that
>> > >> > > > > correspond to
>> > >> > > > > > > each of the 4
>> > >> > > > > > > >> > >> >> >> > ways a user enters the application:
>> > >> registering a
>> > >> > > > > > > publisher, registering a
>> > >> > > > > > > >> > >> >> >> > subscriber, publishing activity, and
>> getting an
>> > >> > > > > activity
>> > >> > > > > > > stream. Where
>> > >> > > > > > > >> > >> >> >> > these are located in the source code can
>> be
>> > >> found
>> > >> > > by
>> > >> > > > > > > searching for the
>> > >> > > > > > > >> > >> >> >> > paths specified in the documentation
>> (search
>> > >> for
>> > >> > > > > > > "/publisherRegister",
>> > >> > > > > > > >> > >> >> >> > "/subscriberRegister", "/publishActivity",
>> > >> > > > > "/getActivity"
>> > >> > > > > > > which will all
>> > >> > > > > > > >> > >> >> >> > lead you to StreamsWebController.java).
>> From
>> > >> the
>> > >> > > > > methods
>> > >> > > > > > > that process
>> > >> > > > > > > >> > >> >> >> > requests, the flow through the
>> application is
>> > >> > > through
>> > >> > > > > > > methods which can be
>> > >> > > > > > > >> > >> >> >> > understood by most Java programmers.
>> > >> > > > > > > >> > >> >> >> > The flow of activities through the current
>> > >> trunk
>> > >> > > > > branch is
>> > >> > > > > > > understood as
>> > >> > > > > > > >> > >> >> >> > follows:
>> > >> > > > > > > >> > >> >> >> > The string "/publisher/register" (the
>> entry
>> > >> point
>> > >> > > to
>> > >> > > > > > > register a publisher
>> > >> > > > > > > >> > >> >> >> > specified in the documentation) is the
>> value of
>> > >> > the
>> > >> > > > > > > >> > >> >> >> > consumer.registrationEndpoint property
>> defined
>> > >> in
>> > >> > > > > > > streams.propertiesThe
>> > >> > > > > > > >> > >> >> >> > camelContext.xml specifies an endpoint
>> with the
>> > >> > id
>> > >> > > > > > > >> > >> >> >> > consumerRegistrationEndpoint to have a uri
>> > >> equal
>> > >> > to
>> > >> > > > the
>> > >> > > > > > > propertyThe
>> > >> > > > > > > >> > >> >> >> > consumerRegistrationEndpoint routes from
>> uri
>> > >> > > > > > > direct:publisher register with
>> > >> > > > > > > >> > >> >> >> > the bean activityRegistrationProcessor
>> nested
>> > >> in
>> > >> > > > > between
>> > >> > > > > > > the routeThe
>> > >> > > > > > > >> > >> >> >> > streams-eip-applicationContext contains a
>> bean
>> > >> > with
>> > >> > > > > the id
>> > >> > > > > > > >> > >> >> >> > activityRegistrationProcessor created for
>> the
>> > >> > class
>> > >> > > > > > > >> > >> >> >> > ActivityPublisherRegistrationProcessorThe
>> > >> > exchange
>> > >> > > > will
>> > >> > > > > > > enter the "process"
>> > >> > > > > > > >> > >> >> >> > method of the class
>> > >> > > > > > > ActivityPublisherRegisitrationProcessor and that this
>> > >> > > > > > > >> > >> >> >> > is because this class implements the
>> > >> "Processor"
>> > >> > > > > interface
>> > >> > > > > > > provided by
>> > >> > > > > > > >> > >> >> >> > CamelThe direct:add-publisher-route takes
>> the
>> > >> > > > exchange
>> > >> > > > > > > output from the
>> > >> > > > > > > >> > >> >> >> > "process" method and routes it to the
>> > >> > > > > > > activityRegistrationProcessor
>> > >> > > > > > > >> > >> >> >> > "register" method. The bean
>> > >> > > > > activityRegistrationProcessor
>> > >> > > > > > > is defined in the
>> > >> > > > > > > >> > >> >> >> > streams-eip-osgi-component-import.xmlThe
>> output
>> > >> > > from
>> > >> > > > > this
>> > >> > > > > > > method is then
>> > >> > > > > > > >> > >> >> >> > sent to the "createNewRouteForConsumer"
>> method
>> > >> of
>> > >> > > > > > > activityConsumerRouter.
>> > >> > > > > > > >> > >> >> >> > This method creates a new route for the
>> newly
>> > >> > > > > registered
>> > >> > > > > > > publisher using
>> > >> > > > > > > >> > >> >> >> > the private static final class
>> > >> > > > > DynamicConsumerRouteBuilder
>> > >> > > > > > > which is
>> > >> > > > > > > >> > >> >> >> > required to extend RouteBuilder which is
>> > >> provided
>> > >> > > by
>> > >> > > > > > > Camel. This
>> > >> > > > > > > >> > >> >> >> > DynamicConsumerRouteBuilder contains
>> several
>> > >> > > methods:
>> > >> > > > > > > >> > >> >> >> > getConsumerReceiveMethod() (which
>> corresponds
>> > >> to
>> > >> > > > @Value
>> > >> > > > > > > >> > >> >> >> > ${consumer.receiveMethod} which
>> corresponds to
>> > >> > > > > "receive"),
>> > >> > > > > > > >> > >> >> >> > getConsumerSplitMethod() (@Value
>> > >> > > > > > > ${consumer.splitMethod},"split"), and
>> > >> > > > > > > >> > >> >> >> > getConsumerActivityQUri() (@Value
>> > >> > > > > ${consumerActivityQUri},
>> > >> > > > > > > >> > >> >> >> > "direct:activityQ"). This is different
>> than the
>> > >> > > > > > > camelContext.xml in that
>> > >> > > > > > > >> > >> >> >> > the route is being created
>> programmatically.
>> > >> What
>> > >> > > > this
>> > >> > > > > is
>> > >> > > > > > > doing is routing
>> > >> > > > > > > >> > >> >> >> > input from the inroute url (which Camel
>> does
>> > >> > > > > automatically
>> > >> > > > > > > through the
>> > >> > > > > > > >> > >> >> >> > configure method which is required to be
>> > >> > > overridden),
>> > >> > > > > to
>> > >> > > > > > > the "receive"
>> > >> > > > > > > >> > >> >> >> > method of ActivityConsumer, then to the
>> "split"
>> > >> > > > method
>> > >> > > > > of
>> > >> > > > > > > ActivityConsumer,
>> > >> > > > > > > >> > >> >> >> > and then to the "direct:activityQ" which
>> if you
>> > >> > > look
>> > >> > > > > back
>> > >> > > > > > > in the
>> > >> > > > > > > >> > >> >> >> > camelContext.xml routes to the
>> > >> > > > > "activemq:queue:activities"
>> > >> > > > > > > which then
>> > >> > > > > > > >> > >> >> >> > routes to "receiveExchange"
>> > >> > > > > > > >> > >> >> >> > This is the process to register a
>> publisher.
>> > >> The
>> > >> > > > > process
>> > >> > > > > > > for registering a
>> > >> > > > > > > >> > >> >> >> > subscriber is relatively the same though
>> it
>> > >> > > involves
>> > >> > > > > > > separate classes with
>> > >> > > > > > > >> > >> >> >> > their own private static final
>> RouteBuilder
>> > >> > class.
>> > >> > > > > From my
>> > >> > > > > > > perspective, the
>> > >> > > > > > > >> > >> >> >> > two most difficult things with setting
>> this
>> > >> > project
>> > >> > > > up
>> > >> > > > > > > were understanding
>> > >> > > > > > > >> > >> >> >> > that the "process" method of the class
>> that
>> > >> > > > implements
>> > >> > > > > > > "Processor" is the
>> > >> > > > > > > >> > >> >> >> > entry point and the
>> DynamicConsumerRouteBuilder
>> > >> > > > creates
>> > >> > > > > > > the second entry
>> > >> > > > > > > >> > >> >> >> > point (The 5th and last points). This
>> made the
>> > >> > > > project
>> > >> > > > > > > very, VERY hard to
>> > >> > > > > > > >> > >> >> >> > understand.
>> > >> > > > > > > >> > >> >> >> > In addition to simplicity of design, the
>> mvn
>> > >> > clean
>> > >> > > > > install
>> > >> > > > > > > of the web
>> > >> > > > > > > >> > >> >> >> > service project is much faster and small
>> scale
>> > >> > > > activity
>> > >> > > > > > > publishing is also
>> > >> > > > > > > >> > >> >> >> > faster (see my email about load testing).
>> These
>> > >> > are
>> > >> > > > > minor
>> > >> > > > > > > points though as
>> > >> > > > > > > >> > >> >> >> > compilation has no effect on deployment.
>> OSGi
>> > >> > does
>> > >> > > > add
>> > >> > > > > the
>> > >> > > > > > > benefit of
>> > >> > > > > > > >> > >> >> >> > modularized programming which is valuable,
>> > >> > though I
>> > >> > > > > think
>> > >> > > > > > > the added
>> > >> > > > > > > >> > >> >> >> > complexity of Camel merits moving the
>> project
>> > >> > away
>> > >> > > > from
>> > >> > > > > > > this paradigm.
>> > >> > > > > > > >> > >> >> >> >
>> > >> > > > > > > >> > >> >> >>
>> > >> > > > > > > >> > >> >> >> I agree that the project is pretty
>> difficult to
>> > >> > > > > understand
>> > >> > > > > > > ATM.  I think
>> > >> > > > > > > >> > >> >> >> what we need to do is think about what the
>> > >> > > > > responsibilities
>> > >> > > > > > > of the code are
>> > >> > > > > > > >> > >> >> >> and allow for different implementations
>> that are
>> > >> > not
>> > >> > > so
>> > >> > > > > > > tightly coupled as
>> > >> > > > > > > >> > >> >> >> they are now.  For instance, having worked
>> with
>> > >> > Storm
>> > >> > > > to
>> > >> > > > > > > ingest millions of
>> > >> > > > > > > >> > >> >> >> activities a day, I personally would like
>> to see
>> > >> > > > streams
>> > >> > > > > be
>> > >> > > > > > > responsible for
>> > >> > > > > > > >> > >> >> >> defining an over-arching orchestration
>> model that
>> > >> > can
>> > >> > > > be
>> > >> > > > > > > implemented within
>> > >> > > > > > > >> > >> >> >> a single war or on top of a distributed
>> system.
>> > >> >  This
>> > >> > > > > would
>> > >> > > > > > > look something
>> > >> > > > > > > >> > >> >> >> like the follows:
>> > >> > > > > > > >> > >> >> >>
>> > >> > > > > > > >> > >> >> >> |__ Streams-Core  (Base classes, interfaces,
>> > >> > > utilities,
>> > >> > > > > > > extensions, etc)
>> > >> > > > > > > >> > >> >> >> |
>> > >> > > > > > > >> > >> >> >> |__ Streams-Storm (Storm implementation of
>> the
>> > >> > core)
>> > >> > > > > > > >> > >> >> >> |
>> > >> > > > > > > >> > >> >> >> |__ Streams-Camel (Camel implementation of
>> the
>> > >> > core)
>> > >> > > > > > > >> > >> >> >> |
>> > >> > > > > > > >> > >> >> >> |__ Streams-WS (Web service implementation)
>> > >> > > > > > > >> > >> >> >>
>> > >> > > > > > > >> > >> >> >>
>> > >> > > > > > > >> > >> >> >> > Danny
>> > >> > > > > > > >> > >> >> >> >
>> > >> > > > > > > >> > >> >> >> >
>> > >> > > > > > > >> > >> >> >> > > From: dsullivan7@hotmail.com
>> > >> > > > > > > >> > >> >> >> > > To: dev@streams.incubator.apache.org
>> > >> > > > > > > >> > >> >> >> > > Subject: [DISCUSS] Switching Streams
>> from
>> > >> Camel
>> > >> > > > > > > deployment to .war
>> > >> > > > > > > >> > >> >> >> > deployment
>> > >> > > > > > > >> > >> >> >> > > Date: Tue, 29 Oct 2013 11:07:39 -0400
>> > >> > > > > > > >> > >> >> >> > >
>> > >> > > > > > > >> > >> >> >> > > The discussion thread for switching
>> Streams
>> > >> > from
>> > >> > > > > > > Camel/osgi/Servicemix
>> > >> > > > > > > >> > >> >> >> > to a single .war deployment
>> > >> > > > > > > >> > >> >> >> >
>> > >> > > > > > > >> > >> >> >> >
>> > >> > > > > > > >> > >> >> >
>> > >> > > > > > > >> > >> >
>> > >> > > > > > > >> > >
>> > >> > > > > > > >>
>> > >> > > > > > > >
>> > >> > > > > > >
>> > >> > > > >
>> > >> > > > >
>> > >> > > >
>> > >> > >
>> > >> >
>> > >>
>>
>>

Re: [DISCUSS] Switching Streams from Camel deployment to .war deployment

[Chris Geer <ch...@cxtsoftware.com>]

On Fri, Nov 1, 2013 at 6:56 AM, Danny Sullivan <ds...@hotmail.com>wrote:

> Perhaps it would be best for me to take a step back and ask what problem
> are we trying to solve by using Camel? My understanding is that it is to
> allow distribution of Streams to multiple servers. It would be helpful to
> me to see another open source application that has this deployment
> structure (it doesn't have to use Camel, just an application that has
> multiple jars communicating with one another). Does anyone have any
> suggestions of software for me to look at?
>

Danny, I wish I could just show you my system we are building. Honestly I
think you are looking at this the wrong way. You are looking at it from a
software development perspective (jars, protocols) and not from a
business/system perspective (messages, components). So let me use an
example.

Let's say I have a business that takes order for widgets. I want the order
taking to be streamlined for the user. To make that happen my order entry
system does some basic checks when the user is placing the order and if
those pass it saves the order to the database. To kick off the other
processing, whenever I save an order to the database I sent a message to a
Topic (pub/sub) saying either an order was created or an order was updated.
Now at this point there is a lot more work I have to do to get the order
shipped to the customer but it's an async action to the order taking. At
this point the order has been placed and the customer can go about their
business.

I also have a component I wrote that listens to the Topic for "Order
Created" messages. It takes each message, retrieves the order does some
inventory processing to get the order ready to ship. At this point the
order is complete and shipped to the customer.

Now, in this simple example you could argue that the order entry system
could just call the order processing component directly but that would
preclude some extensibility in the future.

A few months down the road I get a requirement to be able to notify the
sales team whenever an order is placed by a certain set of customers. In a
traditional model I'd have to modify the order entry component and put in
logic to handle that. In this architecture all I have to do is build a
different component that listens to that same Topic for order by certain
customers. When it gets a notification it can then do the logic to send the
message. The beauty of this is the order entry system doesn't' have to be
modified and there is no risk of inserting a bug into that core process.

So you ask, where does camel come in here. It could be used in lots of
places but the natural fit is for the shipment processing component to be a
camel route that listens to the Topic and calls a java object when messages
come in. Also for the notification components, the whole thing could
probably be a camel route (0 Java code).

This is a pretty simple example and there are lots of other benefits I
didn't talk about like redundancy and deployment flexibility.

Does that help?

>
> > Date: Thu, 31 Oct 2013 20:23:26 -0400
> > Subject: Re: [DISCUSS] Switching Streams from Camel deployment to .war
> deployment
> > From: jletourneau80@gmail.com
> > To: dev@streams.incubator.apache.org
> >
> > So Danny in case it was lost in the ensuing transport discussion,
> > Camel abstracts the transport, so you don't have to care if its http
> > or whatever, only about the message format.  This is helpful if you
> > care about plugging in components that DO care about the transport of
> > course.  To answer your other questions, the downside to having the
> > components all work via http (and not using messaging/Camel) is that
> > everything immediately becomes point to point, which isn't a
> > performance issue in its own right but limits scaling to more than one
> > component being at the receiving end/fulfillment for instance as we've
> > been discussing as being optimal.
> >
> > On Thu, Oct 31, 2013 at 5:38 PM, Matt Franklin <m....@gmail.com>
> wrote:
> > > On Thu, Oct 31, 2013 at 5:34 PM, Chris Geer <ch...@cxtsoftware.com>
> wrote:
> > >
> > >> On Thu, Oct 31, 2013 at 2:32 PM, Matt Franklin <
> m.ben.franklin@gmail.com
> > >> >wrote:
> > >>
> > >> > On Thu, Oct 31, 2013 at 5:28 PM, Chris Geer <ch...@cxtsoftware.com>
> > >> wrote:
> > >> >
> > >> > > On Thu, Oct 31, 2013 at 2:23 PM, Matt Franklin <
> > >> m.ben.franklin@gmail.com
> > >> > > >wrote:
> > >> > >
> > >> > > > On Thu, Oct 31, 2013 at 5:04 PM, Danny Sullivan <
> > >> > dsullivan7@hotmail.com
> > >> > > > >wrote:
> > >> > > >
> > >> > > > > I'm not quite following this, so I apologize. What I'm trying
> to do
> > >> > is
> > >> > > > > programmatically make a request to a jar running on a
> separate jvm
> > >> > and
> > >> > > > get
> > >> > > > > the response from that call all within the same method.
> Similar to
> > >> > this
> > >> > > > > http request:
> > >> > > > >
> > >> > > > > HttpGet httpget = new HttpGet();
> > >> > > > >
> > >> > > > > httpget.setURI(new URI("www.streams-persistence.com"));
> > >> > > > >
> > >> > > > > CloseableHttpResponse response = httpclient.execute(httpget);
> > >> > > > >
> > >> > > > > //do stuff with the response...
> > >> > > > >
> > >> > > > >
> > >> > > > >
> > >> > > > >
> > >> > > > > I imagine that this would translate to something in Camel
> similar
> > >> to
> > >> > > > this:
> > >> > > > >
> > >> > > > >
> > >> > > > > <route>
> > >> > > > >
> > >> > > > > <from uri="bean:subscriberService?method=getAllSubscribers"/>
> > >> > > > >
> > >> > > > > <inOut
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> uri="addressToStreamsPersistenceJar?method=selectAllSubscribersFromDatabase"/>
> > >> > > > >
> > >> > > > > </route>
> > >> > > > >
> > >> > > >
> > >> > > > Camel uses AMQP as a messaging system so that you don't have to
> > >> create
> > >> > > HTTP
> > >> > > > requests between sources, though it is possible to do so.  In
> Storm,
> > >> we
> > >> > > > usually try to use a buffer system like Kafka to do the same.
> > >> > > >
> > >> > >
> > >> > > Camel can use AMQP through ActiveMQ.
> > >> > >
> > >> > >
> > >> > Can being the operative word.  I haven't done much with Camel.  What
> > >> other
> > >> > transports are available.
> > >> >
> > >> > OpenWire (native ActiveMQ)
> > >> AMQP
> > >> STOMP
> > >> HTTP (can be a server or client)
> > >> Mail
> > >> Amazon SQS
> > >> XMPP
> > >> ....hundred or so more
> > >>
> > >> http://camel.apache.org/components.html
> > >
> > >
> > >
> > > Nice.  I was expecting RTFM, so thanks for catering to my laziness.
> > >
> > >
> > >>
> > >>
> > >> >
> > >> > > >
> > >> > > >
> > >> > > > >
> > >> > > > >
> > >> > > > > But it is unclear what the actual implementation would be.
> > >> > > > >
> > >> > > > >
> > >> > > > > This actually brings me to another suggestion. Would there be
> a big
> > >> > > > > performance impact to have communication between the software
> > >> > > components
> > >> > > > > occur between http? Say the 5 software components I outlined
> > >> earlier
> > >> > > were
> > >> > > > > packaged as 5 separate wars. These wars could communicate
> with each
> > >> > > other
> > >> > > > > via get a post requests. This sounds unconventional offhand
> so I'd
> > >> > like
> > >> > > > to
> > >> > > > > hear some thoughts on it.
> > >> > > > >
> > >> > > >
> > >> > > > It is most certainly possible, but IMO probably not the best
> option
> > >> for
> > >> > > > success.  Protocols like PubSubHubbub use HTTP for a message
> > >> transport.
> > >> > > >
> > >> > > >
> > >> > > > > -Danny
> > >> > > > >
> > >> > > > > > Date: Thu, 31 Oct 2013 13:28:41 -0700
> > >> > > > > > Subject: Re: [DISCUSS] Switching Streams from Camel
> deployment to
> > >> > > .war
> > >> > > > > deployment
> > >> > > > > > From: chris@cxtsoftware.com
> > >> > > > > > To: dev@streams.incubator.apache.org
> > >> > > > > >
> > >> > > > > > Or Content Enricher [2]
> > >> > > > > >
> > >> > > > > > [2] http://camel.apache.org/content-enricher.html
> > >> > > > > >
> > >> > > > > >
> > >> > > > > > On Thu, Oct 31, 2013 at 12:37 PM, Jason Letourneau
> > >> > > > > > <jl...@gmail.com>wrote:
> > >> > > > > >
> > >> > > > > > > check out the link here[1]
> > >> > > > > > >
> > >> > > > > > >
> > >> > > > > > > [1]http://camel.apache.org/request-reply.html
> > >> > > > > > >
> > >> > > > > > > On Thu, Oct 31, 2013 at 2:32 PM, Danny Sullivan <
> > >> > > > > dsullivan7@hotmail.com>
> > >> > > > > > > wrote:
> > >> > > > > > > > I have a quick Camel question that I arrived at in the
> > >> > > > > implementation of
> > >> > > > > > > these new components:
> > >> > > > > > > >
> > >> > > > > > > > Lets say I have a method in streams-activity.jar that
> needs
> > >> all
> > >> > > > > > > subscribers in the database. This would require a call to
> the
> > >> > > > > > > streams-persistence.jar. So far, I've seen camel used
> mostly
> > >> for
> > >> > > > > passing
> > >> > > > > > > data through the application, but not for making a single
> > >> > > > > request-reponse
> > >> > > > > > > from within a method. How can I use Camel to get a list
> of all
> > >> > > > > subscribers
> > >> > > > > > > in the streams-persistence.jar from the
> streams-activity.jar?
> > >> > > > > > > >
> > >> > > > > > > >> From: dsullivan7@hotmail.com
> > >> > > > > > > >> To: dev@streams.incubator.apache.org
> > >> > > > > > > >> Subject: RE: [DISCUSS] Switching Streams from Camel
> > >> deployment
> > >> > > to
> > >> > > > > .war
> > >> > > > > > > deployment
> > >> > > > > > > >> Date: Thu, 31 Oct 2013 08:40:44 -0400
> > >> > > > > > > >>
> > >> > > > > > > >> Excellent, I'll write up something as a proof of
> concept and
> > >> > we
> > >> > > > can
> > >> > > > > > > discuss further to make sure everything is vanilla.
> > >> > > > > > > >>
> > >> > > > > > > >> > Date: Wed, 30 Oct 2013 17:01:28 -0400
> > >> > > > > > > >> > Subject: Re: [DISCUSS] Switching Streams from Camel
> > >> > deployment
> > >> > > > to
> > >> > > > > > > .war deployment
> > >> > > > > > > >> > From: jletourneau80@gmail.com
> > >> > > > > > > >> > To: dev@streams.incubator.apache.org
> > >> > > > > > > >> >
> > >> > > > > > > >> > That sounds pretty promising to me.
> > >> > > > > > > >> >
> > >> > > > > > > >> > On Wed, Oct 30, 2013 at 2:38 PM, Danny Sullivan <
> > >> > > > > > > dsullivan7@hotmail.com> wrote:
> > >> > > > > > > >> > > Thanks for the feedback. You have an interesting
> point
> > >> > about
> > >> > > > the
> > >> > > > > > > url linking to a separate processing space. Let me tie my
> > >> answer
> > >> > > into
> > >> > > > > your
> > >> > > > > > > last question about "advocating for the simplicity at
> > >> > registration
> > >> > > to
> > >> > > > > give
> > >> > > > > > > up flexibility at registration, but retaining the inner
> "guts"
> > >> of
> > >> > > > > > > EIP/messaging". Consider a new architecture:
> > >> > > > > > > >> > >
> > >> > > > > > > >> > > streams-web.war: single entry point to
> application, but
> > >> > > > > functions
> > >> > > > > > > ONLY as an entry point. From here Camel routes the
> incoming
> > >> > > requests
> > >> > > > > to 4
> > >> > > > > > > separate jarssubscriber-registration.jar: subscriber
> > >> registration
> > >> > > > > > > publisher-registration.jar: publisher
> registrationactivity.jar:
> > >> > > > returns
> > >> > > > > > > activity (also contains subscriber warehouse and storm
> activity
> > >> > > > > > > aggregator)publish.jar: publishes
> > >> activitystreams-cassandra.jar:
> > >> > > the
> > >> > > > > above
> > >> > > > > > > 4 jars would all have a hook into this jar which would
> function
> > >> > as
> > >> > > a
> > >> > > > > hook
> > >> > > > > > > onto the database. Each jar would have camel route output
> to
> > >> this
> > >> > > > jar.
> > >> > > > > > > >> > >
> > >> > > > > > > >> > > In this implementation, Camel would no longer be
> the
> > >> entry
> > >> > > and
> > >> > > > > exit
> > >> > > > > > > point of a client to the application, but would handle the
> > >> > > > > communication
> > >> > > > > > > between components. The flow of activity through the
> > >> application
> > >> > > > would
> > >> > > > > be
> > >> > > > > > > method based in each jar. This would allow deployment on
> up to
> > >> 6
> > >> > > > > different
> > >> > > > > > > process spaces. However, this does not address that there
> is a
> > >> > > single
> > >> > > > > > > server entry point, but I'm not sure if it was a concern
> in the
> > >> > > first
> > >> > > > > place.
> > >> > > > > > > >> > >
> > >> > > > > > > >> > > My argument, at its basis, is that we should move
> away
> > >> > from
> > >> > > > > using
> > >> > > > > > > Camel as the entry point to the application. I would be
> happy
> > >> to
> > >> > > > > maintain
> > >> > > > > > > messaging between components.
> > >> > > > > > > >> > >
> > >> > > > > > > >> > >> Date: Wed, 30 Oct 2013 12:32:55 -0400
> > >> > > > > > > >> > >> Subject: Re: [DISCUSS] Switching Streams from
> Camel
> > >> > > > deployment
> > >> > > > > to
> > >> > > > > > > .war deployment
> > >> > > > > > > >> > >> From: jletourneau80@gmail.com
> > >> > > > > > > >> > >> To: dev@streams.incubator.apache.org
> > >> > > > > > > >> > >>
> > >> > > > > > > >> > >> An interesting use case that I am holding onto is
> the
> > >> > > ability
> > >> > > > > for
> > >> > > > > > > >> > >> publishers to register via a single URL
> (registration
> > >> > > > > endpoint),
> > >> > > > > > > but
> > >> > > > > > > >> > >> be sent a URL back to post to a different process
> space
> > >> > for
> > >> > > > > actual
> > >> > > > > > > >> > >> publishing.  The same is true on the subscriber
> front.
> > >> > > > >  Currently,
> > >> > > > > > > the
> > >> > > > > > > >> > >> Camel/EIP infrastructure abstracts this because
> > >> different
> > >> > > > > > > components
> > >> > > > > > > >> > >> deployed in different process spaces handling the
> route
> > >> > > > > creation
> > >> > > > > > > can
> > >> > > > > > > >> > >> just be bolted onto a running Streams instance
> without
> > >> > new
> > >> > > > > > > subs/pubs
> > >> > > > > > > >> > >> behaving any differently than existing.  This
> seems to
> > >> > be a
> > >> > > > > > > >> > >> potentially critical scaling point.  Is there a
> way to
> > >> do
> > >> > > > this
> > >> > > > > with
> > >> > > > > > > >> > >> the Spring solution?
> > >> > > > > > > >> > >>
> > >> > > > > > > >> > >> The persistence point is a good one, though I
> would
> > >> > > classify
> > >> > > > > that
> > >> > > > > > > as
> > >> > > > > > > >> > >> "not implemented" vs "not possible" (not that you
> > >> were).
> > >> > > > > > > >> > >>
> > >> > > > > > > >> > >> I'm not married to Camel, I just like the EIP
> approach
> > >> to
> > >> > > > > building
> > >> > > > > > > >> > >> something that is ultimately a messaging system.
>  There
> > >> > are
> > >> > > > > known
> > >> > > > > > > >> > >> patterns that solve at least a subset of the
> problems
> > >> > > Streams
> > >> > > > > is
> > >> > > > > > > >> > >> trying to solve and implementations that can
> handle the
> > >> > > load
> > >> > > > > and
> > >> > > > > > > I'll
> > >> > > > > > > >> > >> reiterate flexibility == complexity almost always.
> > >> > > > > > > >> > >>
> > >> > > > > > > >> > >> It comes right back to the central question: Do
> you
> > >> want
> > >> > > > > > > flexibility
> > >> > > > > > > >> > >> or simplicity?  It doesn't have to be black and
> white
> > >> > > either
> > >> > > > I
> > >> > > > > > > don't
> > >> > > > > > > >> > >> think...
> > >> > > > > > > >> > >>
> > >> > > > > > > >> > >> More pointedly: Where should we give up
> flexibility for
> > >> > > > > > > simplicity?  I
> > >> > > > > > > >> > >> read that Danny is advocating for the simplicity
> at
> > >> > > > > registration to
> > >> > > > > > > >> > >> give up flexibility at registration, but
> retaining the
> > >> > > inner
> > >> > > > > > > "guts" of
> > >> > > > > > > >> > >> EIP/messaging?  Thoughts?
> > >> > > > > > > >> > >>
> > >> > > > > > > >> > >> On Wed, Oct 30, 2013 at 11:37 AM, Danny Sullivan <
> > >> > > > > > > dsullivan7@hotmail.com> wrote:
> > >> > > > > > > >> > >> > My argument is not for the IoC pattern as that
> can be
> > >> > > (and
> > >> > > > > has
> > >> > > > > > > been) implemented alongside Camel. My main argument is
> that the
> > >> > > > syntax
> > >> > > > > at
> > >> > > > > > > the entry point is not only familiar but much simpler.
> This
> > >> > > wouldn't
> > >> > > > > be a
> > >> > > > > > > very strong argument if the Camel implementation wasn't
> much
> > >> more
> > >> > > > > > > complicated but I feel that it is the case. Also, looking
> > >> toward
> > >> > > the
> > >> > > > > > > future, if the server is restarted, in-routes are lost in
> > >> Camel.
> > >> > > The
> > >> > > > > way to
> > >> > > > > > > curb this is to persist the dynamic routes that Camel
> creates,
> > >> > and
> > >> > > > > then on
> > >> > > > > > > start up pull every one of these routes and recreate a
> dynamic
> > >> > > route
> > >> > > > > for
> > >> > > > > > > each one. Not only is this much easier to implement using
> the
> > >> > > Spring
> > >> > > > > web
> > >> > > > > > > implementation, but it already has been implemented and
> you can
> > >> > try
> > >> > > > it
> > >> > > > > by
> > >> > > > > > > checking out the webservice branch, registering a
> subscriber,
> > >> > > > > restarting
> > >> > > > > > > tomcat, and using the same url you had before. This will
> allow
> > >> > > > > subscribers
> > >> > > > > > > to hang on to their urls once they register. (the same is
> true
> > >> > for
> > >> > > > > > > publishers: you can post via the same url after restarting
> > >> > tomcat)
> > >> > > > > > > >> > >> >
> > >> > > > > > > >> > >> >> Date: Wed, 30 Oct 2013 11:00:21 -0400
> > >> > > > > > > >> > >> >> Subject: Re: [DISCUSS] Switching Streams from
> Camel
> > >> > > > > deployment
> > >> > > > > > > to .war deployment
> > >> > > > > > > >> > >> >> From: jletourneau80@gmail.com
> > >> > > > > > > >> > >> >> To: dev@streams.incubator.apache.org
> > >> > > > > > > >> > >> >>
> > >> > > > > > > >> > >> >> To be fair, while the current implementation is
> > >> > heavily
> > >> > > > > > > camel-based,
> > >> > > > > > > >> > >> >> all of the interfaces related to Streams
> > >> functionality
> > >> > > are
> > >> > > > > not.
> > >> > > > > > >  The
> > >> > > > > > > >> > >> >> current model maps to what Matt has outlined
> in my
> > >> > > > opinion,
> > >> > > > > > > though
> > >> > > > > > > >> > >> >> packing names etc. probably don't follow that
> exact
> > >> > > > pattern.
> > >> > > > > > > >> > >> >>
> > >> > > > > > > >> > >> >> With regards to the complexity and different
> > >> > components
> > >> > > in
> > >> > > > > the
> > >> > > > > > > >> > >> >> registration process, this was a cut at the
> > >> > abstraction
> > >> > > > > based
> > >> > > > > > > on the
> > >> > > > > > > >> > >> >> assumption that different implementations may
> be
> > >> > plugged
> > >> > > > in
> > >> > > > > and
> > >> > > > > > > in
> > >> > > > > > > >> > >> >> fact may live on different processor space
> (ie. a
> > >> > > polling
> > >> > > > > > > publisher vs
> > >> > > > > > > >> > >> >> a push publisher may be instantiated on
> different
> > >> > > servers
> > >> > > > > but
> > >> > > > > > > the
> > >> > > > > > > >> > >> >> registration URL is staticly defined).
> > >> > > > > > > >> > >> >>
> > >> > > > > > > >> > >> >> Is the main argument I am seeing for Spring
>  the
> > >> > > > > familiarity of
> > >> > > > > > > its
> > >> > > > > > > >> > >> >> IoC pattern implementation and syntax at the
> entry
> > >> > > point?
> > >> > > > > > > >> > >> >>
> > >> > > > > > > >> > >> >> On Wed, Oct 30, 2013 at 10:53 AM, Danny
> Sullivan <
> > >> > > > > > > dsullivan7@hotmail.com> wrote:
> > >> > > > > > > >> > >> >> > Could you clarify whether the same entry
> points
> > >> > would
> > >> > > > > exist
> > >> > > > > > > for the camel implementation of the core (implementing the
> > >> > > "process"
> > >> > > > > > > method/ using a DynammicRouteBuilder) or would the
> webservice
> > >> be
> > >> > > the
> > >> > > > > sole
> > >> > > > > > > entry point to Streams and after it enters would it hand
> it off
> > >> > to
> > >> > > > > Camel?
> > >> > > > > > > And what would be the entry point for the Storm
> implementation?
> > >> > > > > > > >> > >> >> > -Danny
> > >> > > > > > > >> > >> >> >
> > >> > > > > > > >> > >> >> >> Date: Wed, 30 Oct 2013 10:13:04 -0400
> > >> > > > > > > >> > >> >> >> Subject: Re: [DISCUSS] Switching Streams
> from
> > >> Camel
> > >> > > > > > > deployment to .war deployment
> > >> > > > > > > >> > >> >> >> From: m.ben.franklin@gmail.com
> > >> > > > > > > >> > >> >> >> To: dev@streams.incubator.apache.org
> > >> > > > > > > >> > >> >> >>
> > >> > > > > > > >> > >> >> >> On Tue, Oct 29, 2013 at 2:55 PM, Danny
> Sullivan <
> > >> > > > > > > dsullivan7@hotmail.com>wrote:
> > >> > > > > > > >> > >> >> >>
> > >> > > > > > > >> > >> >> >> > My case for switching from OSGi is for
> > >> simplicity
> > >> > > in
> > >> > > > > > > design. To follow the
> > >> > > > > > > >> > >> >> >> > path of an activity through Streams in the
> > >> > > > webservice,
> > >> > > > > > > there is one main
> > >> > > > > > > >> > >> >> >> > things the developer needs to understand:
> > >> > > > > > > >> > >> >> >> > The @RequestMapping annotation specifies
> the
> > >> HTTP
> > >> > > > entry
> > >> > > > > > > point
> > >> > > > > > > >> > >> >> >> > There are 4 @RequestMapping annotations
> that
> > >> > > > > correspond to
> > >> > > > > > > each of the 4
> > >> > > > > > > >> > >> >> >> > ways a user enters the application:
> > >> registering a
> > >> > > > > > > publisher, registering a
> > >> > > > > > > >> > >> >> >> > subscriber, publishing activity, and
> getting an
> > >> > > > > activity
> > >> > > > > > > stream. Where
> > >> > > > > > > >> > >> >> >> > these are located in the source code can
> be
> > >> found
> > >> > > by
> > >> > > > > > > searching for the
> > >> > > > > > > >> > >> >> >> > paths specified in the documentation
> (search
> > >> for
> > >> > > > > > > "/publisherRegister",
> > >> > > > > > > >> > >> >> >> > "/subscriberRegister", "/publishActivity",
> > >> > > > > "/getActivity"
> > >> > > > > > > which will all
> > >> > > > > > > >> > >> >> >> > lead you to StreamsWebController.java).
> From
> > >> the
> > >> > > > > methods
> > >> > > > > > > that process
> > >> > > > > > > >> > >> >> >> > requests, the flow through the
> application is
> > >> > > through
> > >> > > > > > > methods which can be
> > >> > > > > > > >> > >> >> >> > understood by most Java programmers.
> > >> > > > > > > >> > >> >> >> > The flow of activities through the current
> > >> trunk
> > >> > > > > branch is
> > >> > > > > > > understood as
> > >> > > > > > > >> > >> >> >> > follows:
> > >> > > > > > > >> > >> >> >> > The string "/publisher/register" (the
> entry
> > >> point
> > >> > > to
> > >> > > > > > > register a publisher
> > >> > > > > > > >> > >> >> >> > specified in the documentation) is the
> value of
> > >> > the
> > >> > > > > > > >> > >> >> >> > consumer.registrationEndpoint property
> defined
> > >> in
> > >> > > > > > > streams.propertiesThe
> > >> > > > > > > >> > >> >> >> > camelContext.xml specifies an endpoint
> with the
> > >> > id
> > >> > > > > > > >> > >> >> >> > consumerRegistrationEndpoint to have a uri
> > >> equal
> > >> > to
> > >> > > > the
> > >> > > > > > > propertyThe
> > >> > > > > > > >> > >> >> >> > consumerRegistrationEndpoint routes from
> uri
> > >> > > > > > > direct:publisher register with
> > >> > > > > > > >> > >> >> >> > the bean activityRegistrationProcessor
> nested
> > >> in
> > >> > > > > between
> > >> > > > > > > the routeThe
> > >> > > > > > > >> > >> >> >> > streams-eip-applicationContext contains a
> bean
> > >> > with
> > >> > > > > the id
> > >> > > > > > > >> > >> >> >> > activityRegistrationProcessor created for
> the
> > >> > class
> > >> > > > > > > >> > >> >> >> > ActivityPublisherRegistrationProcessorThe
> > >> > exchange
> > >> > > > will
> > >> > > > > > > enter the "process"
> > >> > > > > > > >> > >> >> >> > method of the class
> > >> > > > > > > ActivityPublisherRegisitrationProcessor and that this
> > >> > > > > > > >> > >> >> >> > is because this class implements the
> > >> "Processor"
> > >> > > > > interface
> > >> > > > > > > provided by
> > >> > > > > > > >> > >> >> >> > CamelThe direct:add-publisher-route takes
> the
> > >> > > > exchange
> > >> > > > > > > output from the
> > >> > > > > > > >> > >> >> >> > "process" method and routes it to the
> > >> > > > > > > activityRegistrationProcessor
> > >> > > > > > > >> > >> >> >> > "register" method. The bean
> > >> > > > > activityRegistrationProcessor
> > >> > > > > > > is defined in the
> > >> > > > > > > >> > >> >> >> > streams-eip-osgi-component-import.xmlThe
> output
> > >> > > from
> > >> > > > > this
> > >> > > > > > > method is then
> > >> > > > > > > >> > >> >> >> > sent to the "createNewRouteForConsumer"
> method
> > >> of
> > >> > > > > > > activityConsumerRouter.
> > >> > > > > > > >> > >> >> >> > This method creates a new route for the
> newly
> > >> > > > > registered
> > >> > > > > > > publisher using
> > >> > > > > > > >> > >> >> >> > the private static final class
> > >> > > > > DynamicConsumerRouteBuilder
> > >> > > > > > > which is
> > >> > > > > > > >> > >> >> >> > required to extend RouteBuilder which is
> > >> provided
> > >> > > by
> > >> > > > > > > Camel. This
> > >> > > > > > > >> > >> >> >> > DynamicConsumerRouteBuilder contains
> several
> > >> > > methods:
> > >> > > > > > > >> > >> >> >> > getConsumerReceiveMethod() (which
> corresponds
> > >> to
> > >> > > > @Value
> > >> > > > > > > >> > >> >> >> > ${consumer.receiveMethod} which
> corresponds to
> > >> > > > > "receive"),
> > >> > > > > > > >> > >> >> >> > getConsumerSplitMethod() (@Value
> > >> > > > > > > ${consumer.splitMethod},"split"), and
> > >> > > > > > > >> > >> >> >> > getConsumerActivityQUri() (@Value
> > >> > > > > ${consumerActivityQUri},
> > >> > > > > > > >> > >> >> >> > "direct:activityQ"). This is different
> than the
> > >> > > > > > > camelContext.xml in that
> > >> > > > > > > >> > >> >> >> > the route is being created
> programmatically.
> > >> What
> > >> > > > this
> > >> > > > > is
> > >> > > > > > > doing is routing
> > >> > > > > > > >> > >> >> >> > input from the inroute url (which Camel
> does
> > >> > > > > automatically
> > >> > > > > > > through the
> > >> > > > > > > >> > >> >> >> > configure method which is required to be
> > >> > > overridden),
> > >> > > > > to
> > >> > > > > > > the "receive"
> > >> > > > > > > >> > >> >> >> > method of ActivityConsumer, then to the
> "split"
> > >> > > > method
> > >> > > > > of
> > >> > > > > > > ActivityConsumer,
> > >> > > > > > > >> > >> >> >> > and then to the "direct:activityQ" which
> if you
> > >> > > look
> > >> > > > > back
> > >> > > > > > > in the
> > >> > > > > > > >> > >> >> >> > camelContext.xml routes to the
> > >> > > > > "activemq:queue:activities"
> > >> > > > > > > which then
> > >> > > > > > > >> > >> >> >> > routes to "receiveExchange"
> > >> > > > > > > >> > >> >> >> > This is the process to register a
> publisher.
> > >> The
> > >> > > > > process
> > >> > > > > > > for registering a
> > >> > > > > > > >> > >> >> >> > subscriber is relatively the same though
> it
> > >> > > involves
> > >> > > > > > > separate classes with
> > >> > > > > > > >> > >> >> >> > their own private static final
> RouteBuilder
> > >> > class.
> > >> > > > > From my
> > >> > > > > > > perspective, the
> > >> > > > > > > >> > >> >> >> > two most difficult things with setting
> this
> > >> > project
> > >> > > > up
> > >> > > > > > > were understanding
> > >> > > > > > > >> > >> >> >> > that the "process" method of the class
> that
> > >> > > > implements
> > >> > > > > > > "Processor" is the
> > >> > > > > > > >> > >> >> >> > entry point and the
> DynamicConsumerRouteBuilder
> > >> > > > creates
> > >> > > > > > > the second entry
> > >> > > > > > > >> > >> >> >> > point (The 5th and last points). This
> made the
> > >> > > > project
> > >> > > > > > > very, VERY hard to
> > >> > > > > > > >> > >> >> >> > understand.
> > >> > > > > > > >> > >> >> >> > In addition to simplicity of design, the
> mvn
> > >> > clean
> > >> > > > > install
> > >> > > > > > > of the web
> > >> > > > > > > >> > >> >> >> > service project is much faster and small
> scale
> > >> > > > activity
> > >> > > > > > > publishing is also
> > >> > > > > > > >> > >> >> >> > faster (see my email about load testing).
> These
> > >> > are
> > >> > > > > minor
> > >> > > > > > > points though as
> > >> > > > > > > >> > >> >> >> > compilation has no effect on deployment.
> OSGi
> > >> > does
> > >> > > > add
> > >> > > > > the
> > >> > > > > > > benefit of
> > >> > > > > > > >> > >> >> >> > modularized programming which is valuable,
> > >> > though I
> > >> > > > > think
> > >> > > > > > > the added
> > >> > > > > > > >> > >> >> >> > complexity of Camel merits moving the
> project
> > >> > away
> > >> > > > from
> > >> > > > > > > this paradigm.
> > >> > > > > > > >> > >> >> >> >
> > >> > > > > > > >> > >> >> >>
> > >> > > > > > > >> > >> >> >> I agree that the project is pretty
> difficult to
> > >> > > > > understand
> > >> > > > > > > ATM.  I think
> > >> > > > > > > >> > >> >> >> what we need to do is think about what the
> > >> > > > > responsibilities
> > >> > > > > > > of the code are
> > >> > > > > > > >> > >> >> >> and allow for different implementations
> that are
> > >> > not
> > >> > > so
> > >> > > > > > > tightly coupled as
> > >> > > > > > > >> > >> >> >> they are now.  For instance, having worked
> with
> > >> > Storm
> > >> > > > to
> > >> > > > > > > ingest millions of
> > >> > > > > > > >> > >> >> >> activities a day, I personally would like
> to see
> > >> > > > streams
> > >> > > > > be
> > >> > > > > > > responsible for
> > >> > > > > > > >> > >> >> >> defining an over-arching orchestration
> model that
> > >> > can
> > >> > > > be
> > >> > > > > > > implemented within
> > >> > > > > > > >> > >> >> >> a single war or on top of a distributed
> system.
> > >> >  This
> > >> > > > > would
> > >> > > > > > > look something
> > >> > > > > > > >> > >> >> >> like the follows:
> > >> > > > > > > >> > >> >> >>
> > >> > > > > > > >> > >> >> >> |__ Streams-Core  (Base classes, interfaces,
> > >> > > utilities,
> > >> > > > > > > extensions, etc)
> > >> > > > > > > >> > >> >> >> |
> > >> > > > > > > >> > >> >> >> |__ Streams-Storm (Storm implementation of
> the
> > >> > core)
> > >> > > > > > > >> > >> >> >> |
> > >> > > > > > > >> > >> >> >> |__ Streams-Camel (Camel implementation of
> the
> > >> > core)
> > >> > > > > > > >> > >> >> >> |
> > >> > > > > > > >> > >> >> >> |__ Streams-WS (Web service implementation)
> > >> > > > > > > >> > >> >> >>
> > >> > > > > > > >> > >> >> >>
> > >> > > > > > > >> > >> >> >> > Danny
> > >> > > > > > > >> > >> >> >> >
> > >> > > > > > > >> > >> >> >> >
> > >> > > > > > > >> > >> >> >> > > From: dsullivan7@hotmail.com
> > >> > > > > > > >> > >> >> >> > > To: dev@streams.incubator.apache.org
> > >> > > > > > > >> > >> >> >> > > Subject: [DISCUSS] Switching Streams
> from
> > >> Camel
> > >> > > > > > > deployment to .war
> > >> > > > > > > >> > >> >> >> > deployment
> > >> > > > > > > >> > >> >> >> > > Date: Tue, 29 Oct 2013 11:07:39 -0400
> > >> > > > > > > >> > >> >> >> > >
> > >> > > > > > > >> > >> >> >> > > The discussion thread for switching
> Streams
> > >> > from
> > >> > > > > > > Camel/osgi/Servicemix
> > >> > > > > > > >> > >> >> >> > to a single .war deployment
> > >> > > > > > > >> > >> >> >> >
> > >> > > > > > > >> > >> >> >> >
> > >> > > > > > > >> > >> >> >
> > >> > > > > > > >> > >> >
> > >> > > > > > > >> > >
> > >> > > > > > > >>
> > >> > > > > > > >
> > >> > > > > > >
> > >> > > > >
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
>
>

RE: [DISCUSS] Switching Streams from Camel deployment to .war deployment

[Danny Sullivan <ds...@hotmail.com>]

Perhaps it would be best for me to take a step back and ask what problem are we trying to solve by using Camel? My understanding is that it is to allow distribution of Streams to multiple servers. It would be helpful to me to see another open source application that has this deployment structure (it doesn't have to use Camel, just an application that has multiple jars communicating with one another). Does anyone have any suggestions of software for me to look at?

> Date: Thu, 31 Oct 2013 20:23:26 -0400
> Subject: Re: [DISCUSS] Switching Streams from Camel deployment to .war deployment
> From: jletourneau80@gmail.com
> To: dev@streams.incubator.apache.org
> 
> So Danny in case it was lost in the ensuing transport discussion,
> Camel abstracts the transport, so you don't have to care if its http
> or whatever, only about the message format.  This is helpful if you
> care about plugging in components that DO care about the transport of
> course.  To answer your other questions, the downside to having the
> components all work via http (and not using messaging/Camel) is that
> everything immediately becomes point to point, which isn't a
> performance issue in its own right but limits scaling to more than one
> component being at the receiving end/fulfillment for instance as we've
> been discussing as being optimal.
> 
> On Thu, Oct 31, 2013 at 5:38 PM, Matt Franklin <m....@gmail.com> wrote:
> > On Thu, Oct 31, 2013 at 5:34 PM, Chris Geer <ch...@cxtsoftware.com> wrote:
> >
> >> On Thu, Oct 31, 2013 at 2:32 PM, Matt Franklin <m.ben.franklin@gmail.com
> >> >wrote:
> >>
> >> > On Thu, Oct 31, 2013 at 5:28 PM, Chris Geer <ch...@cxtsoftware.com>
> >> wrote:
> >> >
> >> > > On Thu, Oct 31, 2013 at 2:23 PM, Matt Franklin <
> >> m.ben.franklin@gmail.com
> >> > > >wrote:
> >> > >
> >> > > > On Thu, Oct 31, 2013 at 5:04 PM, Danny Sullivan <
> >> > dsullivan7@hotmail.com
> >> > > > >wrote:
> >> > > >
> >> > > > > I'm not quite following this, so I apologize. What I'm trying to do
> >> > is
> >> > > > > programmatically make a request to a jar running on a separate jvm
> >> > and
> >> > > > get
> >> > > > > the response from that call all within the same method. Similar to
> >> > this
> >> > > > > http request:
> >> > > > >
> >> > > > > HttpGet httpget = new HttpGet();
> >> > > > >
> >> > > > > httpget.setURI(new URI("www.streams-persistence.com"));
> >> > > > >
> >> > > > > CloseableHttpResponse response = httpclient.execute(httpget);
> >> > > > >
> >> > > > > //do stuff with the response...
> >> > > > >
> >> > > > >
> >> > > > >
> >> > > > >
> >> > > > > I imagine that this would translate to something in Camel similar
> >> to
> >> > > > this:
> >> > > > >
> >> > > > >
> >> > > > > <route>
> >> > > > >
> >> > > > > <from uri="bean:subscriberService?method=getAllSubscribers"/>
> >> > > > >
> >> > > > > <inOut
> >> > > > >
> >> > > >
> >> > >
> >> >
> >> uri="addressToStreamsPersistenceJar?method=selectAllSubscribersFromDatabase"/>
> >> > > > >
> >> > > > > </route>
> >> > > > >
> >> > > >
> >> > > > Camel uses AMQP as a messaging system so that you don't have to
> >> create
> >> > > HTTP
> >> > > > requests between sources, though it is possible to do so.  In Storm,
> >> we
> >> > > > usually try to use a buffer system like Kafka to do the same.
> >> > > >
> >> > >
> >> > > Camel can use AMQP through ActiveMQ.
> >> > >
> >> > >
> >> > Can being the operative word.  I haven't done much with Camel.  What
> >> other
> >> > transports are available.
> >> >
> >> > OpenWire (native ActiveMQ)
> >> AMQP
> >> STOMP
> >> HTTP (can be a server or client)
> >> Mail
> >> Amazon SQS
> >> XMPP
> >> ....hundred or so more
> >>
> >> http://camel.apache.org/components.html
> >
> >
> >
> > Nice.  I was expecting RTFM, so thanks for catering to my laziness.
> >
> >
> >>
> >>
> >> >
> >> > > >
> >> > > >
> >> > > > >
> >> > > > >
> >> > > > > But it is unclear what the actual implementation would be.
> >> > > > >
> >> > > > >
> >> > > > > This actually brings me to another suggestion. Would there be a big
> >> > > > > performance impact to have communication between the software
> >> > > components
> >> > > > > occur between http? Say the 5 software components I outlined
> >> earlier
> >> > > were
> >> > > > > packaged as 5 separate wars. These wars could communicate with each
> >> > > other
> >> > > > > via get a post requests. This sounds unconventional offhand so I'd
> >> > like
> >> > > > to
> >> > > > > hear some thoughts on it.
> >> > > > >
> >> > > >
> >> > > > It is most certainly possible, but IMO probably not the best option
> >> for
> >> > > > success.  Protocols like PubSubHubbub use HTTP for a message
> >> transport.
> >> > > >
> >> > > >
> >> > > > > -Danny
> >> > > > >
> >> > > > > > Date: Thu, 31 Oct 2013 13:28:41 -0700
> >> > > > > > Subject: Re: [DISCUSS] Switching Streams from Camel deployment to
> >> > > .war
> >> > > > > deployment
> >> > > > > > From: chris@cxtsoftware.com
> >> > > > > > To: dev@streams.incubator.apache.org
> >> > > > > >
> >> > > > > > Or Content Enricher [2]
> >> > > > > >
> >> > > > > > [2] http://camel.apache.org/content-enricher.html
> >> > > > > >
> >> > > > > >
> >> > > > > > On Thu, Oct 31, 2013 at 12:37 PM, Jason Letourneau
> >> > > > > > <jl...@gmail.com>wrote:
> >> > > > > >
> >> > > > > > > check out the link here[1]
> >> > > > > > >
> >> > > > > > >
> >> > > > > > > [1]http://camel.apache.org/request-reply.html
> >> > > > > > >
> >> > > > > > > On Thu, Oct 31, 2013 at 2:32 PM, Danny Sullivan <
> >> > > > > dsullivan7@hotmail.com>
> >> > > > > > > wrote:
> >> > > > > > > > I have a quick Camel question that I arrived at in the
> >> > > > > implementation of
> >> > > > > > > these new components:
> >> > > > > > > >
> >> > > > > > > > Lets say I have a method in streams-activity.jar that needs
> >> all
> >> > > > > > > subscribers in the database. This would require a call to the
> >> > > > > > > streams-persistence.jar. So far, I've seen camel used mostly
> >> for
> >> > > > > passing
> >> > > > > > > data through the application, but not for making a single
> >> > > > > request-reponse
> >> > > > > > > from within a method. How can I use Camel to get a list of all
> >> > > > > subscribers
> >> > > > > > > in the streams-persistence.jar from the streams-activity.jar?
> >> > > > > > > >
> >> > > > > > > >> From: dsullivan7@hotmail.com
> >> > > > > > > >> To: dev@streams.incubator.apache.org
> >> > > > > > > >> Subject: RE: [DISCUSS] Switching Streams from Camel
> >> deployment
> >> > > to
> >> > > > > .war
> >> > > > > > > deployment
> >> > > > > > > >> Date: Thu, 31 Oct 2013 08:40:44 -0400
> >> > > > > > > >>
> >> > > > > > > >> Excellent, I'll write up something as a proof of concept and
> >> > we
> >> > > > can
> >> > > > > > > discuss further to make sure everything is vanilla.
> >> > > > > > > >>
> >> > > > > > > >> > Date: Wed, 30 Oct 2013 17:01:28 -0400
> >> > > > > > > >> > Subject: Re: [DISCUSS] Switching Streams from Camel
> >> > deployment
> >> > > > to
> >> > > > > > > .war deployment
> >> > > > > > > >> > From: jletourneau80@gmail.com
> >> > > > > > > >> > To: dev@streams.incubator.apache.org
> >> > > > > > > >> >
> >> > > > > > > >> > That sounds pretty promising to me.
> >> > > > > > > >> >
> >> > > > > > > >> > On Wed, Oct 30, 2013 at 2:38 PM, Danny Sullivan <
> >> > > > > > > dsullivan7@hotmail.com> wrote:
> >> > > > > > > >> > > Thanks for the feedback. You have an interesting point
> >> > about
> >> > > > the
> >> > > > > > > url linking to a separate processing space. Let me tie my
> >> answer
> >> > > into
> >> > > > > your
> >> > > > > > > last question about "advocating for the simplicity at
> >> > registration
> >> > > to
> >> > > > > give
> >> > > > > > > up flexibility at registration, but retaining the inner "guts"
> >> of
> >> > > > > > > EIP/messaging". Consider a new architecture:
> >> > > > > > > >> > >
> >> > > > > > > >> > > streams-web.war: single entry point to application, but
> >> > > > > functions
> >> > > > > > > ONLY as an entry point. From here Camel routes the incoming
> >> > > requests
> >> > > > > to 4
> >> > > > > > > separate jarssubscriber-registration.jar: subscriber
> >> registration
> >> > > > > > > publisher-registration.jar: publisher registrationactivity.jar:
> >> > > > returns
> >> > > > > > > activity (also contains subscriber warehouse and storm activity
> >> > > > > > > aggregator)publish.jar: publishes
> >> activitystreams-cassandra.jar:
> >> > > the
> >> > > > > above
> >> > > > > > > 4 jars would all have a hook into this jar which would function
> >> > as
> >> > > a
> >> > > > > hook
> >> > > > > > > onto the database. Each jar would have camel route output to
> >> this
> >> > > > jar.
> >> > > > > > > >> > >
> >> > > > > > > >> > > In this implementation, Camel would no longer be the
> >> entry
> >> > > and
> >> > > > > exit
> >> > > > > > > point of a client to the application, but would handle the
> >> > > > > communication
> >> > > > > > > between components. The flow of activity through the
> >> application
> >> > > > would
> >> > > > > be
> >> > > > > > > method based in each jar. This would allow deployment on up to
> >> 6
> >> > > > > different
> >> > > > > > > process spaces. However, this does not address that there is a
> >> > > single
> >> > > > > > > server entry point, but I'm not sure if it was a concern in the
> >> > > first
> >> > > > > place.
> >> > > > > > > >> > >
> >> > > > > > > >> > > My argument, at its basis, is that we should move away
> >> > from
> >> > > > > using
> >> > > > > > > Camel as the entry point to the application. I would be happy
> >> to
> >> > > > > maintain
> >> > > > > > > messaging between components.
> >> > > > > > > >> > >
> >> > > > > > > >> > >> Date: Wed, 30 Oct 2013 12:32:55 -0400
> >> > > > > > > >> > >> Subject: Re: [DISCUSS] Switching Streams from Camel
> >> > > > deployment
> >> > > > > to
> >> > > > > > > .war deployment
> >> > > > > > > >> > >> From: jletourneau80@gmail.com
> >> > > > > > > >> > >> To: dev@streams.incubator.apache.org
> >> > > > > > > >> > >>
> >> > > > > > > >> > >> An interesting use case that I am holding onto is the
> >> > > ability
> >> > > > > for
> >> > > > > > > >> > >> publishers to register via a single URL (registration
> >> > > > > endpoint),
> >> > > > > > > but
> >> > > > > > > >> > >> be sent a URL back to post to a different process space
> >> > for
> >> > > > > actual
> >> > > > > > > >> > >> publishing.  The same is true on the subscriber front.
> >> > > > >  Currently,
> >> > > > > > > the
> >> > > > > > > >> > >> Camel/EIP infrastructure abstracts this because
> >> different
> >> > > > > > > components
> >> > > > > > > >> > >> deployed in different process spaces handling the route
> >> > > > > creation
> >> > > > > > > can
> >> > > > > > > >> > >> just be bolted onto a running Streams instance without
> >> > new
> >> > > > > > > subs/pubs
> >> > > > > > > >> > >> behaving any differently than existing.  This seems to
> >> > be a
> >> > > > > > > >> > >> potentially critical scaling point.  Is there a way to
> >> do
> >> > > > this
> >> > > > > with
> >> > > > > > > >> > >> the Spring solution?
> >> > > > > > > >> > >>
> >> > > > > > > >> > >> The persistence point is a good one, though I would
> >> > > classify
> >> > > > > that
> >> > > > > > > as
> >> > > > > > > >> > >> "not implemented" vs "not possible" (not that you
> >> were).
> >> > > > > > > >> > >>
> >> > > > > > > >> > >> I'm not married to Camel, I just like the EIP approach
> >> to
> >> > > > > building
> >> > > > > > > >> > >> something that is ultimately a messaging system.  There
> >> > are
> >> > > > > known
> >> > > > > > > >> > >> patterns that solve at least a subset of the problems
> >> > > Streams
> >> > > > > is
> >> > > > > > > >> > >> trying to solve and implementations that can handle the
> >> > > load
> >> > > > > and
> >> > > > > > > I'll
> >> > > > > > > >> > >> reiterate flexibility == complexity almost always.
> >> > > > > > > >> > >>
> >> > > > > > > >> > >> It comes right back to the central question: Do you
> >> want
> >> > > > > > > flexibility
> >> > > > > > > >> > >> or simplicity?  It doesn't have to be black and white
> >> > > either
> >> > > > I
> >> > > > > > > don't
> >> > > > > > > >> > >> think...
> >> > > > > > > >> > >>
> >> > > > > > > >> > >> More pointedly: Where should we give up flexibility for
> >> > > > > > > simplicity?  I
> >> > > > > > > >> > >> read that Danny is advocating for the simplicity at
> >> > > > > registration to
> >> > > > > > > >> > >> give up flexibility at registration, but retaining the
> >> > > inner
> >> > > > > > > "guts" of
> >> > > > > > > >> > >> EIP/messaging?  Thoughts?
> >> > > > > > > >> > >>
> >> > > > > > > >> > >> On Wed, Oct 30, 2013 at 11:37 AM, Danny Sullivan <
> >> > > > > > > dsullivan7@hotmail.com> wrote:
> >> > > > > > > >> > >> > My argument is not for the IoC pattern as that can be
> >> > > (and
> >> > > > > has
> >> > > > > > > been) implemented alongside Camel. My main argument is that the
> >> > > > syntax
> >> > > > > at
> >> > > > > > > the entry point is not only familiar but much simpler. This
> >> > > wouldn't
> >> > > > > be a
> >> > > > > > > very strong argument if the Camel implementation wasn't much
> >> more
> >> > > > > > > complicated but I feel that it is the case. Also, looking
> >> toward
> >> > > the
> >> > > > > > > future, if the server is restarted, in-routes are lost in
> >> Camel.
> >> > > The
> >> > > > > way to
> >> > > > > > > curb this is to persist the dynamic routes that Camel creates,
> >> > and
> >> > > > > then on
> >> > > > > > > start up pull every one of these routes and recreate a dynamic
> >> > > route
> >> > > > > for
> >> > > > > > > each one. Not only is this much easier to implement using the
> >> > > Spring
> >> > > > > web
> >> > > > > > > implementation, but it already has been implemented and you can
> >> > try
> >> > > > it
> >> > > > > by
> >> > > > > > > checking out the webservice branch, registering a subscriber,
> >> > > > > restarting
> >> > > > > > > tomcat, and using the same url you had before. This will allow
> >> > > > > subscribers
> >> > > > > > > to hang on to their urls once they register. (the same is true
> >> > for
> >> > > > > > > publishers: you can post via the same url after restarting
> >> > tomcat)
> >> > > > > > > >> > >> >
> >> > > > > > > >> > >> >> Date: Wed, 30 Oct 2013 11:00:21 -0400
> >> > > > > > > >> > >> >> Subject: Re: [DISCUSS] Switching Streams from Camel
> >> > > > > deployment
> >> > > > > > > to .war deployment
> >> > > > > > > >> > >> >> From: jletourneau80@gmail.com
> >> > > > > > > >> > >> >> To: dev@streams.incubator.apache.org
> >> > > > > > > >> > >> >>
> >> > > > > > > >> > >> >> To be fair, while the current implementation is
> >> > heavily
> >> > > > > > > camel-based,
> >> > > > > > > >> > >> >> all of the interfaces related to Streams
> >> functionality
> >> > > are
> >> > > > > not.
> >> > > > > > >  The
> >> > > > > > > >> > >> >> current model maps to what Matt has outlined in my
> >> > > > opinion,
> >> > > > > > > though
> >> > > > > > > >> > >> >> packing names etc. probably don't follow that exact
> >> > > > pattern.
> >> > > > > > > >> > >> >>
> >> > > > > > > >> > >> >> With regards to the complexity and different
> >> > components
> >> > > in
> >> > > > > the
> >> > > > > > > >> > >> >> registration process, this was a cut at the
> >> > abstraction
> >> > > > > based
> >> > > > > > > on the
> >> > > > > > > >> > >> >> assumption that different implementations may be
> >> > plugged
> >> > > > in
> >> > > > > and
> >> > > > > > > in
> >> > > > > > > >> > >> >> fact may live on different processor space (ie. a
> >> > > polling
> >> > > > > > > publisher vs
> >> > > > > > > >> > >> >> a push publisher may be instantiated on different
> >> > > servers
> >> > > > > but
> >> > > > > > > the
> >> > > > > > > >> > >> >> registration URL is staticly defined).
> >> > > > > > > >> > >> >>
> >> > > > > > > >> > >> >> Is the main argument I am seeing for Spring  the
> >> > > > > familiarity of
> >> > > > > > > its
> >> > > > > > > >> > >> >> IoC pattern implementation and syntax at the entry
> >> > > point?
> >> > > > > > > >> > >> >>
> >> > > > > > > >> > >> >> On Wed, Oct 30, 2013 at 10:53 AM, Danny Sullivan <
> >> > > > > > > dsullivan7@hotmail.com> wrote:
> >> > > > > > > >> > >> >> > Could you clarify whether the same entry points
> >> > would
> >> > > > > exist
> >> > > > > > > for the camel implementation of the core (implementing the
> >> > > "process"
> >> > > > > > > method/ using a DynammicRouteBuilder) or would the webservice
> >> be
> >> > > the
> >> > > > > sole
> >> > > > > > > entry point to Streams and after it enters would it hand it off
> >> > to
> >> > > > > Camel?
> >> > > > > > > And what would be the entry point for the Storm implementation?
> >> > > > > > > >> > >> >> > -Danny
> >> > > > > > > >> > >> >> >
> >> > > > > > > >> > >> >> >> Date: Wed, 30 Oct 2013 10:13:04 -0400
> >> > > > > > > >> > >> >> >> Subject: Re: [DISCUSS] Switching Streams from
> >> Camel
> >> > > > > > > deployment to .war deployment
> >> > > > > > > >> > >> >> >> From: m.ben.franklin@gmail.com
> >> > > > > > > >> > >> >> >> To: dev@streams.incubator.apache.org
> >> > > > > > > >> > >> >> >>
> >> > > > > > > >> > >> >> >> On Tue, Oct 29, 2013 at 2:55 PM, Danny Sullivan <
> >> > > > > > > dsullivan7@hotmail.com>wrote:
> >> > > > > > > >> > >> >> >>
> >> > > > > > > >> > >> >> >> > My case for switching from OSGi is for
> >> simplicity
> >> > > in
> >> > > > > > > design. To follow the
> >> > > > > > > >> > >> >> >> > path of an activity through Streams in the
> >> > > > webservice,
> >> > > > > > > there is one main
> >> > > > > > > >> > >> >> >> > things the developer needs to understand:
> >> > > > > > > >> > >> >> >> > The @RequestMapping annotation specifies the
> >> HTTP
> >> > > > entry
> >> > > > > > > point
> >> > > > > > > >> > >> >> >> > There are 4 @RequestMapping annotations that
> >> > > > > correspond to
> >> > > > > > > each of the 4
> >> > > > > > > >> > >> >> >> > ways a user enters the application:
> >> registering a
> >> > > > > > > publisher, registering a
> >> > > > > > > >> > >> >> >> > subscriber, publishing activity, and getting an
> >> > > > > activity
> >> > > > > > > stream. Where
> >> > > > > > > >> > >> >> >> > these are located in the source code can be
> >> found
> >> > > by
> >> > > > > > > searching for the
> >> > > > > > > >> > >> >> >> > paths specified in the documentation (search
> >> for
> >> > > > > > > "/publisherRegister",
> >> > > > > > > >> > >> >> >> > "/subscriberRegister", "/publishActivity",
> >> > > > > "/getActivity"
> >> > > > > > > which will all
> >> > > > > > > >> > >> >> >> > lead you to StreamsWebController.java). From
> >> the
> >> > > > > methods
> >> > > > > > > that process
> >> > > > > > > >> > >> >> >> > requests, the flow through the application is
> >> > > through
> >> > > > > > > methods which can be
> >> > > > > > > >> > >> >> >> > understood by most Java programmers.
> >> > > > > > > >> > >> >> >> > The flow of activities through the current
> >> trunk
> >> > > > > branch is
> >> > > > > > > understood as
> >> > > > > > > >> > >> >> >> > follows:
> >> > > > > > > >> > >> >> >> > The string "/publisher/register" (the entry
> >> point
> >> > > to
> >> > > > > > > register a publisher
> >> > > > > > > >> > >> >> >> > specified in the documentation) is the value of
> >> > the
> >> > > > > > > >> > >> >> >> > consumer.registrationEndpoint property defined
> >> in
> >> > > > > > > streams.propertiesThe
> >> > > > > > > >> > >> >> >> > camelContext.xml specifies an endpoint with the
> >> > id
> >> > > > > > > >> > >> >> >> > consumerRegistrationEndpoint to have a uri
> >> equal
> >> > to
> >> > > > the
> >> > > > > > > propertyThe
> >> > > > > > > >> > >> >> >> > consumerRegistrationEndpoint routes from uri
> >> > > > > > > direct:publisher register with
> >> > > > > > > >> > >> >> >> > the bean activityRegistrationProcessor nested
> >> in
> >> > > > > between
> >> > > > > > > the routeThe
> >> > > > > > > >> > >> >> >> > streams-eip-applicationContext contains a bean
> >> > with
> >> > > > > the id
> >> > > > > > > >> > >> >> >> > activityRegistrationProcessor created for the
> >> > class
> >> > > > > > > >> > >> >> >> > ActivityPublisherRegistrationProcessorThe
> >> > exchange
> >> > > > will
> >> > > > > > > enter the "process"
> >> > > > > > > >> > >> >> >> > method of the class
> >> > > > > > > ActivityPublisherRegisitrationProcessor and that this
> >> > > > > > > >> > >> >> >> > is because this class implements the
> >> "Processor"
> >> > > > > interface
> >> > > > > > > provided by
> >> > > > > > > >> > >> >> >> > CamelThe direct:add-publisher-route takes the
> >> > > > exchange
> >> > > > > > > output from the
> >> > > > > > > >> > >> >> >> > "process" method and routes it to the
> >> > > > > > > activityRegistrationProcessor
> >> > > > > > > >> > >> >> >> > "register" method. The bean
> >> > > > > activityRegistrationProcessor
> >> > > > > > > is defined in the
> >> > > > > > > >> > >> >> >> > streams-eip-osgi-component-import.xmlThe output
> >> > > from
> >> > > > > this
> >> > > > > > > method is then
> >> > > > > > > >> > >> >> >> > sent to the "createNewRouteForConsumer" method
> >> of
> >> > > > > > > activityConsumerRouter.
> >> > > > > > > >> > >> >> >> > This method creates a new route for the newly
> >> > > > > registered
> >> > > > > > > publisher using
> >> > > > > > > >> > >> >> >> > the private static final class
> >> > > > > DynamicConsumerRouteBuilder
> >> > > > > > > which is
> >> > > > > > > >> > >> >> >> > required to extend RouteBuilder which is
> >> provided
> >> > > by
> >> > > > > > > Camel. This
> >> > > > > > > >> > >> >> >> > DynamicConsumerRouteBuilder contains several
> >> > > methods:
> >> > > > > > > >> > >> >> >> > getConsumerReceiveMethod() (which corresponds
> >> to
> >> > > > @Value
> >> > > > > > > >> > >> >> >> > ${consumer.receiveMethod} which corresponds to
> >> > > > > "receive"),
> >> > > > > > > >> > >> >> >> > getConsumerSplitMethod() (@Value
> >> > > > > > > ${consumer.splitMethod},"split"), and
> >> > > > > > > >> > >> >> >> > getConsumerActivityQUri() (@Value
> >> > > > > ${consumerActivityQUri},
> >> > > > > > > >> > >> >> >> > "direct:activityQ"). This is different than the
> >> > > > > > > camelContext.xml in that
> >> > > > > > > >> > >> >> >> > the route is being created programmatically.
> >> What
> >> > > > this
> >> > > > > is
> >> > > > > > > doing is routing
> >> > > > > > > >> > >> >> >> > input from the inroute url (which Camel does
> >> > > > > automatically
> >> > > > > > > through the
> >> > > > > > > >> > >> >> >> > configure method which is required to be
> >> > > overridden),
> >> > > > > to
> >> > > > > > > the "receive"
> >> > > > > > > >> > >> >> >> > method of ActivityConsumer, then to the "split"
> >> > > > method
> >> > > > > of
> >> > > > > > > ActivityConsumer,
> >> > > > > > > >> > >> >> >> > and then to the "direct:activityQ" which if you
> >> > > look
> >> > > > > back
> >> > > > > > > in the
> >> > > > > > > >> > >> >> >> > camelContext.xml routes to the
> >> > > > > "activemq:queue:activities"
> >> > > > > > > which then
> >> > > > > > > >> > >> >> >> > routes to "receiveExchange"
> >> > > > > > > >> > >> >> >> > This is the process to register a publisher.
> >> The
> >> > > > > process
> >> > > > > > > for registering a
> >> > > > > > > >> > >> >> >> > subscriber is relatively the same though it
> >> > > involves
> >> > > > > > > separate classes with
> >> > > > > > > >> > >> >> >> > their own private static final RouteBuilder
> >> > class.
> >> > > > > From my
> >> > > > > > > perspective, the
> >> > > > > > > >> > >> >> >> > two most difficult things with setting this
> >> > project
> >> > > > up
> >> > > > > > > were understanding
> >> > > > > > > >> > >> >> >> > that the "process" method of the class that
> >> > > > implements
> >> > > > > > > "Processor" is the
> >> > > > > > > >> > >> >> >> > entry point and the DynamicConsumerRouteBuilder
> >> > > > creates
> >> > > > > > > the second entry
> >> > > > > > > >> > >> >> >> > point (The 5th and last points). This made the
> >> > > > project
> >> > > > > > > very, VERY hard to
> >> > > > > > > >> > >> >> >> > understand.
> >> > > > > > > >> > >> >> >> > In addition to simplicity of design, the mvn
> >> > clean
> >> > > > > install
> >> > > > > > > of the web
> >> > > > > > > >> > >> >> >> > service project is much faster and small scale
> >> > > > activity
> >> > > > > > > publishing is also
> >> > > > > > > >> > >> >> >> > faster (see my email about load testing). These
> >> > are
> >> > > > > minor
> >> > > > > > > points though as
> >> > > > > > > >> > >> >> >> > compilation has no effect on deployment. OSGi
> >> > does
> >> > > > add
> >> > > > > the
> >> > > > > > > benefit of
> >> > > > > > > >> > >> >> >> > modularized programming which is valuable,
> >> > though I
> >> > > > > think
> >> > > > > > > the added
> >> > > > > > > >> > >> >> >> > complexity of Camel merits moving the project
> >> > away
> >> > > > from
> >> > > > > > > this paradigm.
> >> > > > > > > >> > >> >> >> >
> >> > > > > > > >> > >> >> >>
> >> > > > > > > >> > >> >> >> I agree that the project is pretty difficult to
> >> > > > > understand
> >> > > > > > > ATM.  I think
> >> > > > > > > >> > >> >> >> what we need to do is think about what the
> >> > > > > responsibilities
> >> > > > > > > of the code are
> >> > > > > > > >> > >> >> >> and allow for different implementations that are
> >> > not
> >> > > so
> >> > > > > > > tightly coupled as
> >> > > > > > > >> > >> >> >> they are now.  For instance, having worked with
> >> > Storm
> >> > > > to
> >> > > > > > > ingest millions of
> >> > > > > > > >> > >> >> >> activities a day, I personally would like to see
> >> > > > streams
> >> > > > > be
> >> > > > > > > responsible for
> >> > > > > > > >> > >> >> >> defining an over-arching orchestration model that
> >> > can
> >> > > > be
> >> > > > > > > implemented within
> >> > > > > > > >> > >> >> >> a single war or on top of a distributed system.
> >> >  This
> >> > > > > would
> >> > > > > > > look something
> >> > > > > > > >> > >> >> >> like the follows:
> >> > > > > > > >> > >> >> >>
> >> > > > > > > >> > >> >> >> |__ Streams-Core  (Base classes, interfaces,
> >> > > utilities,
> >> > > > > > > extensions, etc)
> >> > > > > > > >> > >> >> >> |
> >> > > > > > > >> > >> >> >> |__ Streams-Storm (Storm implementation of the
> >> > core)
> >> > > > > > > >> > >> >> >> |
> >> > > > > > > >> > >> >> >> |__ Streams-Camel (Camel implementation of the
> >> > core)
> >> > > > > > > >> > >> >> >> |
> >> > > > > > > >> > >> >> >> |__ Streams-WS (Web service implementation)
> >> > > > > > > >> > >> >> >>
> >> > > > > > > >> > >> >> >>
> >> > > > > > > >> > >> >> >> > Danny
> >> > > > > > > >> > >> >> >> >
> >> > > > > > > >> > >> >> >> >
> >> > > > > > > >> > >> >> >> > > From: dsullivan7@hotmail.com
> >> > > > > > > >> > >> >> >> > > To: dev@streams.incubator.apache.org
> >> > > > > > > >> > >> >> >> > > Subject: [DISCUSS] Switching Streams from
> >> Camel
> >> > > > > > > deployment to .war
> >> > > > > > > >> > >> >> >> > deployment
> >> > > > > > > >> > >> >> >> > > Date: Tue, 29 Oct 2013 11:07:39 -0400
> >> > > > > > > >> > >> >> >> > >
> >> > > > > > > >> > >> >> >> > > The discussion thread for switching Streams
> >> > from
> >> > > > > > > Camel/osgi/Servicemix
> >> > > > > > > >> > >> >> >> > to a single .war deployment
> >> > > > > > > >> > >> >> >> >
> >> > > > > > > >> > >> >> >> >
> >> > > > > > > >> > >> >> >
> >> > > > > > > >> > >> >
> >> > > > > > > >> > >
> >> > > > > > > >>
> >> > > > > > > >
> >> > > > > > >
> >> > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
 		 	   		  

Re: [DISCUSS] Switching Streams from Camel deployment to .war deployment

[Jason Letourneau <jl...@gmail.com>]

So Danny in case it was lost in the ensuing transport discussion,
Camel abstracts the transport, so you don't have to care if its http
or whatever, only about the message format.  This is helpful if you
care about plugging in components that DO care about the transport of
course.  To answer your other questions, the downside to having the
components all work via http (and not using messaging/Camel) is that
everything immediately becomes point to point, which isn't a
performance issue in its own right but limits scaling to more than one
component being at the receiving end/fulfillment for instance as we've
been discussing as being optimal.

On Thu, Oct 31, 2013 at 5:38 PM, Matt Franklin <m....@gmail.com> wrote:
> On Thu, Oct 31, 2013 at 5:34 PM, Chris Geer <ch...@cxtsoftware.com> wrote:
>
>> On Thu, Oct 31, 2013 at 2:32 PM, Matt Franklin <m.ben.franklin@gmail.com
>> >wrote:
>>
>> > On Thu, Oct 31, 2013 at 5:28 PM, Chris Geer <ch...@cxtsoftware.com>
>> wrote:
>> >
>> > > On Thu, Oct 31, 2013 at 2:23 PM, Matt Franklin <
>> m.ben.franklin@gmail.com
>> > > >wrote:
>> > >
>> > > > On Thu, Oct 31, 2013 at 5:04 PM, Danny Sullivan <
>> > dsullivan7@hotmail.com
>> > > > >wrote:
>> > > >
>> > > > > I'm not quite following this, so I apologize. What I'm trying to do
>> > is
>> > > > > programmatically make a request to a jar running on a separate jvm
>> > and
>> > > > get
>> > > > > the response from that call all within the same method. Similar to
>> > this
>> > > > > http request:
>> > > > >
>> > > > > HttpGet httpget = new HttpGet();
>> > > > >
>> > > > > httpget.setURI(new URI("www.streams-persistence.com"));
>> > > > >
>> > > > > CloseableHttpResponse response = httpclient.execute(httpget);
>> > > > >
>> > > > > //do stuff with the response...
>> > > > >
>> > > > >
>> > > > >
>> > > > >
>> > > > > I imagine that this would translate to something in Camel similar
>> to
>> > > > this:
>> > > > >
>> > > > >
>> > > > > <route>
>> > > > >
>> > > > > <from uri="bean:subscriberService?method=getAllSubscribers"/>
>> > > > >
>> > > > > <inOut
>> > > > >
>> > > >
>> > >
>> >
>> uri="addressToStreamsPersistenceJar?method=selectAllSubscribersFromDatabase"/>
>> > > > >
>> > > > > </route>
>> > > > >
>> > > >
>> > > > Camel uses AMQP as a messaging system so that you don't have to
>> create
>> > > HTTP
>> > > > requests between sources, though it is possible to do so.  In Storm,
>> we
>> > > > usually try to use a buffer system like Kafka to do the same.
>> > > >
>> > >
>> > > Camel can use AMQP through ActiveMQ.
>> > >
>> > >
>> > Can being the operative word.  I haven't done much with Camel.  What
>> other
>> > transports are available.
>> >
>> > OpenWire (native ActiveMQ)
>> AMQP
>> STOMP
>> HTTP (can be a server or client)
>> Mail
>> Amazon SQS
>> XMPP
>> ....hundred or so more
>>
>> http://camel.apache.org/components.html
>
>
>
> Nice.  I was expecting RTFM, so thanks for catering to my laziness.
>
>
>>
>>
>> >
>> > > >
>> > > >
>> > > > >
>> > > > >
>> > > > > But it is unclear what the actual implementation would be.
>> > > > >
>> > > > >
>> > > > > This actually brings me to another suggestion. Would there be a big
>> > > > > performance impact to have communication between the software
>> > > components
>> > > > > occur between http? Say the 5 software components I outlined
>> earlier
>> > > were
>> > > > > packaged as 5 separate wars. These wars could communicate with each
>> > > other
>> > > > > via get a post requests. This sounds unconventional offhand so I'd
>> > like
>> > > > to
>> > > > > hear some thoughts on it.
>> > > > >
>> > > >
>> > > > It is most certainly possible, but IMO probably not the best option
>> for
>> > > > success.  Protocols like PubSubHubbub use HTTP for a message
>> transport.
>> > > >
>> > > >
>> > > > > -Danny
>> > > > >
>> > > > > > Date: Thu, 31 Oct 2013 13:28:41 -0700
>> > > > > > Subject: Re: [DISCUSS] Switching Streams from Camel deployment to
>> > > .war
>> > > > > deployment
>> > > > > > From: chris@cxtsoftware.com
>> > > > > > To: dev@streams.incubator.apache.org
>> > > > > >
>> > > > > > Or Content Enricher [2]
>> > > > > >
>> > > > > > [2] http://camel.apache.org/content-enricher.html
>> > > > > >
>> > > > > >
>> > > > > > On Thu, Oct 31, 2013 at 12:37 PM, Jason Letourneau
>> > > > > > <jl...@gmail.com>wrote:
>> > > > > >
>> > > > > > > check out the link here[1]
>> > > > > > >
>> > > > > > >
>> > > > > > > [1]http://camel.apache.org/request-reply.html
>> > > > > > >
>> > > > > > > On Thu, Oct 31, 2013 at 2:32 PM, Danny Sullivan <
>> > > > > dsullivan7@hotmail.com>
>> > > > > > > wrote:
>> > > > > > > > I have a quick Camel question that I arrived at in the
>> > > > > implementation of
>> > > > > > > these new components:
>> > > > > > > >
>> > > > > > > > Lets say I have a method in streams-activity.jar that needs
>> all
>> > > > > > > subscribers in the database. This would require a call to the
>> > > > > > > streams-persistence.jar. So far, I've seen camel used mostly
>> for
>> > > > > passing
>> > > > > > > data through the application, but not for making a single
>> > > > > request-reponse
>> > > > > > > from within a method. How can I use Camel to get a list of all
>> > > > > subscribers
>> > > > > > > in the streams-persistence.jar from the streams-activity.jar?
>> > > > > > > >
>> > > > > > > >> From: dsullivan7@hotmail.com
>> > > > > > > >> To: dev@streams.incubator.apache.org
>> > > > > > > >> Subject: RE: [DISCUSS] Switching Streams from Camel
>> deployment
>> > > to
>> > > > > .war
>> > > > > > > deployment
>> > > > > > > >> Date: Thu, 31 Oct 2013 08:40:44 -0400
>> > > > > > > >>
>> > > > > > > >> Excellent, I'll write up something as a proof of concept and
>> > we
>> > > > can
>> > > > > > > discuss further to make sure everything is vanilla.
>> > > > > > > >>
>> > > > > > > >> > Date: Wed, 30 Oct 2013 17:01:28 -0400
>> > > > > > > >> > Subject: Re: [DISCUSS] Switching Streams from Camel
>> > deployment
>> > > > to
>> > > > > > > .war deployment
>> > > > > > > >> > From: jletourneau80@gmail.com
>> > > > > > > >> > To: dev@streams.incubator.apache.org
>> > > > > > > >> >
>> > > > > > > >> > That sounds pretty promising to me.
>> > > > > > > >> >
>> > > > > > > >> > On Wed, Oct 30, 2013 at 2:38 PM, Danny Sullivan <
>> > > > > > > dsullivan7@hotmail.com> wrote:
>> > > > > > > >> > > Thanks for the feedback. You have an interesting point
>> > about
>> > > > the
>> > > > > > > url linking to a separate processing space. Let me tie my
>> answer
>> > > into
>> > > > > your
>> > > > > > > last question about "advocating for the simplicity at
>> > registration
>> > > to
>> > > > > give
>> > > > > > > up flexibility at registration, but retaining the inner "guts"
>> of
>> > > > > > > EIP/messaging". Consider a new architecture:
>> > > > > > > >> > >
>> > > > > > > >> > > streams-web.war: single entry point to application, but
>> > > > > functions
>> > > > > > > ONLY as an entry point. From here Camel routes the incoming
>> > > requests
>> > > > > to 4
>> > > > > > > separate jarssubscriber-registration.jar: subscriber
>> registration
>> > > > > > > publisher-registration.jar: publisher registrationactivity.jar:
>> > > > returns
>> > > > > > > activity (also contains subscriber warehouse and storm activity
>> > > > > > > aggregator)publish.jar: publishes
>> activitystreams-cassandra.jar:
>> > > the
>> > > > > above
>> > > > > > > 4 jars would all have a hook into this jar which would function
>> > as
>> > > a
>> > > > > hook
>> > > > > > > onto the database. Each jar would have camel route output to
>> this
>> > > > jar.
>> > > > > > > >> > >
>> > > > > > > >> > > In this implementation, Camel would no longer be the
>> entry
>> > > and
>> > > > > exit
>> > > > > > > point of a client to the application, but would handle the
>> > > > > communication
>> > > > > > > between components. The flow of activity through the
>> application
>> > > > would
>> > > > > be
>> > > > > > > method based in each jar. This would allow deployment on up to
>> 6
>> > > > > different
>> > > > > > > process spaces. However, this does not address that there is a
>> > > single
>> > > > > > > server entry point, but I'm not sure if it was a concern in the
>> > > first
>> > > > > place.
>> > > > > > > >> > >
>> > > > > > > >> > > My argument, at its basis, is that we should move away
>> > from
>> > > > > using
>> > > > > > > Camel as the entry point to the application. I would be happy
>> to
>> > > > > maintain
>> > > > > > > messaging between components.
>> > > > > > > >> > >
>> > > > > > > >> > >> Date: Wed, 30 Oct 2013 12:32:55 -0400
>> > > > > > > >> > >> Subject: Re: [DISCUSS] Switching Streams from Camel
>> > > > deployment
>> > > > > to
>> > > > > > > .war deployment
>> > > > > > > >> > >> From: jletourneau80@gmail.com
>> > > > > > > >> > >> To: dev@streams.incubator.apache.org
>> > > > > > > >> > >>
>> > > > > > > >> > >> An interesting use case that I am holding onto is the
>> > > ability
>> > > > > for
>> > > > > > > >> > >> publishers to register via a single URL (registration
>> > > > > endpoint),
>> > > > > > > but
>> > > > > > > >> > >> be sent a URL back to post to a different process space
>> > for
>> > > > > actual
>> > > > > > > >> > >> publishing.  The same is true on the subscriber front.
>> > > > >  Currently,
>> > > > > > > the
>> > > > > > > >> > >> Camel/EIP infrastructure abstracts this because
>> different
>> > > > > > > components
>> > > > > > > >> > >> deployed in different process spaces handling the route
>> > > > > creation
>> > > > > > > can
>> > > > > > > >> > >> just be bolted onto a running Streams instance without
>> > new
>> > > > > > > subs/pubs
>> > > > > > > >> > >> behaving any differently than existing.  This seems to
>> > be a
>> > > > > > > >> > >> potentially critical scaling point.  Is there a way to
>> do
>> > > > this
>> > > > > with
>> > > > > > > >> > >> the Spring solution?
>> > > > > > > >> > >>
>> > > > > > > >> > >> The persistence point is a good one, though I would
>> > > classify
>> > > > > that
>> > > > > > > as
>> > > > > > > >> > >> "not implemented" vs "not possible" (not that you
>> were).
>> > > > > > > >> > >>
>> > > > > > > >> > >> I'm not married to Camel, I just like the EIP approach
>> to
>> > > > > building
>> > > > > > > >> > >> something that is ultimately a messaging system.  There
>> > are
>> > > > > known
>> > > > > > > >> > >> patterns that solve at least a subset of the problems
>> > > Streams
>> > > > > is
>> > > > > > > >> > >> trying to solve and implementations that can handle the
>> > > load
>> > > > > and
>> > > > > > > I'll
>> > > > > > > >> > >> reiterate flexibility == complexity almost always.
>> > > > > > > >> > >>
>> > > > > > > >> > >> It comes right back to the central question: Do you
>> want
>> > > > > > > flexibility
>> > > > > > > >> > >> or simplicity?  It doesn't have to be black and white
>> > > either
>> > > > I
>> > > > > > > don't
>> > > > > > > >> > >> think...
>> > > > > > > >> > >>
>> > > > > > > >> > >> More pointedly: Where should we give up flexibility for
>> > > > > > > simplicity?  I
>> > > > > > > >> > >> read that Danny is advocating for the simplicity at
>> > > > > registration to
>> > > > > > > >> > >> give up flexibility at registration, but retaining the
>> > > inner
>> > > > > > > "guts" of
>> > > > > > > >> > >> EIP/messaging?  Thoughts?
>> > > > > > > >> > >>
>> > > > > > > >> > >> On Wed, Oct 30, 2013 at 11:37 AM, Danny Sullivan <
>> > > > > > > dsullivan7@hotmail.com> wrote:
>> > > > > > > >> > >> > My argument is not for the IoC pattern as that can be
>> > > (and
>> > > > > has
>> > > > > > > been) implemented alongside Camel. My main argument is that the
>> > > > syntax
>> > > > > at
>> > > > > > > the entry point is not only familiar but much simpler. This
>> > > wouldn't
>> > > > > be a
>> > > > > > > very strong argument if the Camel implementation wasn't much
>> more
>> > > > > > > complicated but I feel that it is the case. Also, looking
>> toward
>> > > the
>> > > > > > > future, if the server is restarted, in-routes are lost in
>> Camel.
>> > > The
>> > > > > way to
>> > > > > > > curb this is to persist the dynamic routes that Camel creates,
>> > and
>> > > > > then on
>> > > > > > > start up pull every one of these routes and recreate a dynamic
>> > > route
>> > > > > for
>> > > > > > > each one. Not only is this much easier to implement using the
>> > > Spring
>> > > > > web
>> > > > > > > implementation, but it already has been implemented and you can
>> > try
>> > > > it
>> > > > > by
>> > > > > > > checking out the webservice branch, registering a subscriber,
>> > > > > restarting
>> > > > > > > tomcat, and using the same url you had before. This will allow
>> > > > > subscribers
>> > > > > > > to hang on to their urls once they register. (the same is true
>> > for
>> > > > > > > publishers: you can post via the same url after restarting
>> > tomcat)
>> > > > > > > >> > >> >
>> > > > > > > >> > >> >> Date: Wed, 30 Oct 2013 11:00:21 -0400
>> > > > > > > >> > >> >> Subject: Re: [DISCUSS] Switching Streams from Camel
>> > > > > deployment
>> > > > > > > to .war deployment
>> > > > > > > >> > >> >> From: jletourneau80@gmail.com
>> > > > > > > >> > >> >> To: dev@streams.incubator.apache.org
>> > > > > > > >> > >> >>
>> > > > > > > >> > >> >> To be fair, while the current implementation is
>> > heavily
>> > > > > > > camel-based,
>> > > > > > > >> > >> >> all of the interfaces related to Streams
>> functionality
>> > > are
>> > > > > not.
>> > > > > > >  The
>> > > > > > > >> > >> >> current model maps to what Matt has outlined in my
>> > > > opinion,
>> > > > > > > though
>> > > > > > > >> > >> >> packing names etc. probably don't follow that exact
>> > > > pattern.
>> > > > > > > >> > >> >>
>> > > > > > > >> > >> >> With regards to the complexity and different
>> > components
>> > > in
>> > > > > the
>> > > > > > > >> > >> >> registration process, this was a cut at the
>> > abstraction
>> > > > > based
>> > > > > > > on the
>> > > > > > > >> > >> >> assumption that different implementations may be
>> > plugged
>> > > > in
>> > > > > and
>> > > > > > > in
>> > > > > > > >> > >> >> fact may live on different processor space (ie. a
>> > > polling
>> > > > > > > publisher vs
>> > > > > > > >> > >> >> a push publisher may be instantiated on different
>> > > servers
>> > > > > but
>> > > > > > > the
>> > > > > > > >> > >> >> registration URL is staticly defined).
>> > > > > > > >> > >> >>
>> > > > > > > >> > >> >> Is the main argument I am seeing for Spring  the
>> > > > > familiarity of
>> > > > > > > its
>> > > > > > > >> > >> >> IoC pattern implementation and syntax at the entry
>> > > point?
>> > > > > > > >> > >> >>
>> > > > > > > >> > >> >> On Wed, Oct 30, 2013 at 10:53 AM, Danny Sullivan <
>> > > > > > > dsullivan7@hotmail.com> wrote:
>> > > > > > > >> > >> >> > Could you clarify whether the same entry points
>> > would
>> > > > > exist
>> > > > > > > for the camel implementation of the core (implementing the
>> > > "process"
>> > > > > > > method/ using a DynammicRouteBuilder) or would the webservice
>> be
>> > > the
>> > > > > sole
>> > > > > > > entry point to Streams and after it enters would it hand it off
>> > to
>> > > > > Camel?
>> > > > > > > And what would be the entry point for the Storm implementation?
>> > > > > > > >> > >> >> > -Danny
>> > > > > > > >> > >> >> >
>> > > > > > > >> > >> >> >> Date: Wed, 30 Oct 2013 10:13:04 -0400
>> > > > > > > >> > >> >> >> Subject: Re: [DISCUSS] Switching Streams from
>> Camel
>> > > > > > > deployment to .war deployment
>> > > > > > > >> > >> >> >> From: m.ben.franklin@gmail.com
>> > > > > > > >> > >> >> >> To: dev@streams.incubator.apache.org
>> > > > > > > >> > >> >> >>
>> > > > > > > >> > >> >> >> On Tue, Oct 29, 2013 at 2:55 PM, Danny Sullivan <
>> > > > > > > dsullivan7@hotmail.com>wrote:
>> > > > > > > >> > >> >> >>
>> > > > > > > >> > >> >> >> > My case for switching from OSGi is for
>> simplicity
>> > > in
>> > > > > > > design. To follow the
>> > > > > > > >> > >> >> >> > path of an activity through Streams in the
>> > > > webservice,
>> > > > > > > there is one main
>> > > > > > > >> > >> >> >> > things the developer needs to understand:
>> > > > > > > >> > >> >> >> > The @RequestMapping annotation specifies the
>> HTTP
>> > > > entry
>> > > > > > > point
>> > > > > > > >> > >> >> >> > There are 4 @RequestMapping annotations that
>> > > > > correspond to
>> > > > > > > each of the 4
>> > > > > > > >> > >> >> >> > ways a user enters the application:
>> registering a
>> > > > > > > publisher, registering a
>> > > > > > > >> > >> >> >> > subscriber, publishing activity, and getting an
>> > > > > activity
>> > > > > > > stream. Where
>> > > > > > > >> > >> >> >> > these are located in the source code can be
>> found
>> > > by
>> > > > > > > searching for the
>> > > > > > > >> > >> >> >> > paths specified in the documentation (search
>> for
>> > > > > > > "/publisherRegister",
>> > > > > > > >> > >> >> >> > "/subscriberRegister", "/publishActivity",
>> > > > > "/getActivity"
>> > > > > > > which will all
>> > > > > > > >> > >> >> >> > lead you to StreamsWebController.java). From
>> the
>> > > > > methods
>> > > > > > > that process
>> > > > > > > >> > >> >> >> > requests, the flow through the application is
>> > > through
>> > > > > > > methods which can be
>> > > > > > > >> > >> >> >> > understood by most Java programmers.
>> > > > > > > >> > >> >> >> > The flow of activities through the current
>> trunk
>> > > > > branch is
>> > > > > > > understood as
>> > > > > > > >> > >> >> >> > follows:
>> > > > > > > >> > >> >> >> > The string "/publisher/register" (the entry
>> point
>> > > to
>> > > > > > > register a publisher
>> > > > > > > >> > >> >> >> > specified in the documentation) is the value of
>> > the
>> > > > > > > >> > >> >> >> > consumer.registrationEndpoint property defined
>> in
>> > > > > > > streams.propertiesThe
>> > > > > > > >> > >> >> >> > camelContext.xml specifies an endpoint with the
>> > id
>> > > > > > > >> > >> >> >> > consumerRegistrationEndpoint to have a uri
>> equal
>> > to
>> > > > the
>> > > > > > > propertyThe
>> > > > > > > >> > >> >> >> > consumerRegistrationEndpoint routes from uri
>> > > > > > > direct:publisher register with
>> > > > > > > >> > >> >> >> > the bean activityRegistrationProcessor nested
>> in
>> > > > > between
>> > > > > > > the routeThe
>> > > > > > > >> > >> >> >> > streams-eip-applicationContext contains a bean
>> > with
>> > > > > the id
>> > > > > > > >> > >> >> >> > activityRegistrationProcessor created for the
>> > class
>> > > > > > > >> > >> >> >> > ActivityPublisherRegistrationProcessorThe
>> > exchange
>> > > > will
>> > > > > > > enter the "process"
>> > > > > > > >> > >> >> >> > method of the class
>> > > > > > > ActivityPublisherRegisitrationProcessor and that this
>> > > > > > > >> > >> >> >> > is because this class implements the
>> "Processor"
>> > > > > interface
>> > > > > > > provided by
>> > > > > > > >> > >> >> >> > CamelThe direct:add-publisher-route takes the
>> > > > exchange
>> > > > > > > output from the
>> > > > > > > >> > >> >> >> > "process" method and routes it to the
>> > > > > > > activityRegistrationProcessor
>> > > > > > > >> > >> >> >> > "register" method. The bean
>> > > > > activityRegistrationProcessor
>> > > > > > > is defined in the
>> > > > > > > >> > >> >> >> > streams-eip-osgi-component-import.xmlThe output
>> > > from
>> > > > > this
>> > > > > > > method is then
>> > > > > > > >> > >> >> >> > sent to the "createNewRouteForConsumer" method
>> of
>> > > > > > > activityConsumerRouter.
>> > > > > > > >> > >> >> >> > This method creates a new route for the newly
>> > > > > registered
>> > > > > > > publisher using
>> > > > > > > >> > >> >> >> > the private static final class
>> > > > > DynamicConsumerRouteBuilder
>> > > > > > > which is
>> > > > > > > >> > >> >> >> > required to extend RouteBuilder which is
>> provided
>> > > by
>> > > > > > > Camel. This
>> > > > > > > >> > >> >> >> > DynamicConsumerRouteBuilder contains several
>> > > methods:
>> > > > > > > >> > >> >> >> > getConsumerReceiveMethod() (which corresponds
>> to
>> > > > @Value
>> > > > > > > >> > >> >> >> > ${consumer.receiveMethod} which corresponds to
>> > > > > "receive"),
>> > > > > > > >> > >> >> >> > getConsumerSplitMethod() (@Value
>> > > > > > > ${consumer.splitMethod},"split"), and
>> > > > > > > >> > >> >> >> > getConsumerActivityQUri() (@Value
>> > > > > ${consumerActivityQUri},
>> > > > > > > >> > >> >> >> > "direct:activityQ"). This is different than the
>> > > > > > > camelContext.xml in that
>> > > > > > > >> > >> >> >> > the route is being created programmatically.
>> What
>> > > > this
>> > > > > is
>> > > > > > > doing is routing
>> > > > > > > >> > >> >> >> > input from the inroute url (which Camel does
>> > > > > automatically
>> > > > > > > through the
>> > > > > > > >> > >> >> >> > configure method which is required to be
>> > > overridden),
>> > > > > to
>> > > > > > > the "receive"
>> > > > > > > >> > >> >> >> > method of ActivityConsumer, then to the "split"
>> > > > method
>> > > > > of
>> > > > > > > ActivityConsumer,
>> > > > > > > >> > >> >> >> > and then to the "direct:activityQ" which if you
>> > > look
>> > > > > back
>> > > > > > > in the
>> > > > > > > >> > >> >> >> > camelContext.xml routes to the
>> > > > > "activemq:queue:activities"
>> > > > > > > which then
>> > > > > > > >> > >> >> >> > routes to "receiveExchange"
>> > > > > > > >> > >> >> >> > This is the process to register a publisher.
>> The
>> > > > > process
>> > > > > > > for registering a
>> > > > > > > >> > >> >> >> > subscriber is relatively the same though it
>> > > involves
>> > > > > > > separate classes with
>> > > > > > > >> > >> >> >> > their own private static final RouteBuilder
>> > class.
>> > > > > From my
>> > > > > > > perspective, the
>> > > > > > > >> > >> >> >> > two most difficult things with setting this
>> > project
>> > > > up
>> > > > > > > were understanding
>> > > > > > > >> > >> >> >> > that the "process" method of the class that
>> > > > implements
>> > > > > > > "Processor" is the
>> > > > > > > >> > >> >> >> > entry point and the DynamicConsumerRouteBuilder
>> > > > creates
>> > > > > > > the second entry
>> > > > > > > >> > >> >> >> > point (The 5th and last points). This made the
>> > > > project
>> > > > > > > very, VERY hard to
>> > > > > > > >> > >> >> >> > understand.
>> > > > > > > >> > >> >> >> > In addition to simplicity of design, the mvn
>> > clean
>> > > > > install
>> > > > > > > of the web
>> > > > > > > >> > >> >> >> > service project is much faster and small scale
>> > > > activity
>> > > > > > > publishing is also
>> > > > > > > >> > >> >> >> > faster (see my email about load testing). These
>> > are
>> > > > > minor
>> > > > > > > points though as
>> > > > > > > >> > >> >> >> > compilation has no effect on deployment. OSGi
>> > does
>> > > > add
>> > > > > the
>> > > > > > > benefit of
>> > > > > > > >> > >> >> >> > modularized programming which is valuable,
>> > though I
>> > > > > think
>> > > > > > > the added
>> > > > > > > >> > >> >> >> > complexity of Camel merits moving the project
>> > away
>> > > > from
>> > > > > > > this paradigm.
>> > > > > > > >> > >> >> >> >
>> > > > > > > >> > >> >> >>
>> > > > > > > >> > >> >> >> I agree that the project is pretty difficult to
>> > > > > understand
>> > > > > > > ATM.  I think
>> > > > > > > >> > >> >> >> what we need to do is think about what the
>> > > > > responsibilities
>> > > > > > > of the code are
>> > > > > > > >> > >> >> >> and allow for different implementations that are
>> > not
>> > > so
>> > > > > > > tightly coupled as
>> > > > > > > >> > >> >> >> they are now.  For instance, having worked with
>> > Storm
>> > > > to
>> > > > > > > ingest millions of
>> > > > > > > >> > >> >> >> activities a day, I personally would like to see
>> > > > streams
>> > > > > be
>> > > > > > > responsible for
>> > > > > > > >> > >> >> >> defining an over-arching orchestration model that
>> > can
>> > > > be
>> > > > > > > implemented within
>> > > > > > > >> > >> >> >> a single war or on top of a distributed system.
>> >  This
>> > > > > would
>> > > > > > > look something
>> > > > > > > >> > >> >> >> like the follows:
>> > > > > > > >> > >> >> >>
>> > > > > > > >> > >> >> >> |__ Streams-Core  (Base classes, interfaces,
>> > > utilities,
>> > > > > > > extensions, etc)
>> > > > > > > >> > >> >> >> |
>> > > > > > > >> > >> >> >> |__ Streams-Storm (Storm implementation of the
>> > core)
>> > > > > > > >> > >> >> >> |
>> > > > > > > >> > >> >> >> |__ Streams-Camel (Camel implementation of the
>> > core)
>> > > > > > > >> > >> >> >> |
>> > > > > > > >> > >> >> >> |__ Streams-WS (Web service implementation)
>> > > > > > > >> > >> >> >>
>> > > > > > > >> > >> >> >>
>> > > > > > > >> > >> >> >> > Danny
>> > > > > > > >> > >> >> >> >
>> > > > > > > >> > >> >> >> >
>> > > > > > > >> > >> >> >> > > From: dsullivan7@hotmail.com
>> > > > > > > >> > >> >> >> > > To: dev@streams.incubator.apache.org
>> > > > > > > >> > >> >> >> > > Subject: [DISCUSS] Switching Streams from
>> Camel
>> > > > > > > deployment to .war
>> > > > > > > >> > >> >> >> > deployment
>> > > > > > > >> > >> >> >> > > Date: Tue, 29 Oct 2013 11:07:39 -0400
>> > > > > > > >> > >> >> >> > >
>> > > > > > > >> > >> >> >> > > The discussion thread for switching Streams
>> > from
>> > > > > > > Camel/osgi/Servicemix
>> > > > > > > >> > >> >> >> > to a single .war deployment
>> > > > > > > >> > >> >> >> >
>> > > > > > > >> > >> >> >> >
>> > > > > > > >> > >> >> >
>> > > > > > > >> > >> >
>> > > > > > > >> > >
>> > > > > > > >>
>> > > > > > > >
>> > > > > > >
>> > > > >
>> > > > >
>> > > >
>> > >
>> >
>>

Re: [DISCUSS] Switching Streams from Camel deployment to .war deployment

[Matt Franklin <m....@gmail.com>]

On Thu, Oct 31, 2013 at 5:34 PM, Chris Geer <ch...@cxtsoftware.com> wrote:

> On Thu, Oct 31, 2013 at 2:32 PM, Matt Franklin <m.ben.franklin@gmail.com
> >wrote:
>
> > On Thu, Oct 31, 2013 at 5:28 PM, Chris Geer <ch...@cxtsoftware.com>
> wrote:
> >
> > > On Thu, Oct 31, 2013 at 2:23 PM, Matt Franklin <
> m.ben.franklin@gmail.com
> > > >wrote:
> > >
> > > > On Thu, Oct 31, 2013 at 5:04 PM, Danny Sullivan <
> > dsullivan7@hotmail.com
> > > > >wrote:
> > > >
> > > > > I'm not quite following this, so I apologize. What I'm trying to do
> > is
> > > > > programmatically make a request to a jar running on a separate jvm
> > and
> > > > get
> > > > > the response from that call all within the same method. Similar to
> > this
> > > > > http request:
> > > > >
> > > > > HttpGet httpget = new HttpGet();
> > > > >
> > > > > httpget.setURI(new URI("www.streams-persistence.com"));
> > > > >
> > > > > CloseableHttpResponse response = httpclient.execute(httpget);
> > > > >
> > > > > //do stuff with the response...
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > I imagine that this would translate to something in Camel similar
> to
> > > > this:
> > > > >
> > > > >
> > > > > <route>
> > > > >
> > > > > <from uri="bean:subscriberService?method=getAllSubscribers"/>
> > > > >
> > > > > <inOut
> > > > >
> > > >
> > >
> >
> uri="addressToStreamsPersistenceJar?method=selectAllSubscribersFromDatabase"/>
> > > > >
> > > > > </route>
> > > > >
> > > >
> > > > Camel uses AMQP as a messaging system so that you don't have to
> create
> > > HTTP
> > > > requests between sources, though it is possible to do so.  In Storm,
> we
> > > > usually try to use a buffer system like Kafka to do the same.
> > > >
> > >
> > > Camel can use AMQP through ActiveMQ.
> > >
> > >
> > Can being the operative word.  I haven't done much with Camel.  What
> other
> > transports are available.
> >
> > OpenWire (native ActiveMQ)
> AMQP
> STOMP
> HTTP (can be a server or client)
> Mail
> Amazon SQS
> XMPP
> ....hundred or so more
>
> http://camel.apache.org/components.html



Nice.  I was expecting RTFM, so thanks for catering to my laziness.


>
>
> >
> > > >
> > > >
> > > > >
> > > > >
> > > > > But it is unclear what the actual implementation would be.
> > > > >
> > > > >
> > > > > This actually brings me to another suggestion. Would there be a big
> > > > > performance impact to have communication between the software
> > > components
> > > > > occur between http? Say the 5 software components I outlined
> earlier
> > > were
> > > > > packaged as 5 separate wars. These wars could communicate with each
> > > other
> > > > > via get a post requests. This sounds unconventional offhand so I'd
> > like
> > > > to
> > > > > hear some thoughts on it.
> > > > >
> > > >
> > > > It is most certainly possible, but IMO probably not the best option
> for
> > > > success.  Protocols like PubSubHubbub use HTTP for a message
> transport.
> > > >
> > > >
> > > > > -Danny
> > > > >
> > > > > > Date: Thu, 31 Oct 2013 13:28:41 -0700
> > > > > > Subject: Re: [DISCUSS] Switching Streams from Camel deployment to
> > > .war
> > > > > deployment
> > > > > > From: chris@cxtsoftware.com
> > > > > > To: dev@streams.incubator.apache.org
> > > > > >
> > > > > > Or Content Enricher [2]
> > > > > >
> > > > > > [2] http://camel.apache.org/content-enricher.html
> > > > > >
> > > > > >
> > > > > > On Thu, Oct 31, 2013 at 12:37 PM, Jason Letourneau
> > > > > > <jl...@gmail.com>wrote:
> > > > > >
> > > > > > > check out the link here[1]
> > > > > > >
> > > > > > >
> > > > > > > [1]http://camel.apache.org/request-reply.html
> > > > > > >
> > > > > > > On Thu, Oct 31, 2013 at 2:32 PM, Danny Sullivan <
> > > > > dsullivan7@hotmail.com>
> > > > > > > wrote:
> > > > > > > > I have a quick Camel question that I arrived at in the
> > > > > implementation of
> > > > > > > these new components:
> > > > > > > >
> > > > > > > > Lets say I have a method in streams-activity.jar that needs
> all
> > > > > > > subscribers in the database. This would require a call to the
> > > > > > > streams-persistence.jar. So far, I've seen camel used mostly
> for
> > > > > passing
> > > > > > > data through the application, but not for making a single
> > > > > request-reponse
> > > > > > > from within a method. How can I use Camel to get a list of all
> > > > > subscribers
> > > > > > > in the streams-persistence.jar from the streams-activity.jar?
> > > > > > > >
> > > > > > > >> From: dsullivan7@hotmail.com
> > > > > > > >> To: dev@streams.incubator.apache.org
> > > > > > > >> Subject: RE: [DISCUSS] Switching Streams from Camel
> deployment
> > > to
> > > > > .war
> > > > > > > deployment
> > > > > > > >> Date: Thu, 31 Oct 2013 08:40:44 -0400
> > > > > > > >>
> > > > > > > >> Excellent, I'll write up something as a proof of concept and
> > we
> > > > can
> > > > > > > discuss further to make sure everything is vanilla.
> > > > > > > >>
> > > > > > > >> > Date: Wed, 30 Oct 2013 17:01:28 -0400
> > > > > > > >> > Subject: Re: [DISCUSS] Switching Streams from Camel
> > deployment
> > > > to
> > > > > > > .war deployment
> > > > > > > >> > From: jletourneau80@gmail.com
> > > > > > > >> > To: dev@streams.incubator.apache.org
> > > > > > > >> >
> > > > > > > >> > That sounds pretty promising to me.
> > > > > > > >> >
> > > > > > > >> > On Wed, Oct 30, 2013 at 2:38 PM, Danny Sullivan <
> > > > > > > dsullivan7@hotmail.com> wrote:
> > > > > > > >> > > Thanks for the feedback. You have an interesting point
> > about
> > > > the
> > > > > > > url linking to a separate processing space. Let me tie my
> answer
> > > into
> > > > > your
> > > > > > > last question about "advocating for the simplicity at
> > registration
> > > to
> > > > > give
> > > > > > > up flexibility at registration, but retaining the inner "guts"
> of
> > > > > > > EIP/messaging". Consider a new architecture:
> > > > > > > >> > >
> > > > > > > >> > > streams-web.war: single entry point to application, but
> > > > > functions
> > > > > > > ONLY as an entry point. From here Camel routes the incoming
> > > requests
> > > > > to 4
> > > > > > > separate jarssubscriber-registration.jar: subscriber
> registration
> > > > > > > publisher-registration.jar: publisher registrationactivity.jar:
> > > > returns
> > > > > > > activity (also contains subscriber warehouse and storm activity
> > > > > > > aggregator)publish.jar: publishes
> activitystreams-cassandra.jar:
> > > the
> > > > > above
> > > > > > > 4 jars would all have a hook into this jar which would function
> > as
> > > a
> > > > > hook
> > > > > > > onto the database. Each jar would have camel route output to
> this
> > > > jar.
> > > > > > > >> > >
> > > > > > > >> > > In this implementation, Camel would no longer be the
> entry
> > > and
> > > > > exit
> > > > > > > point of a client to the application, but would handle the
> > > > > communication
> > > > > > > between components. The flow of activity through the
> application
> > > > would
> > > > > be
> > > > > > > method based in each jar. This would allow deployment on up to
> 6
> > > > > different
> > > > > > > process spaces. However, this does not address that there is a
> > > single
> > > > > > > server entry point, but I'm not sure if it was a concern in the
> > > first
> > > > > place.
> > > > > > > >> > >
> > > > > > > >> > > My argument, at its basis, is that we should move away
> > from
> > > > > using
> > > > > > > Camel as the entry point to the application. I would be happy
> to
> > > > > maintain
> > > > > > > messaging between components.
> > > > > > > >> > >
> > > > > > > >> > >> Date: Wed, 30 Oct 2013 12:32:55 -0400
> > > > > > > >> > >> Subject: Re: [DISCUSS] Switching Streams from Camel
> > > > deployment
> > > > > to
> > > > > > > .war deployment
> > > > > > > >> > >> From: jletourneau80@gmail.com
> > > > > > > >> > >> To: dev@streams.incubator.apache.org
> > > > > > > >> > >>
> > > > > > > >> > >> An interesting use case that I am holding onto is the
> > > ability
> > > > > for
> > > > > > > >> > >> publishers to register via a single URL (registration
> > > > > endpoint),
> > > > > > > but
> > > > > > > >> > >> be sent a URL back to post to a different process space
> > for
> > > > > actual
> > > > > > > >> > >> publishing.  The same is true on the subscriber front.
> > > > >  Currently,
> > > > > > > the
> > > > > > > >> > >> Camel/EIP infrastructure abstracts this because
> different
> > > > > > > components
> > > > > > > >> > >> deployed in different process spaces handling the route
> > > > > creation
> > > > > > > can
> > > > > > > >> > >> just be bolted onto a running Streams instance without
> > new
> > > > > > > subs/pubs
> > > > > > > >> > >> behaving any differently than existing.  This seems to
> > be a
> > > > > > > >> > >> potentially critical scaling point.  Is there a way to
> do
> > > > this
> > > > > with
> > > > > > > >> > >> the Spring solution?
> > > > > > > >> > >>
> > > > > > > >> > >> The persistence point is a good one, though I would
> > > classify
> > > > > that
> > > > > > > as
> > > > > > > >> > >> "not implemented" vs "not possible" (not that you
> were).
> > > > > > > >> > >>
> > > > > > > >> > >> I'm not married to Camel, I just like the EIP approach
> to
> > > > > building
> > > > > > > >> > >> something that is ultimately a messaging system.  There
> > are
> > > > > known
> > > > > > > >> > >> patterns that solve at least a subset of the problems
> > > Streams
> > > > > is
> > > > > > > >> > >> trying to solve and implementations that can handle the
> > > load
> > > > > and
> > > > > > > I'll
> > > > > > > >> > >> reiterate flexibility == complexity almost always.
> > > > > > > >> > >>
> > > > > > > >> > >> It comes right back to the central question: Do you
> want
> > > > > > > flexibility
> > > > > > > >> > >> or simplicity?  It doesn't have to be black and white
> > > either
> > > > I
> > > > > > > don't
> > > > > > > >> > >> think...
> > > > > > > >> > >>
> > > > > > > >> > >> More pointedly: Where should we give up flexibility for
> > > > > > > simplicity?  I
> > > > > > > >> > >> read that Danny is advocating for the simplicity at
> > > > > registration to
> > > > > > > >> > >> give up flexibility at registration, but retaining the
> > > inner
> > > > > > > "guts" of
> > > > > > > >> > >> EIP/messaging?  Thoughts?
> > > > > > > >> > >>
> > > > > > > >> > >> On Wed, Oct 30, 2013 at 11:37 AM, Danny Sullivan <
> > > > > > > dsullivan7@hotmail.com> wrote:
> > > > > > > >> > >> > My argument is not for the IoC pattern as that can be
> > > (and
> > > > > has
> > > > > > > been) implemented alongside Camel. My main argument is that the
> > > > syntax
> > > > > at
> > > > > > > the entry point is not only familiar but much simpler. This
> > > wouldn't
> > > > > be a
> > > > > > > very strong argument if the Camel implementation wasn't much
> more
> > > > > > > complicated but I feel that it is the case. Also, looking
> toward
> > > the
> > > > > > > future, if the server is restarted, in-routes are lost in
> Camel.
> > > The
> > > > > way to
> > > > > > > curb this is to persist the dynamic routes that Camel creates,
> > and
> > > > > then on
> > > > > > > start up pull every one of these routes and recreate a dynamic
> > > route
> > > > > for
> > > > > > > each one. Not only is this much easier to implement using the
> > > Spring
> > > > > web
> > > > > > > implementation, but it already has been implemented and you can
> > try
> > > > it
> > > > > by
> > > > > > > checking out the webservice branch, registering a subscriber,
> > > > > restarting
> > > > > > > tomcat, and using the same url you had before. This will allow
> > > > > subscribers
> > > > > > > to hang on to their urls once they register. (the same is true
> > for
> > > > > > > publishers: you can post via the same url after restarting
> > tomcat)
> > > > > > > >> > >> >
> > > > > > > >> > >> >> Date: Wed, 30 Oct 2013 11:00:21 -0400
> > > > > > > >> > >> >> Subject: Re: [DISCUSS] Switching Streams from Camel
> > > > > deployment
> > > > > > > to .war deployment
> > > > > > > >> > >> >> From: jletourneau80@gmail.com
> > > > > > > >> > >> >> To: dev@streams.incubator.apache.org
> > > > > > > >> > >> >>
> > > > > > > >> > >> >> To be fair, while the current implementation is
> > heavily
> > > > > > > camel-based,
> > > > > > > >> > >> >> all of the interfaces related to Streams
> functionality
> > > are
> > > > > not.
> > > > > > >  The
> > > > > > > >> > >> >> current model maps to what Matt has outlined in my
> > > > opinion,
> > > > > > > though
> > > > > > > >> > >> >> packing names etc. probably don't follow that exact
> > > > pattern.
> > > > > > > >> > >> >>
> > > > > > > >> > >> >> With regards to the complexity and different
> > components
> > > in
> > > > > the
> > > > > > > >> > >> >> registration process, this was a cut at the
> > abstraction
> > > > > based
> > > > > > > on the
> > > > > > > >> > >> >> assumption that different implementations may be
> > plugged
> > > > in
> > > > > and
> > > > > > > in
> > > > > > > >> > >> >> fact may live on different processor space (ie. a
> > > polling
> > > > > > > publisher vs
> > > > > > > >> > >> >> a push publisher may be instantiated on different
> > > servers
> > > > > but
> > > > > > > the
> > > > > > > >> > >> >> registration URL is staticly defined).
> > > > > > > >> > >> >>
> > > > > > > >> > >> >> Is the main argument I am seeing for Spring  the
> > > > > familiarity of
> > > > > > > its
> > > > > > > >> > >> >> IoC pattern implementation and syntax at the entry
> > > point?
> > > > > > > >> > >> >>
> > > > > > > >> > >> >> On Wed, Oct 30, 2013 at 10:53 AM, Danny Sullivan <
> > > > > > > dsullivan7@hotmail.com> wrote:
> > > > > > > >> > >> >> > Could you clarify whether the same entry points
> > would
> > > > > exist
> > > > > > > for the camel implementation of the core (implementing the
> > > "process"
> > > > > > > method/ using a DynammicRouteBuilder) or would the webservice
> be
> > > the
> > > > > sole
> > > > > > > entry point to Streams and after it enters would it hand it off
> > to
> > > > > Camel?
> > > > > > > And what would be the entry point for the Storm implementation?
> > > > > > > >> > >> >> > -Danny
> > > > > > > >> > >> >> >
> > > > > > > >> > >> >> >> Date: Wed, 30 Oct 2013 10:13:04 -0400
> > > > > > > >> > >> >> >> Subject: Re: [DISCUSS] Switching Streams from
> Camel
> > > > > > > deployment to .war deployment
> > > > > > > >> > >> >> >> From: m.ben.franklin@gmail.com
> > > > > > > >> > >> >> >> To: dev@streams.incubator.apache.org
> > > > > > > >> > >> >> >>
> > > > > > > >> > >> >> >> On Tue, Oct 29, 2013 at 2:55 PM, Danny Sullivan <
> > > > > > > dsullivan7@hotmail.com>wrote:
> > > > > > > >> > >> >> >>
> > > > > > > >> > >> >> >> > My case for switching from OSGi is for
> simplicity
> > > in
> > > > > > > design. To follow the
> > > > > > > >> > >> >> >> > path of an activity through Streams in the
> > > > webservice,
> > > > > > > there is one main
> > > > > > > >> > >> >> >> > things the developer needs to understand:
> > > > > > > >> > >> >> >> > The @RequestMapping annotation specifies the
> HTTP
> > > > entry
> > > > > > > point
> > > > > > > >> > >> >> >> > There are 4 @RequestMapping annotations that
> > > > > correspond to
> > > > > > > each of the 4
> > > > > > > >> > >> >> >> > ways a user enters the application:
> registering a
> > > > > > > publisher, registering a
> > > > > > > >> > >> >> >> > subscriber, publishing activity, and getting an
> > > > > activity
> > > > > > > stream. Where
> > > > > > > >> > >> >> >> > these are located in the source code can be
> found
> > > by
> > > > > > > searching for the
> > > > > > > >> > >> >> >> > paths specified in the documentation (search
> for
> > > > > > > "/publisherRegister",
> > > > > > > >> > >> >> >> > "/subscriberRegister", "/publishActivity",
> > > > > "/getActivity"
> > > > > > > which will all
> > > > > > > >> > >> >> >> > lead you to StreamsWebController.java). From
> the
> > > > > methods
> > > > > > > that process
> > > > > > > >> > >> >> >> > requests, the flow through the application is
> > > through
> > > > > > > methods which can be
> > > > > > > >> > >> >> >> > understood by most Java programmers.
> > > > > > > >> > >> >> >> > The flow of activities through the current
> trunk
> > > > > branch is
> > > > > > > understood as
> > > > > > > >> > >> >> >> > follows:
> > > > > > > >> > >> >> >> > The string "/publisher/register" (the entry
> point
> > > to
> > > > > > > register a publisher
> > > > > > > >> > >> >> >> > specified in the documentation) is the value of
> > the
> > > > > > > >> > >> >> >> > consumer.registrationEndpoint property defined
> in
> > > > > > > streams.propertiesThe
> > > > > > > >> > >> >> >> > camelContext.xml specifies an endpoint with the
> > id
> > > > > > > >> > >> >> >> > consumerRegistrationEndpoint to have a uri
> equal
> > to
> > > > the
> > > > > > > propertyThe
> > > > > > > >> > >> >> >> > consumerRegistrationEndpoint routes from uri
> > > > > > > direct:publisher register with
> > > > > > > >> > >> >> >> > the bean activityRegistrationProcessor nested
> in
> > > > > between
> > > > > > > the routeThe
> > > > > > > >> > >> >> >> > streams-eip-applicationContext contains a bean
> > with
> > > > > the id
> > > > > > > >> > >> >> >> > activityRegistrationProcessor created for the
> > class
> > > > > > > >> > >> >> >> > ActivityPublisherRegistrationProcessorThe
> > exchange
> > > > will
> > > > > > > enter the "process"
> > > > > > > >> > >> >> >> > method of the class
> > > > > > > ActivityPublisherRegisitrationProcessor and that this
> > > > > > > >> > >> >> >> > is because this class implements the
> "Processor"
> > > > > interface
> > > > > > > provided by
> > > > > > > >> > >> >> >> > CamelThe direct:add-publisher-route takes the
> > > > exchange
> > > > > > > output from the
> > > > > > > >> > >> >> >> > "process" method and routes it to the
> > > > > > > activityRegistrationProcessor
> > > > > > > >> > >> >> >> > "register" method. The bean
> > > > > activityRegistrationProcessor
> > > > > > > is defined in the
> > > > > > > >> > >> >> >> > streams-eip-osgi-component-import.xmlThe output
> > > from
> > > > > this
> > > > > > > method is then
> > > > > > > >> > >> >> >> > sent to the "createNewRouteForConsumer" method
> of
> > > > > > > activityConsumerRouter.
> > > > > > > >> > >> >> >> > This method creates a new route for the newly
> > > > > registered
> > > > > > > publisher using
> > > > > > > >> > >> >> >> > the private static final class
> > > > > DynamicConsumerRouteBuilder
> > > > > > > which is
> > > > > > > >> > >> >> >> > required to extend RouteBuilder which is
> provided
> > > by
> > > > > > > Camel. This
> > > > > > > >> > >> >> >> > DynamicConsumerRouteBuilder contains several
> > > methods:
> > > > > > > >> > >> >> >> > getConsumerReceiveMethod() (which corresponds
> to
> > > > @Value
> > > > > > > >> > >> >> >> > ${consumer.receiveMethod} which corresponds to
> > > > > "receive"),
> > > > > > > >> > >> >> >> > getConsumerSplitMethod() (@Value
> > > > > > > ${consumer.splitMethod},"split"), and
> > > > > > > >> > >> >> >> > getConsumerActivityQUri() (@Value
> > > > > ${consumerActivityQUri},
> > > > > > > >> > >> >> >> > "direct:activityQ"). This is different than the
> > > > > > > camelContext.xml in that
> > > > > > > >> > >> >> >> > the route is being created programmatically.
> What
> > > > this
> > > > > is
> > > > > > > doing is routing
> > > > > > > >> > >> >> >> > input from the inroute url (which Camel does
> > > > > automatically
> > > > > > > through the
> > > > > > > >> > >> >> >> > configure method which is required to be
> > > overridden),
> > > > > to
> > > > > > > the "receive"
> > > > > > > >> > >> >> >> > method of ActivityConsumer, then to the "split"
> > > > method
> > > > > of
> > > > > > > ActivityConsumer,
> > > > > > > >> > >> >> >> > and then to the "direct:activityQ" which if you
> > > look
> > > > > back
> > > > > > > in the
> > > > > > > >> > >> >> >> > camelContext.xml routes to the
> > > > > "activemq:queue:activities"
> > > > > > > which then
> > > > > > > >> > >> >> >> > routes to "receiveExchange"
> > > > > > > >> > >> >> >> > This is the process to register a publisher.
> The
> > > > > process
> > > > > > > for registering a
> > > > > > > >> > >> >> >> > subscriber is relatively the same though it
> > > involves
> > > > > > > separate classes with
> > > > > > > >> > >> >> >> > their own private static final RouteBuilder
> > class.
> > > > > From my
> > > > > > > perspective, the
> > > > > > > >> > >> >> >> > two most difficult things with setting this
> > project
> > > > up
> > > > > > > were understanding
> > > > > > > >> > >> >> >> > that the "process" method of the class that
> > > > implements
> > > > > > > "Processor" is the
> > > > > > > >> > >> >> >> > entry point and the DynamicConsumerRouteBuilder
> > > > creates
> > > > > > > the second entry
> > > > > > > >> > >> >> >> > point (The 5th and last points). This made the
> > > > project
> > > > > > > very, VERY hard to
> > > > > > > >> > >> >> >> > understand.
> > > > > > > >> > >> >> >> > In addition to simplicity of design, the mvn
> > clean
> > > > > install
> > > > > > > of the web
> > > > > > > >> > >> >> >> > service project is much faster and small scale
> > > > activity
> > > > > > > publishing is also
> > > > > > > >> > >> >> >> > faster (see my email about load testing). These
> > are
> > > > > minor
> > > > > > > points though as
> > > > > > > >> > >> >> >> > compilation has no effect on deployment. OSGi
> > does
> > > > add
> > > > > the
> > > > > > > benefit of
> > > > > > > >> > >> >> >> > modularized programming which is valuable,
> > though I
> > > > > think
> > > > > > > the added
> > > > > > > >> > >> >> >> > complexity of Camel merits moving the project
> > away
> > > > from
> > > > > > > this paradigm.
> > > > > > > >> > >> >> >> >
> > > > > > > >> > >> >> >>
> > > > > > > >> > >> >> >> I agree that the project is pretty difficult to
> > > > > understand
> > > > > > > ATM.  I think
> > > > > > > >> > >> >> >> what we need to do is think about what the
> > > > > responsibilities
> > > > > > > of the code are
> > > > > > > >> > >> >> >> and allow for different implementations that are
> > not
> > > so
> > > > > > > tightly coupled as
> > > > > > > >> > >> >> >> they are now.  For instance, having worked with
> > Storm
> > > > to
> > > > > > > ingest millions of
> > > > > > > >> > >> >> >> activities a day, I personally would like to see
> > > > streams
> > > > > be
> > > > > > > responsible for
> > > > > > > >> > >> >> >> defining an over-arching orchestration model that
> > can
> > > > be
> > > > > > > implemented within
> > > > > > > >> > >> >> >> a single war or on top of a distributed system.
> >  This
> > > > > would
> > > > > > > look something
> > > > > > > >> > >> >> >> like the follows:
> > > > > > > >> > >> >> >>
> > > > > > > >> > >> >> >> |__ Streams-Core  (Base classes, interfaces,
> > > utilities,
> > > > > > > extensions, etc)
> > > > > > > >> > >> >> >> |
> > > > > > > >> > >> >> >> |__ Streams-Storm (Storm implementation of the
> > core)
> > > > > > > >> > >> >> >> |
> > > > > > > >> > >> >> >> |__ Streams-Camel (Camel implementation of the
> > core)
> > > > > > > >> > >> >> >> |
> > > > > > > >> > >> >> >> |__ Streams-WS (Web service implementation)
> > > > > > > >> > >> >> >>
> > > > > > > >> > >> >> >>
> > > > > > > >> > >> >> >> > Danny
> > > > > > > >> > >> >> >> >
> > > > > > > >> > >> >> >> >
> > > > > > > >> > >> >> >> > > From: dsullivan7@hotmail.com
> > > > > > > >> > >> >> >> > > To: dev@streams.incubator.apache.org
> > > > > > > >> > >> >> >> > > Subject: [DISCUSS] Switching Streams from
> Camel
> > > > > > > deployment to .war
> > > > > > > >> > >> >> >> > deployment
> > > > > > > >> > >> >> >> > > Date: Tue, 29 Oct 2013 11:07:39 -0400
> > > > > > > >> > >> >> >> > >
> > > > > > > >> > >> >> >> > > The discussion thread for switching Streams
> > from
> > > > > > > Camel/osgi/Servicemix
> > > > > > > >> > >> >> >> > to a single .war deployment
> > > > > > > >> > >> >> >> >
> > > > > > > >> > >> >> >> >
> > > > > > > >> > >> >> >
> > > > > > > >> > >> >
> > > > > > > >> > >
> > > > > > > >>
> > > > > > > >
> > > > > > >
> > > > >
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] Switching Streams from Camel deployment to .war deployment

[Chris Geer <ch...@cxtsoftware.com>]

On Thu, Oct 31, 2013 at 2:32 PM, Matt Franklin <m....@gmail.com>wrote:

> On Thu, Oct 31, 2013 at 5:28 PM, Chris Geer <ch...@cxtsoftware.com> wrote:
>
> > On Thu, Oct 31, 2013 at 2:23 PM, Matt Franklin <m.ben.franklin@gmail.com
> > >wrote:
> >
> > > On Thu, Oct 31, 2013 at 5:04 PM, Danny Sullivan <
> dsullivan7@hotmail.com
> > > >wrote:
> > >
> > > > I'm not quite following this, so I apologize. What I'm trying to do
> is
> > > > programmatically make a request to a jar running on a separate jvm
> and
> > > get
> > > > the response from that call all within the same method. Similar to
> this
> > > > http request:
> > > >
> > > > HttpGet httpget = new HttpGet();
> > > >
> > > > httpget.setURI(new URI("www.streams-persistence.com"));
> > > >
> > > > CloseableHttpResponse response = httpclient.execute(httpget);
> > > >
> > > > //do stuff with the response...
> > > >
> > > >
> > > >
> > > >
> > > > I imagine that this would translate to something in Camel similar to
> > > this:
> > > >
> > > >
> > > > <route>
> > > >
> > > > <from uri="bean:subscriberService?method=getAllSubscribers"/>
> > > >
> > > > <inOut
> > > >
> > >
> >
> uri="addressToStreamsPersistenceJar?method=selectAllSubscribersFromDatabase"/>
> > > >
> > > > </route>
> > > >
> > >
> > > Camel uses AMQP as a messaging system so that you don't have to create
> > HTTP
> > > requests between sources, though it is possible to do so.  In Storm, we
> > > usually try to use a buffer system like Kafka to do the same.
> > >
> >
> > Camel can use AMQP through ActiveMQ.
> >
> >
> Can being the operative word.  I haven't done much with Camel.  What other
> transports are available.
>
> OpenWire (native ActiveMQ)
AMQP
STOMP
HTTP (can be a server or client)
Mail
Amazon SQS
XMPP
....hundred or so more

http://camel.apache.org/components.html

>
> > >
> > >
> > > >
> > > >
> > > > But it is unclear what the actual implementation would be.
> > > >
> > > >
> > > > This actually brings me to another suggestion. Would there be a big
> > > > performance impact to have communication between the software
> > components
> > > > occur between http? Say the 5 software components I outlined earlier
> > were
> > > > packaged as 5 separate wars. These wars could communicate with each
> > other
> > > > via get a post requests. This sounds unconventional offhand so I'd
> like
> > > to
> > > > hear some thoughts on it.
> > > >
> > >
> > > It is most certainly possible, but IMO probably not the best option for
> > > success.  Protocols like PubSubHubbub use HTTP for a message transport.
> > >
> > >
> > > > -Danny
> > > >
> > > > > Date: Thu, 31 Oct 2013 13:28:41 -0700
> > > > > Subject: Re: [DISCUSS] Switching Streams from Camel deployment to
> > .war
> > > > deployment
> > > > > From: chris@cxtsoftware.com
> > > > > To: dev@streams.incubator.apache.org
> > > > >
> > > > > Or Content Enricher [2]
> > > > >
> > > > > [2] http://camel.apache.org/content-enricher.html
> > > > >
> > > > >
> > > > > On Thu, Oct 31, 2013 at 12:37 PM, Jason Letourneau
> > > > > <jl...@gmail.com>wrote:
> > > > >
> > > > > > check out the link here[1]
> > > > > >
> > > > > >
> > > > > > [1]http://camel.apache.org/request-reply.html
> > > > > >
> > > > > > On Thu, Oct 31, 2013 at 2:32 PM, Danny Sullivan <
> > > > dsullivan7@hotmail.com>
> > > > > > wrote:
> > > > > > > I have a quick Camel question that I arrived at in the
> > > > implementation of
> > > > > > these new components:
> > > > > > >
> > > > > > > Lets say I have a method in streams-activity.jar that needs all
> > > > > > subscribers in the database. This would require a call to the
> > > > > > streams-persistence.jar. So far, I've seen camel used mostly for
> > > > passing
> > > > > > data through the application, but not for making a single
> > > > request-reponse
> > > > > > from within a method. How can I use Camel to get a list of all
> > > > subscribers
> > > > > > in the streams-persistence.jar from the streams-activity.jar?
> > > > > > >
> > > > > > >> From: dsullivan7@hotmail.com
> > > > > > >> To: dev@streams.incubator.apache.org
> > > > > > >> Subject: RE: [DISCUSS] Switching Streams from Camel deployment
> > to
> > > > .war
> > > > > > deployment
> > > > > > >> Date: Thu, 31 Oct 2013 08:40:44 -0400
> > > > > > >>
> > > > > > >> Excellent, I'll write up something as a proof of concept and
> we
> > > can
> > > > > > discuss further to make sure everything is vanilla.
> > > > > > >>
> > > > > > >> > Date: Wed, 30 Oct 2013 17:01:28 -0400
> > > > > > >> > Subject: Re: [DISCUSS] Switching Streams from Camel
> deployment
> > > to
> > > > > > .war deployment
> > > > > > >> > From: jletourneau80@gmail.com
> > > > > > >> > To: dev@streams.incubator.apache.org
> > > > > > >> >
> > > > > > >> > That sounds pretty promising to me.
> > > > > > >> >
> > > > > > >> > On Wed, Oct 30, 2013 at 2:38 PM, Danny Sullivan <
> > > > > > dsullivan7@hotmail.com> wrote:
> > > > > > >> > > Thanks for the feedback. You have an interesting point
> about
> > > the
> > > > > > url linking to a separate processing space. Let me tie my answer
> > into
> > > > your
> > > > > > last question about "advocating for the simplicity at
> registration
> > to
> > > > give
> > > > > > up flexibility at registration, but retaining the inner "guts" of
> > > > > > EIP/messaging". Consider a new architecture:
> > > > > > >> > >
> > > > > > >> > > streams-web.war: single entry point to application, but
> > > > functions
> > > > > > ONLY as an entry point. From here Camel routes the incoming
> > requests
> > > > to 4
> > > > > > separate jarssubscriber-registration.jar: subscriber registration
> > > > > > publisher-registration.jar: publisher registrationactivity.jar:
> > > returns
> > > > > > activity (also contains subscriber warehouse and storm activity
> > > > > > aggregator)publish.jar: publishes activitystreams-cassandra.jar:
> > the
> > > > above
> > > > > > 4 jars would all have a hook into this jar which would function
> as
> > a
> > > > hook
> > > > > > onto the database. Each jar would have camel route output to this
> > > jar.
> > > > > > >> > >
> > > > > > >> > > In this implementation, Camel would no longer be the entry
> > and
> > > > exit
> > > > > > point of a client to the application, but would handle the
> > > > communication
> > > > > > between components. The flow of activity through the application
> > > would
> > > > be
> > > > > > method based in each jar. This would allow deployment on up to 6
> > > > different
> > > > > > process spaces. However, this does not address that there is a
> > single
> > > > > > server entry point, but I'm not sure if it was a concern in the
> > first
> > > > place.
> > > > > > >> > >
> > > > > > >> > > My argument, at its basis, is that we should move away
> from
> > > > using
> > > > > > Camel as the entry point to the application. I would be happy to
> > > > maintain
> > > > > > messaging between components.
> > > > > > >> > >
> > > > > > >> > >> Date: Wed, 30 Oct 2013 12:32:55 -0400
> > > > > > >> > >> Subject: Re: [DISCUSS] Switching Streams from Camel
> > > deployment
> > > > to
> > > > > > .war deployment
> > > > > > >> > >> From: jletourneau80@gmail.com
> > > > > > >> > >> To: dev@streams.incubator.apache.org
> > > > > > >> > >>
> > > > > > >> > >> An interesting use case that I am holding onto is the
> > ability
> > > > for
> > > > > > >> > >> publishers to register via a single URL (registration
> > > > endpoint),
> > > > > > but
> > > > > > >> > >> be sent a URL back to post to a different process space
> for
> > > > actual
> > > > > > >> > >> publishing.  The same is true on the subscriber front.
> > > >  Currently,
> > > > > > the
> > > > > > >> > >> Camel/EIP infrastructure abstracts this because different
> > > > > > components
> > > > > > >> > >> deployed in different process spaces handling the route
> > > > creation
> > > > > > can
> > > > > > >> > >> just be bolted onto a running Streams instance without
> new
> > > > > > subs/pubs
> > > > > > >> > >> behaving any differently than existing.  This seems to
> be a
> > > > > > >> > >> potentially critical scaling point.  Is there a way to do
> > > this
> > > > with
> > > > > > >> > >> the Spring solution?
> > > > > > >> > >>
> > > > > > >> > >> The persistence point is a good one, though I would
> > classify
> > > > that
> > > > > > as
> > > > > > >> > >> "not implemented" vs "not possible" (not that you were).
> > > > > > >> > >>
> > > > > > >> > >> I'm not married to Camel, I just like the EIP approach to
> > > > building
> > > > > > >> > >> something that is ultimately a messaging system.  There
> are
> > > > known
> > > > > > >> > >> patterns that solve at least a subset of the problems
> > Streams
> > > > is
> > > > > > >> > >> trying to solve and implementations that can handle the
> > load
> > > > and
> > > > > > I'll
> > > > > > >> > >> reiterate flexibility == complexity almost always.
> > > > > > >> > >>
> > > > > > >> > >> It comes right back to the central question: Do you want
> > > > > > flexibility
> > > > > > >> > >> or simplicity?  It doesn't have to be black and white
> > either
> > > I
> > > > > > don't
> > > > > > >> > >> think...
> > > > > > >> > >>
> > > > > > >> > >> More pointedly: Where should we give up flexibility for
> > > > > > simplicity?  I
> > > > > > >> > >> read that Danny is advocating for the simplicity at
> > > > registration to
> > > > > > >> > >> give up flexibility at registration, but retaining the
> > inner
> > > > > > "guts" of
> > > > > > >> > >> EIP/messaging?  Thoughts?
> > > > > > >> > >>
> > > > > > >> > >> On Wed, Oct 30, 2013 at 11:37 AM, Danny Sullivan <
> > > > > > dsullivan7@hotmail.com> wrote:
> > > > > > >> > >> > My argument is not for the IoC pattern as that can be
> > (and
> > > > has
> > > > > > been) implemented alongside Camel. My main argument is that the
> > > syntax
> > > > at
> > > > > > the entry point is not only familiar but much simpler. This
> > wouldn't
> > > > be a
> > > > > > very strong argument if the Camel implementation wasn't much more
> > > > > > complicated but I feel that it is the case. Also, looking toward
> > the
> > > > > > future, if the server is restarted, in-routes are lost in Camel.
> > The
> > > > way to
> > > > > > curb this is to persist the dynamic routes that Camel creates,
> and
> > > > then on
> > > > > > start up pull every one of these routes and recreate a dynamic
> > route
> > > > for
> > > > > > each one. Not only is this much easier to implement using the
> > Spring
> > > > web
> > > > > > implementation, but it already has been implemented and you can
> try
> > > it
> > > > by
> > > > > > checking out the webservice branch, registering a subscriber,
> > > > restarting
> > > > > > tomcat, and using the same url you had before. This will allow
> > > > subscribers
> > > > > > to hang on to their urls once they register. (the same is true
> for
> > > > > > publishers: you can post via the same url after restarting
> tomcat)
> > > > > > >> > >> >
> > > > > > >> > >> >> Date: Wed, 30 Oct 2013 11:00:21 -0400
> > > > > > >> > >> >> Subject: Re: [DISCUSS] Switching Streams from Camel
> > > > deployment
> > > > > > to .war deployment
> > > > > > >> > >> >> From: jletourneau80@gmail.com
> > > > > > >> > >> >> To: dev@streams.incubator.apache.org
> > > > > > >> > >> >>
> > > > > > >> > >> >> To be fair, while the current implementation is
> heavily
> > > > > > camel-based,
> > > > > > >> > >> >> all of the interfaces related to Streams functionality
> > are
> > > > not.
> > > > > >  The
> > > > > > >> > >> >> current model maps to what Matt has outlined in my
> > > opinion,
> > > > > > though
> > > > > > >> > >> >> packing names etc. probably don't follow that exact
> > > pattern.
> > > > > > >> > >> >>
> > > > > > >> > >> >> With regards to the complexity and different
> components
> > in
> > > > the
> > > > > > >> > >> >> registration process, this was a cut at the
> abstraction
> > > > based
> > > > > > on the
> > > > > > >> > >> >> assumption that different implementations may be
> plugged
> > > in
> > > > and
> > > > > > in
> > > > > > >> > >> >> fact may live on different processor space (ie. a
> > polling
> > > > > > publisher vs
> > > > > > >> > >> >> a push publisher may be instantiated on different
> > servers
> > > > but
> > > > > > the
> > > > > > >> > >> >> registration URL is staticly defined).
> > > > > > >> > >> >>
> > > > > > >> > >> >> Is the main argument I am seeing for Spring  the
> > > > familiarity of
> > > > > > its
> > > > > > >> > >> >> IoC pattern implementation and syntax at the entry
> > point?
> > > > > > >> > >> >>
> > > > > > >> > >> >> On Wed, Oct 30, 2013 at 10:53 AM, Danny Sullivan <
> > > > > > dsullivan7@hotmail.com> wrote:
> > > > > > >> > >> >> > Could you clarify whether the same entry points
> would
> > > > exist
> > > > > > for the camel implementation of the core (implementing the
> > "process"
> > > > > > method/ using a DynammicRouteBuilder) or would the webservice be
> > the
> > > > sole
> > > > > > entry point to Streams and after it enters would it hand it off
> to
> > > > Camel?
> > > > > > And what would be the entry point for the Storm implementation?
> > > > > > >> > >> >> > -Danny
> > > > > > >> > >> >> >
> > > > > > >> > >> >> >> Date: Wed, 30 Oct 2013 10:13:04 -0400
> > > > > > >> > >> >> >> Subject: Re: [DISCUSS] Switching Streams from Camel
> > > > > > deployment to .war deployment
> > > > > > >> > >> >> >> From: m.ben.franklin@gmail.com
> > > > > > >> > >> >> >> To: dev@streams.incubator.apache.org
> > > > > > >> > >> >> >>
> > > > > > >> > >> >> >> On Tue, Oct 29, 2013 at 2:55 PM, Danny Sullivan <
> > > > > > dsullivan7@hotmail.com>wrote:
> > > > > > >> > >> >> >>
> > > > > > >> > >> >> >> > My case for switching from OSGi is for simplicity
> > in
> > > > > > design. To follow the
> > > > > > >> > >> >> >> > path of an activity through Streams in the
> > > webservice,
> > > > > > there is one main
> > > > > > >> > >> >> >> > things the developer needs to understand:
> > > > > > >> > >> >> >> > The @RequestMapping annotation specifies the HTTP
> > > entry
> > > > > > point
> > > > > > >> > >> >> >> > There are 4 @RequestMapping annotations that
> > > > correspond to
> > > > > > each of the 4
> > > > > > >> > >> >> >> > ways a user enters the application: registering a
> > > > > > publisher, registering a
> > > > > > >> > >> >> >> > subscriber, publishing activity, and getting an
> > > > activity
> > > > > > stream. Where
> > > > > > >> > >> >> >> > these are located in the source code can be found
> > by
> > > > > > searching for the
> > > > > > >> > >> >> >> > paths specified in the documentation (search for
> > > > > > "/publisherRegister",
> > > > > > >> > >> >> >> > "/subscriberRegister", "/publishActivity",
> > > > "/getActivity"
> > > > > > which will all
> > > > > > >> > >> >> >> > lead you to StreamsWebController.java). From the
> > > > methods
> > > > > > that process
> > > > > > >> > >> >> >> > requests, the flow through the application is
> > through
> > > > > > methods which can be
> > > > > > >> > >> >> >> > understood by most Java programmers.
> > > > > > >> > >> >> >> > The flow of activities through the current trunk
> > > > branch is
> > > > > > understood as
> > > > > > >> > >> >> >> > follows:
> > > > > > >> > >> >> >> > The string "/publisher/register" (the entry point
> > to
> > > > > > register a publisher
> > > > > > >> > >> >> >> > specified in the documentation) is the value of
> the
> > > > > > >> > >> >> >> > consumer.registrationEndpoint property defined in
> > > > > > streams.propertiesThe
> > > > > > >> > >> >> >> > camelContext.xml specifies an endpoint with the
> id
> > > > > > >> > >> >> >> > consumerRegistrationEndpoint to have a uri equal
> to
> > > the
> > > > > > propertyThe
> > > > > > >> > >> >> >> > consumerRegistrationEndpoint routes from uri
> > > > > > direct:publisher register with
> > > > > > >> > >> >> >> > the bean activityRegistrationProcessor nested in
> > > > between
> > > > > > the routeThe
> > > > > > >> > >> >> >> > streams-eip-applicationContext contains a bean
> with
> > > > the id
> > > > > > >> > >> >> >> > activityRegistrationProcessor created for the
> class
> > > > > > >> > >> >> >> > ActivityPublisherRegistrationProcessorThe
> exchange
> > > will
> > > > > > enter the "process"
> > > > > > >> > >> >> >> > method of the class
> > > > > > ActivityPublisherRegisitrationProcessor and that this
> > > > > > >> > >> >> >> > is because this class implements the "Processor"
> > > > interface
> > > > > > provided by
> > > > > > >> > >> >> >> > CamelThe direct:add-publisher-route takes the
> > > exchange
> > > > > > output from the
> > > > > > >> > >> >> >> > "process" method and routes it to the
> > > > > > activityRegistrationProcessor
> > > > > > >> > >> >> >> > "register" method. The bean
> > > > activityRegistrationProcessor
> > > > > > is defined in the
> > > > > > >> > >> >> >> > streams-eip-osgi-component-import.xmlThe output
> > from
> > > > this
> > > > > > method is then
> > > > > > >> > >> >> >> > sent to the "createNewRouteForConsumer" method of
> > > > > > activityConsumerRouter.
> > > > > > >> > >> >> >> > This method creates a new route for the newly
> > > > registered
> > > > > > publisher using
> > > > > > >> > >> >> >> > the private static final class
> > > > DynamicConsumerRouteBuilder
> > > > > > which is
> > > > > > >> > >> >> >> > required to extend RouteBuilder which is provided
> > by
> > > > > > Camel. This
> > > > > > >> > >> >> >> > DynamicConsumerRouteBuilder contains several
> > methods:
> > > > > > >> > >> >> >> > getConsumerReceiveMethod() (which corresponds to
> > > @Value
> > > > > > >> > >> >> >> > ${consumer.receiveMethod} which corresponds to
> > > > "receive"),
> > > > > > >> > >> >> >> > getConsumerSplitMethod() (@Value
> > > > > > ${consumer.splitMethod},"split"), and
> > > > > > >> > >> >> >> > getConsumerActivityQUri() (@Value
> > > > ${consumerActivityQUri},
> > > > > > >> > >> >> >> > "direct:activityQ"). This is different than the
> > > > > > camelContext.xml in that
> > > > > > >> > >> >> >> > the route is being created programmatically. What
> > > this
> > > > is
> > > > > > doing is routing
> > > > > > >> > >> >> >> > input from the inroute url (which Camel does
> > > > automatically
> > > > > > through the
> > > > > > >> > >> >> >> > configure method which is required to be
> > overridden),
> > > > to
> > > > > > the "receive"
> > > > > > >> > >> >> >> > method of ActivityConsumer, then to the "split"
> > > method
> > > > of
> > > > > > ActivityConsumer,
> > > > > > >> > >> >> >> > and then to the "direct:activityQ" which if you
> > look
> > > > back
> > > > > > in the
> > > > > > >> > >> >> >> > camelContext.xml routes to the
> > > > "activemq:queue:activities"
> > > > > > which then
> > > > > > >> > >> >> >> > routes to "receiveExchange"
> > > > > > >> > >> >> >> > This is the process to register a publisher. The
> > > > process
> > > > > > for registering a
> > > > > > >> > >> >> >> > subscriber is relatively the same though it
> > involves
> > > > > > separate classes with
> > > > > > >> > >> >> >> > their own private static final RouteBuilder
> class.
> > > > From my
> > > > > > perspective, the
> > > > > > >> > >> >> >> > two most difficult things with setting this
> project
> > > up
> > > > > > were understanding
> > > > > > >> > >> >> >> > that the "process" method of the class that
> > > implements
> > > > > > "Processor" is the
> > > > > > >> > >> >> >> > entry point and the DynamicConsumerRouteBuilder
> > > creates
> > > > > > the second entry
> > > > > > >> > >> >> >> > point (The 5th and last points). This made the
> > > project
> > > > > > very, VERY hard to
> > > > > > >> > >> >> >> > understand.
> > > > > > >> > >> >> >> > In addition to simplicity of design, the mvn
> clean
> > > > install
> > > > > > of the web
> > > > > > >> > >> >> >> > service project is much faster and small scale
> > > activity
> > > > > > publishing is also
> > > > > > >> > >> >> >> > faster (see my email about load testing). These
> are
> > > > minor
> > > > > > points though as
> > > > > > >> > >> >> >> > compilation has no effect on deployment. OSGi
> does
> > > add
> > > > the
> > > > > > benefit of
> > > > > > >> > >> >> >> > modularized programming which is valuable,
> though I
> > > > think
> > > > > > the added
> > > > > > >> > >> >> >> > complexity of Camel merits moving the project
> away
> > > from
> > > > > > this paradigm.
> > > > > > >> > >> >> >> >
> > > > > > >> > >> >> >>
> > > > > > >> > >> >> >> I agree that the project is pretty difficult to
> > > > understand
> > > > > > ATM.  I think
> > > > > > >> > >> >> >> what we need to do is think about what the
> > > > responsibilities
> > > > > > of the code are
> > > > > > >> > >> >> >> and allow for different implementations that are
> not
> > so
> > > > > > tightly coupled as
> > > > > > >> > >> >> >> they are now.  For instance, having worked with
> Storm
> > > to
> > > > > > ingest millions of
> > > > > > >> > >> >> >> activities a day, I personally would like to see
> > > streams
> > > > be
> > > > > > responsible for
> > > > > > >> > >> >> >> defining an over-arching orchestration model that
> can
> > > be
> > > > > > implemented within
> > > > > > >> > >> >> >> a single war or on top of a distributed system.
>  This
> > > > would
> > > > > > look something
> > > > > > >> > >> >> >> like the follows:
> > > > > > >> > >> >> >>
> > > > > > >> > >> >> >> |__ Streams-Core  (Base classes, interfaces,
> > utilities,
> > > > > > extensions, etc)
> > > > > > >> > >> >> >> |
> > > > > > >> > >> >> >> |__ Streams-Storm (Storm implementation of the
> core)
> > > > > > >> > >> >> >> |
> > > > > > >> > >> >> >> |__ Streams-Camel (Camel implementation of the
> core)
> > > > > > >> > >> >> >> |
> > > > > > >> > >> >> >> |__ Streams-WS (Web service implementation)
> > > > > > >> > >> >> >>
> > > > > > >> > >> >> >>
> > > > > > >> > >> >> >> > Danny
> > > > > > >> > >> >> >> >
> > > > > > >> > >> >> >> >
> > > > > > >> > >> >> >> > > From: dsullivan7@hotmail.com
> > > > > > >> > >> >> >> > > To: dev@streams.incubator.apache.org
> > > > > > >> > >> >> >> > > Subject: [DISCUSS] Switching Streams from Camel
> > > > > > deployment to .war
> > > > > > >> > >> >> >> > deployment
> > > > > > >> > >> >> >> > > Date: Tue, 29 Oct 2013 11:07:39 -0400
> > > > > > >> > >> >> >> > >
> > > > > > >> > >> >> >> > > The discussion thread for switching Streams
> from
> > > > > > Camel/osgi/Servicemix
> > > > > > >> > >> >> >> > to a single .war deployment
> > > > > > >> > >> >> >> >
> > > > > > >> > >> >> >> >
> > > > > > >> > >> >> >
> > > > > > >> > >> >
> > > > > > >> > >
> > > > > > >>
> > > > > > >
> > > > > >
> > > >
> > > >
> > >
> >
>

Re: [DISCUSS] Switching Streams from Camel deployment to .war deployment

[Matt Franklin <m....@gmail.com>]

On Thu, Oct 31, 2013 at 5:28 PM, Chris Geer <ch...@cxtsoftware.com> wrote:

> On Thu, Oct 31, 2013 at 2:23 PM, Matt Franklin <m.ben.franklin@gmail.com
> >wrote:
>
> > On Thu, Oct 31, 2013 at 5:04 PM, Danny Sullivan <dsullivan7@hotmail.com
> > >wrote:
> >
> > > I'm not quite following this, so I apologize. What I'm trying to do is
> > > programmatically make a request to a jar running on a separate jvm and
> > get
> > > the response from that call all within the same method. Similar to this
> > > http request:
> > >
> > > HttpGet httpget = new HttpGet();
> > >
> > > httpget.setURI(new URI("www.streams-persistence.com"));
> > >
> > > CloseableHttpResponse response = httpclient.execute(httpget);
> > >
> > > //do stuff with the response...
> > >
> > >
> > >
> > >
> > > I imagine that this would translate to something in Camel similar to
> > this:
> > >
> > >
> > > <route>
> > >
> > > <from uri="bean:subscriberService?method=getAllSubscribers"/>
> > >
> > > <inOut
> > >
> >
> uri="addressToStreamsPersistenceJar?method=selectAllSubscribersFromDatabase"/>
> > >
> > > </route>
> > >
> >
> > Camel uses AMQP as a messaging system so that you don't have to create
> HTTP
> > requests between sources, though it is possible to do so.  In Storm, we
> > usually try to use a buffer system like Kafka to do the same.
> >
>
> Camel can use AMQP through ActiveMQ.
>
>
Can being the operative word.  I haven't done much with Camel.  What other
transports are available.


> >
> >
> > >
> > >
> > > But it is unclear what the actual implementation would be.
> > >
> > >
> > > This actually brings me to another suggestion. Would there be a big
> > > performance impact to have communication between the software
> components
> > > occur between http? Say the 5 software components I outlined earlier
> were
> > > packaged as 5 separate wars. These wars could communicate with each
> other
> > > via get a post requests. This sounds unconventional offhand so I'd like
> > to
> > > hear some thoughts on it.
> > >
> >
> > It is most certainly possible, but IMO probably not the best option for
> > success.  Protocols like PubSubHubbub use HTTP for a message transport.
> >
> >
> > > -Danny
> > >
> > > > Date: Thu, 31 Oct 2013 13:28:41 -0700
> > > > Subject: Re: [DISCUSS] Switching Streams from Camel deployment to
> .war
> > > deployment
> > > > From: chris@cxtsoftware.com
> > > > To: dev@streams.incubator.apache.org
> > > >
> > > > Or Content Enricher [2]
> > > >
> > > > [2] http://camel.apache.org/content-enricher.html
> > > >
> > > >
> > > > On Thu, Oct 31, 2013 at 12:37 PM, Jason Letourneau
> > > > <jl...@gmail.com>wrote:
> > > >
> > > > > check out the link here[1]
> > > > >
> > > > >
> > > > > [1]http://camel.apache.org/request-reply.html
> > > > >
> > > > > On Thu, Oct 31, 2013 at 2:32 PM, Danny Sullivan <
> > > dsullivan7@hotmail.com>
> > > > > wrote:
> > > > > > I have a quick Camel question that I arrived at in the
> > > implementation of
> > > > > these new components:
> > > > > >
> > > > > > Lets say I have a method in streams-activity.jar that needs all
> > > > > subscribers in the database. This would require a call to the
> > > > > streams-persistence.jar. So far, I've seen camel used mostly for
> > > passing
> > > > > data through the application, but not for making a single
> > > request-reponse
> > > > > from within a method. How can I use Camel to get a list of all
> > > subscribers
> > > > > in the streams-persistence.jar from the streams-activity.jar?
> > > > > >
> > > > > >> From: dsullivan7@hotmail.com
> > > > > >> To: dev@streams.incubator.apache.org
> > > > > >> Subject: RE: [DISCUSS] Switching Streams from Camel deployment
> to
> > > .war
> > > > > deployment
> > > > > >> Date: Thu, 31 Oct 2013 08:40:44 -0400
> > > > > >>
> > > > > >> Excellent, I'll write up something as a proof of concept and we
> > can
> > > > > discuss further to make sure everything is vanilla.
> > > > > >>
> > > > > >> > Date: Wed, 30 Oct 2013 17:01:28 -0400
> > > > > >> > Subject: Re: [DISCUSS] Switching Streams from Camel deployment
> > to
> > > > > .war deployment
> > > > > >> > From: jletourneau80@gmail.com
> > > > > >> > To: dev@streams.incubator.apache.org
> > > > > >> >
> > > > > >> > That sounds pretty promising to me.
> > > > > >> >
> > > > > >> > On Wed, Oct 30, 2013 at 2:38 PM, Danny Sullivan <
> > > > > dsullivan7@hotmail.com> wrote:
> > > > > >> > > Thanks for the feedback. You have an interesting point about
> > the
> > > > > url linking to a separate processing space. Let me tie my answer
> into
> > > your
> > > > > last question about "advocating for the simplicity at registration
> to
> > > give
> > > > > up flexibility at registration, but retaining the inner "guts" of
> > > > > EIP/messaging". Consider a new architecture:
> > > > > >> > >
> > > > > >> > > streams-web.war: single entry point to application, but
> > > functions
> > > > > ONLY as an entry point. From here Camel routes the incoming
> requests
> > > to 4
> > > > > separate jarssubscriber-registration.jar: subscriber registration
> > > > > publisher-registration.jar: publisher registrationactivity.jar:
> > returns
> > > > > activity (also contains subscriber warehouse and storm activity
> > > > > aggregator)publish.jar: publishes activitystreams-cassandra.jar:
> the
> > > above
> > > > > 4 jars would all have a hook into this jar which would function as
> a
> > > hook
> > > > > onto the database. Each jar would have camel route output to this
> > jar.
> > > > > >> > >
> > > > > >> > > In this implementation, Camel would no longer be the entry
> and
> > > exit
> > > > > point of a client to the application, but would handle the
> > > communication
> > > > > between components. The flow of activity through the application
> > would
> > > be
> > > > > method based in each jar. This would allow deployment on up to 6
> > > different
> > > > > process spaces. However, this does not address that there is a
> single
> > > > > server entry point, but I'm not sure if it was a concern in the
> first
> > > place.
> > > > > >> > >
> > > > > >> > > My argument, at its basis, is that we should move away from
> > > using
> > > > > Camel as the entry point to the application. I would be happy to
> > > maintain
> > > > > messaging between components.
> > > > > >> > >
> > > > > >> > >> Date: Wed, 30 Oct 2013 12:32:55 -0400
> > > > > >> > >> Subject: Re: [DISCUSS] Switching Streams from Camel
> > deployment
> > > to
> > > > > .war deployment
> > > > > >> > >> From: jletourneau80@gmail.com
> > > > > >> > >> To: dev@streams.incubator.apache.org
> > > > > >> > >>
> > > > > >> > >> An interesting use case that I am holding onto is the
> ability
> > > for
> > > > > >> > >> publishers to register via a single URL (registration
> > > endpoint),
> > > > > but
> > > > > >> > >> be sent a URL back to post to a different process space for
> > > actual
> > > > > >> > >> publishing.  The same is true on the subscriber front.
> > >  Currently,
> > > > > the
> > > > > >> > >> Camel/EIP infrastructure abstracts this because different
> > > > > components
> > > > > >> > >> deployed in different process spaces handling the route
> > > creation
> > > > > can
> > > > > >> > >> just be bolted onto a running Streams instance without new
> > > > > subs/pubs
> > > > > >> > >> behaving any differently than existing.  This seems to be a
> > > > > >> > >> potentially critical scaling point.  Is there a way to do
> > this
> > > with
> > > > > >> > >> the Spring solution?
> > > > > >> > >>
> > > > > >> > >> The persistence point is a good one, though I would
> classify
> > > that
> > > > > as
> > > > > >> > >> "not implemented" vs "not possible" (not that you were).
> > > > > >> > >>
> > > > > >> > >> I'm not married to Camel, I just like the EIP approach to
> > > building
> > > > > >> > >> something that is ultimately a messaging system.  There are
> > > known
> > > > > >> > >> patterns that solve at least a subset of the problems
> Streams
> > > is
> > > > > >> > >> trying to solve and implementations that can handle the
> load
> > > and
> > > > > I'll
> > > > > >> > >> reiterate flexibility == complexity almost always.
> > > > > >> > >>
> > > > > >> > >> It comes right back to the central question: Do you want
> > > > > flexibility
> > > > > >> > >> or simplicity?  It doesn't have to be black and white
> either
> > I
> > > > > don't
> > > > > >> > >> think...
> > > > > >> > >>
> > > > > >> > >> More pointedly: Where should we give up flexibility for
> > > > > simplicity?  I
> > > > > >> > >> read that Danny is advocating for the simplicity at
> > > registration to
> > > > > >> > >> give up flexibility at registration, but retaining the
> inner
> > > > > "guts" of
> > > > > >> > >> EIP/messaging?  Thoughts?
> > > > > >> > >>
> > > > > >> > >> On Wed, Oct 30, 2013 at 11:37 AM, Danny Sullivan <
> > > > > dsullivan7@hotmail.com> wrote:
> > > > > >> > >> > My argument is not for the IoC pattern as that can be
> (and
> > > has
> > > > > been) implemented alongside Camel. My main argument is that the
> > syntax
> > > at
> > > > > the entry point is not only familiar but much simpler. This
> wouldn't
> > > be a
> > > > > very strong argument if the Camel implementation wasn't much more
> > > > > complicated but I feel that it is the case. Also, looking toward
> the
> > > > > future, if the server is restarted, in-routes are lost in Camel.
> The
> > > way to
> > > > > curb this is to persist the dynamic routes that Camel creates, and
> > > then on
> > > > > start up pull every one of these routes and recreate a dynamic
> route
> > > for
> > > > > each one. Not only is this much easier to implement using the
> Spring
> > > web
> > > > > implementation, but it already has been implemented and you can try
> > it
> > > by
> > > > > checking out the webservice branch, registering a subscriber,
> > > restarting
> > > > > tomcat, and using the same url you had before. This will allow
> > > subscribers
> > > > > to hang on to their urls once they register. (the same is true for
> > > > > publishers: you can post via the same url after restarting tomcat)
> > > > > >> > >> >
> > > > > >> > >> >> Date: Wed, 30 Oct 2013 11:00:21 -0400
> > > > > >> > >> >> Subject: Re: [DISCUSS] Switching Streams from Camel
> > > deployment
> > > > > to .war deployment
> > > > > >> > >> >> From: jletourneau80@gmail.com
> > > > > >> > >> >> To: dev@streams.incubator.apache.org
> > > > > >> > >> >>
> > > > > >> > >> >> To be fair, while the current implementation is heavily
> > > > > camel-based,
> > > > > >> > >> >> all of the interfaces related to Streams functionality
> are
> > > not.
> > > > >  The
> > > > > >> > >> >> current model maps to what Matt has outlined in my
> > opinion,
> > > > > though
> > > > > >> > >> >> packing names etc. probably don't follow that exact
> > pattern.
> > > > > >> > >> >>
> > > > > >> > >> >> With regards to the complexity and different components
> in
> > > the
> > > > > >> > >> >> registration process, this was a cut at the abstraction
> > > based
> > > > > on the
> > > > > >> > >> >> assumption that different implementations may be plugged
> > in
> > > and
> > > > > in
> > > > > >> > >> >> fact may live on different processor space (ie. a
> polling
> > > > > publisher vs
> > > > > >> > >> >> a push publisher may be instantiated on different
> servers
> > > but
> > > > > the
> > > > > >> > >> >> registration URL is staticly defined).
> > > > > >> > >> >>
> > > > > >> > >> >> Is the main argument I am seeing for Spring  the
> > > familiarity of
> > > > > its
> > > > > >> > >> >> IoC pattern implementation and syntax at the entry
> point?
> > > > > >> > >> >>
> > > > > >> > >> >> On Wed, Oct 30, 2013 at 10:53 AM, Danny Sullivan <
> > > > > dsullivan7@hotmail.com> wrote:
> > > > > >> > >> >> > Could you clarify whether the same entry points would
> > > exist
> > > > > for the camel implementation of the core (implementing the
> "process"
> > > > > method/ using a DynammicRouteBuilder) or would the webservice be
> the
> > > sole
> > > > > entry point to Streams and after it enters would it hand it off to
> > > Camel?
> > > > > And what would be the entry point for the Storm implementation?
> > > > > >> > >> >> > -Danny
> > > > > >> > >> >> >
> > > > > >> > >> >> >> Date: Wed, 30 Oct 2013 10:13:04 -0400
> > > > > >> > >> >> >> Subject: Re: [DISCUSS] Switching Streams from Camel
> > > > > deployment to .war deployment
> > > > > >> > >> >> >> From: m.ben.franklin@gmail.com
> > > > > >> > >> >> >> To: dev@streams.incubator.apache.org
> > > > > >> > >> >> >>
> > > > > >> > >> >> >> On Tue, Oct 29, 2013 at 2:55 PM, Danny Sullivan <
> > > > > dsullivan7@hotmail.com>wrote:
> > > > > >> > >> >> >>
> > > > > >> > >> >> >> > My case for switching from OSGi is for simplicity
> in
> > > > > design. To follow the
> > > > > >> > >> >> >> > path of an activity through Streams in the
> > webservice,
> > > > > there is one main
> > > > > >> > >> >> >> > things the developer needs to understand:
> > > > > >> > >> >> >> > The @RequestMapping annotation specifies the HTTP
> > entry
> > > > > point
> > > > > >> > >> >> >> > There are 4 @RequestMapping annotations that
> > > correspond to
> > > > > each of the 4
> > > > > >> > >> >> >> > ways a user enters the application: registering a
> > > > > publisher, registering a
> > > > > >> > >> >> >> > subscriber, publishing activity, and getting an
> > > activity
> > > > > stream. Where
> > > > > >> > >> >> >> > these are located in the source code can be found
> by
> > > > > searching for the
> > > > > >> > >> >> >> > paths specified in the documentation (search for
> > > > > "/publisherRegister",
> > > > > >> > >> >> >> > "/subscriberRegister", "/publishActivity",
> > > "/getActivity"
> > > > > which will all
> > > > > >> > >> >> >> > lead you to StreamsWebController.java). From the
> > > methods
> > > > > that process
> > > > > >> > >> >> >> > requests, the flow through the application is
> through
> > > > > methods which can be
> > > > > >> > >> >> >> > understood by most Java programmers.
> > > > > >> > >> >> >> > The flow of activities through the current trunk
> > > branch is
> > > > > understood as
> > > > > >> > >> >> >> > follows:
> > > > > >> > >> >> >> > The string "/publisher/register" (the entry point
> to
> > > > > register a publisher
> > > > > >> > >> >> >> > specified in the documentation) is the value of the
> > > > > >> > >> >> >> > consumer.registrationEndpoint property defined in
> > > > > streams.propertiesThe
> > > > > >> > >> >> >> > camelContext.xml specifies an endpoint with the id
> > > > > >> > >> >> >> > consumerRegistrationEndpoint to have a uri equal to
> > the
> > > > > propertyThe
> > > > > >> > >> >> >> > consumerRegistrationEndpoint routes from uri
> > > > > direct:publisher register with
> > > > > >> > >> >> >> > the bean activityRegistrationProcessor nested in
> > > between
> > > > > the routeThe
> > > > > >> > >> >> >> > streams-eip-applicationContext contains a bean with
> > > the id
> > > > > >> > >> >> >> > activityRegistrationProcessor created for the class
> > > > > >> > >> >> >> > ActivityPublisherRegistrationProcessorThe exchange
> > will
> > > > > enter the "process"
> > > > > >> > >> >> >> > method of the class
> > > > > ActivityPublisherRegisitrationProcessor and that this
> > > > > >> > >> >> >> > is because this class implements the "Processor"
> > > interface
> > > > > provided by
> > > > > >> > >> >> >> > CamelThe direct:add-publisher-route takes the
> > exchange
> > > > > output from the
> > > > > >> > >> >> >> > "process" method and routes it to the
> > > > > activityRegistrationProcessor
> > > > > >> > >> >> >> > "register" method. The bean
> > > activityRegistrationProcessor
> > > > > is defined in the
> > > > > >> > >> >> >> > streams-eip-osgi-component-import.xmlThe output
> from
> > > this
> > > > > method is then
> > > > > >> > >> >> >> > sent to the "createNewRouteForConsumer" method of
> > > > > activityConsumerRouter.
> > > > > >> > >> >> >> > This method creates a new route for the newly
> > > registered
> > > > > publisher using
> > > > > >> > >> >> >> > the private static final class
> > > DynamicConsumerRouteBuilder
> > > > > which is
> > > > > >> > >> >> >> > required to extend RouteBuilder which is provided
> by
> > > > > Camel. This
> > > > > >> > >> >> >> > DynamicConsumerRouteBuilder contains several
> methods:
> > > > > >> > >> >> >> > getConsumerReceiveMethod() (which corresponds to
> > @Value
> > > > > >> > >> >> >> > ${consumer.receiveMethod} which corresponds to
> > > "receive"),
> > > > > >> > >> >> >> > getConsumerSplitMethod() (@Value
> > > > > ${consumer.splitMethod},"split"), and
> > > > > >> > >> >> >> > getConsumerActivityQUri() (@Value
> > > ${consumerActivityQUri},
> > > > > >> > >> >> >> > "direct:activityQ"). This is different than the
> > > > > camelContext.xml in that
> > > > > >> > >> >> >> > the route is being created programmatically. What
> > this
> > > is
> > > > > doing is routing
> > > > > >> > >> >> >> > input from the inroute url (which Camel does
> > > automatically
> > > > > through the
> > > > > >> > >> >> >> > configure method which is required to be
> overridden),
> > > to
> > > > > the "receive"
> > > > > >> > >> >> >> > method of ActivityConsumer, then to the "split"
> > method
> > > of
> > > > > ActivityConsumer,
> > > > > >> > >> >> >> > and then to the "direct:activityQ" which if you
> look
> > > back
> > > > > in the
> > > > > >> > >> >> >> > camelContext.xml routes to the
> > > "activemq:queue:activities"
> > > > > which then
> > > > > >> > >> >> >> > routes to "receiveExchange"
> > > > > >> > >> >> >> > This is the process to register a publisher. The
> > > process
> > > > > for registering a
> > > > > >> > >> >> >> > subscriber is relatively the same though it
> involves
> > > > > separate classes with
> > > > > >> > >> >> >> > their own private static final RouteBuilder class.
> > > From my
> > > > > perspective, the
> > > > > >> > >> >> >> > two most difficult things with setting this project
> > up
> > > > > were understanding
> > > > > >> > >> >> >> > that the "process" method of the class that
> > implements
> > > > > "Processor" is the
> > > > > >> > >> >> >> > entry point and the DynamicConsumerRouteBuilder
> > creates
> > > > > the second entry
> > > > > >> > >> >> >> > point (The 5th and last points). This made the
> > project
> > > > > very, VERY hard to
> > > > > >> > >> >> >> > understand.
> > > > > >> > >> >> >> > In addition to simplicity of design, the mvn clean
> > > install
> > > > > of the web
> > > > > >> > >> >> >> > service project is much faster and small scale
> > activity
> > > > > publishing is also
> > > > > >> > >> >> >> > faster (see my email about load testing). These are
> > > minor
> > > > > points though as
> > > > > >> > >> >> >> > compilation has no effect on deployment. OSGi does
> > add
> > > the
> > > > > benefit of
> > > > > >> > >> >> >> > modularized programming which is valuable, though I
> > > think
> > > > > the added
> > > > > >> > >> >> >> > complexity of Camel merits moving the project away
> > from
> > > > > this paradigm.
> > > > > >> > >> >> >> >
> > > > > >> > >> >> >>
> > > > > >> > >> >> >> I agree that the project is pretty difficult to
> > > understand
> > > > > ATM.  I think
> > > > > >> > >> >> >> what we need to do is think about what the
> > > responsibilities
> > > > > of the code are
> > > > > >> > >> >> >> and allow for different implementations that are not
> so
> > > > > tightly coupled as
> > > > > >> > >> >> >> they are now.  For instance, having worked with Storm
> > to
> > > > > ingest millions of
> > > > > >> > >> >> >> activities a day, I personally would like to see
> > streams
> > > be
> > > > > responsible for
> > > > > >> > >> >> >> defining an over-arching orchestration model that can
> > be
> > > > > implemented within
> > > > > >> > >> >> >> a single war or on top of a distributed system.  This
> > > would
> > > > > look something
> > > > > >> > >> >> >> like the follows:
> > > > > >> > >> >> >>
> > > > > >> > >> >> >> |__ Streams-Core  (Base classes, interfaces,
> utilities,
> > > > > extensions, etc)
> > > > > >> > >> >> >> |
> > > > > >> > >> >> >> |__ Streams-Storm (Storm implementation of the core)
> > > > > >> > >> >> >> |
> > > > > >> > >> >> >> |__ Streams-Camel (Camel implementation of the core)
> > > > > >> > >> >> >> |
> > > > > >> > >> >> >> |__ Streams-WS (Web service implementation)
> > > > > >> > >> >> >>
> > > > > >> > >> >> >>
> > > > > >> > >> >> >> > Danny
> > > > > >> > >> >> >> >
> > > > > >> > >> >> >> >
> > > > > >> > >> >> >> > > From: dsullivan7@hotmail.com
> > > > > >> > >> >> >> > > To: dev@streams.incubator.apache.org
> > > > > >> > >> >> >> > > Subject: [DISCUSS] Switching Streams from Camel
> > > > > deployment to .war
> > > > > >> > >> >> >> > deployment
> > > > > >> > >> >> >> > > Date: Tue, 29 Oct 2013 11:07:39 -0400
> > > > > >> > >> >> >> > >
> > > > > >> > >> >> >> > > The discussion thread for switching Streams from
> > > > > Camel/osgi/Servicemix
> > > > > >> > >> >> >> > to a single .war deployment
> > > > > >> > >> >> >> >
> > > > > >> > >> >> >> >
> > > > > >> > >> >> >
> > > > > >> > >> >
> > > > > >> > >
> > > > > >>
> > > > > >
> > > > >
> > >
> > >
> >
>

Re: [DISCUSS] Switching Streams from Camel deployment to .war deployment

[Chris Geer <ch...@cxtsoftware.com>]

On Thu, Oct 31, 2013 at 2:23 PM, Matt Franklin <m....@gmail.com>wrote:

> On Thu, Oct 31, 2013 at 5:04 PM, Danny Sullivan <dsullivan7@hotmail.com
> >wrote:
>
> > I'm not quite following this, so I apologize. What I'm trying to do is
> > programmatically make a request to a jar running on a separate jvm and
> get
> > the response from that call all within the same method. Similar to this
> > http request:
> >
> > HttpGet httpget = new HttpGet();
> >
> > httpget.setURI(new URI("www.streams-persistence.com"));
> >
> > CloseableHttpResponse response = httpclient.execute(httpget);
> >
> > //do stuff with the response...
> >
> >
> >
> >
> > I imagine that this would translate to something in Camel similar to
> this:
> >
> >
> > <route>
> >
> > <from uri="bean:subscriberService?method=getAllSubscribers"/>
> >
> > <inOut
> >
> uri="addressToStreamsPersistenceJar?method=selectAllSubscribersFromDatabase"/>
> >
> > </route>
> >
>
> Camel uses AMQP as a messaging system so that you don't have to create HTTP
> requests between sources, though it is possible to do so.  In Storm, we
> usually try to use a buffer system like Kafka to do the same.
>

Camel can use AMQP through ActiveMQ.

>
>
> >
> >
> > But it is unclear what the actual implementation would be.
> >
> >
> > This actually brings me to another suggestion. Would there be a big
> > performance impact to have communication between the software components
> > occur between http? Say the 5 software components I outlined earlier were
> > packaged as 5 separate wars. These wars could communicate with each other
> > via get a post requests. This sounds unconventional offhand so I'd like
> to
> > hear some thoughts on it.
> >
>
> It is most certainly possible, but IMO probably not the best option for
> success.  Protocols like PubSubHubbub use HTTP for a message transport.
>
>
> > -Danny
> >
> > > Date: Thu, 31 Oct 2013 13:28:41 -0700
> > > Subject: Re: [DISCUSS] Switching Streams from Camel deployment to .war
> > deployment
> > > From: chris@cxtsoftware.com
> > > To: dev@streams.incubator.apache.org
> > >
> > > Or Content Enricher [2]
> > >
> > > [2] http://camel.apache.org/content-enricher.html
> > >
> > >
> > > On Thu, Oct 31, 2013 at 12:37 PM, Jason Letourneau
> > > <jl...@gmail.com>wrote:
> > >
> > > > check out the link here[1]
> > > >
> > > >
> > > > [1]http://camel.apache.org/request-reply.html
> > > >
> > > > On Thu, Oct 31, 2013 at 2:32 PM, Danny Sullivan <
> > dsullivan7@hotmail.com>
> > > > wrote:
> > > > > I have a quick Camel question that I arrived at in the
> > implementation of
> > > > these new components:
> > > > >
> > > > > Lets say I have a method in streams-activity.jar that needs all
> > > > subscribers in the database. This would require a call to the
> > > > streams-persistence.jar. So far, I've seen camel used mostly for
> > passing
> > > > data through the application, but not for making a single
> > request-reponse
> > > > from within a method. How can I use Camel to get a list of all
> > subscribers
> > > > in the streams-persistence.jar from the streams-activity.jar?
> > > > >
> > > > >> From: dsullivan7@hotmail.com
> > > > >> To: dev@streams.incubator.apache.org
> > > > >> Subject: RE: [DISCUSS] Switching Streams from Camel deployment to
> > .war
> > > > deployment
> > > > >> Date: Thu, 31 Oct 2013 08:40:44 -0400
> > > > >>
> > > > >> Excellent, I'll write up something as a proof of concept and we
> can
> > > > discuss further to make sure everything is vanilla.
> > > > >>
> > > > >> > Date: Wed, 30 Oct 2013 17:01:28 -0400
> > > > >> > Subject: Re: [DISCUSS] Switching Streams from Camel deployment
> to
> > > > .war deployment
> > > > >> > From: jletourneau80@gmail.com
> > > > >> > To: dev@streams.incubator.apache.org
> > > > >> >
> > > > >> > That sounds pretty promising to me.
> > > > >> >
> > > > >> > On Wed, Oct 30, 2013 at 2:38 PM, Danny Sullivan <
> > > > dsullivan7@hotmail.com> wrote:
> > > > >> > > Thanks for the feedback. You have an interesting point about
> the
> > > > url linking to a separate processing space. Let me tie my answer into
> > your
> > > > last question about "advocating for the simplicity at registration to
> > give
> > > > up flexibility at registration, but retaining the inner "guts" of
> > > > EIP/messaging". Consider a new architecture:
> > > > >> > >
> > > > >> > > streams-web.war: single entry point to application, but
> > functions
> > > > ONLY as an entry point. From here Camel routes the incoming requests
> > to 4
> > > > separate jarssubscriber-registration.jar: subscriber registration
> > > > publisher-registration.jar: publisher registrationactivity.jar:
> returns
> > > > activity (also contains subscriber warehouse and storm activity
> > > > aggregator)publish.jar: publishes activitystreams-cassandra.jar: the
> > above
> > > > 4 jars would all have a hook into this jar which would function as a
> > hook
> > > > onto the database. Each jar would have camel route output to this
> jar.
> > > > >> > >
> > > > >> > > In this implementation, Camel would no longer be the entry and
> > exit
> > > > point of a client to the application, but would handle the
> > communication
> > > > between components. The flow of activity through the application
> would
> > be
> > > > method based in each jar. This would allow deployment on up to 6
> > different
> > > > process spaces. However, this does not address that there is a single
> > > > server entry point, but I'm not sure if it was a concern in the first
> > place.
> > > > >> > >
> > > > >> > > My argument, at its basis, is that we should move away from
> > using
> > > > Camel as the entry point to the application. I would be happy to
> > maintain
> > > > messaging between components.
> > > > >> > >
> > > > >> > >> Date: Wed, 30 Oct 2013 12:32:55 -0400
> > > > >> > >> Subject: Re: [DISCUSS] Switching Streams from Camel
> deployment
> > to
> > > > .war deployment
> > > > >> > >> From: jletourneau80@gmail.com
> > > > >> > >> To: dev@streams.incubator.apache.org
> > > > >> > >>
> > > > >> > >> An interesting use case that I am holding onto is the ability
> > for
> > > > >> > >> publishers to register via a single URL (registration
> > endpoint),
> > > > but
> > > > >> > >> be sent a URL back to post to a different process space for
> > actual
> > > > >> > >> publishing.  The same is true on the subscriber front.
> >  Currently,
> > > > the
> > > > >> > >> Camel/EIP infrastructure abstracts this because different
> > > > components
> > > > >> > >> deployed in different process spaces handling the route
> > creation
> > > > can
> > > > >> > >> just be bolted onto a running Streams instance without new
> > > > subs/pubs
> > > > >> > >> behaving any differently than existing.  This seems to be a
> > > > >> > >> potentially critical scaling point.  Is there a way to do
> this
> > with
> > > > >> > >> the Spring solution?
> > > > >> > >>
> > > > >> > >> The persistence point is a good one, though I would classify
> > that
> > > > as
> > > > >> > >> "not implemented" vs "not possible" (not that you were).
> > > > >> > >>
> > > > >> > >> I'm not married to Camel, I just like the EIP approach to
> > building
> > > > >> > >> something that is ultimately a messaging system.  There are
> > known
> > > > >> > >> patterns that solve at least a subset of the problems Streams
> > is
> > > > >> > >> trying to solve and implementations that can handle the load
> > and
> > > > I'll
> > > > >> > >> reiterate flexibility == complexity almost always.
> > > > >> > >>
> > > > >> > >> It comes right back to the central question: Do you want
> > > > flexibility
> > > > >> > >> or simplicity?  It doesn't have to be black and white either
> I
> > > > don't
> > > > >> > >> think...
> > > > >> > >>
> > > > >> > >> More pointedly: Where should we give up flexibility for
> > > > simplicity?  I
> > > > >> > >> read that Danny is advocating for the simplicity at
> > registration to
> > > > >> > >> give up flexibility at registration, but retaining the inner
> > > > "guts" of
> > > > >> > >> EIP/messaging?  Thoughts?
> > > > >> > >>
> > > > >> > >> On Wed, Oct 30, 2013 at 11:37 AM, Danny Sullivan <
> > > > dsullivan7@hotmail.com> wrote:
> > > > >> > >> > My argument is not for the IoC pattern as that can be (and
> > has
> > > > been) implemented alongside Camel. My main argument is that the
> syntax
> > at
> > > > the entry point is not only familiar but much simpler. This wouldn't
> > be a
> > > > very strong argument if the Camel implementation wasn't much more
> > > > complicated but I feel that it is the case. Also, looking toward the
> > > > future, if the server is restarted, in-routes are lost in Camel. The
> > way to
> > > > curb this is to persist the dynamic routes that Camel creates, and
> > then on
> > > > start up pull every one of these routes and recreate a dynamic route
> > for
> > > > each one. Not only is this much easier to implement using the Spring
> > web
> > > > implementation, but it already has been implemented and you can try
> it
> > by
> > > > checking out the webservice branch, registering a subscriber,
> > restarting
> > > > tomcat, and using the same url you had before. This will allow
> > subscribers
> > > > to hang on to their urls once they register. (the same is true for
> > > > publishers: you can post via the same url after restarting tomcat)
> > > > >> > >> >
> > > > >> > >> >> Date: Wed, 30 Oct 2013 11:00:21 -0400
> > > > >> > >> >> Subject: Re: [DISCUSS] Switching Streams from Camel
> > deployment
> > > > to .war deployment
> > > > >> > >> >> From: jletourneau80@gmail.com
> > > > >> > >> >> To: dev@streams.incubator.apache.org
> > > > >> > >> >>
> > > > >> > >> >> To be fair, while the current implementation is heavily
> > > > camel-based,
> > > > >> > >> >> all of the interfaces related to Streams functionality are
> > not.
> > > >  The
> > > > >> > >> >> current model maps to what Matt has outlined in my
> opinion,
> > > > though
> > > > >> > >> >> packing names etc. probably don't follow that exact
> pattern.
> > > > >> > >> >>
> > > > >> > >> >> With regards to the complexity and different components in
> > the
> > > > >> > >> >> registration process, this was a cut at the abstraction
> > based
> > > > on the
> > > > >> > >> >> assumption that different implementations may be plugged
> in
> > and
> > > > in
> > > > >> > >> >> fact may live on different processor space (ie. a polling
> > > > publisher vs
> > > > >> > >> >> a push publisher may be instantiated on different servers
> > but
> > > > the
> > > > >> > >> >> registration URL is staticly defined).
> > > > >> > >> >>
> > > > >> > >> >> Is the main argument I am seeing for Spring  the
> > familiarity of
> > > > its
> > > > >> > >> >> IoC pattern implementation and syntax at the entry point?
> > > > >> > >> >>
> > > > >> > >> >> On Wed, Oct 30, 2013 at 10:53 AM, Danny Sullivan <
> > > > dsullivan7@hotmail.com> wrote:
> > > > >> > >> >> > Could you clarify whether the same entry points would
> > exist
> > > > for the camel implementation of the core (implementing the "process"
> > > > method/ using a DynammicRouteBuilder) or would the webservice be the
> > sole
> > > > entry point to Streams and after it enters would it hand it off to
> > Camel?
> > > > And what would be the entry point for the Storm implementation?
> > > > >> > >> >> > -Danny
> > > > >> > >> >> >
> > > > >> > >> >> >> Date: Wed, 30 Oct 2013 10:13:04 -0400
> > > > >> > >> >> >> Subject: Re: [DISCUSS] Switching Streams from Camel
> > > > deployment to .war deployment
> > > > >> > >> >> >> From: m.ben.franklin@gmail.com
> > > > >> > >> >> >> To: dev@streams.incubator.apache.org
> > > > >> > >> >> >>
> > > > >> > >> >> >> On Tue, Oct 29, 2013 at 2:55 PM, Danny Sullivan <
> > > > dsullivan7@hotmail.com>wrote:
> > > > >> > >> >> >>
> > > > >> > >> >> >> > My case for switching from OSGi is for simplicity in
> > > > design. To follow the
> > > > >> > >> >> >> > path of an activity through Streams in the
> webservice,
> > > > there is one main
> > > > >> > >> >> >> > things the developer needs to understand:
> > > > >> > >> >> >> > The @RequestMapping annotation specifies the HTTP
> entry
> > > > point
> > > > >> > >> >> >> > There are 4 @RequestMapping annotations that
> > correspond to
> > > > each of the 4
> > > > >> > >> >> >> > ways a user enters the application: registering a
> > > > publisher, registering a
> > > > >> > >> >> >> > subscriber, publishing activity, and getting an
> > activity
> > > > stream. Where
> > > > >> > >> >> >> > these are located in the source code can be found by
> > > > searching for the
> > > > >> > >> >> >> > paths specified in the documentation (search for
> > > > "/publisherRegister",
> > > > >> > >> >> >> > "/subscriberRegister", "/publishActivity",
> > "/getActivity"
> > > > which will all
> > > > >> > >> >> >> > lead you to StreamsWebController.java). From the
> > methods
> > > > that process
> > > > >> > >> >> >> > requests, the flow through the application is through
> > > > methods which can be
> > > > >> > >> >> >> > understood by most Java programmers.
> > > > >> > >> >> >> > The flow of activities through the current trunk
> > branch is
> > > > understood as
> > > > >> > >> >> >> > follows:
> > > > >> > >> >> >> > The string "/publisher/register" (the entry point to
> > > > register a publisher
> > > > >> > >> >> >> > specified in the documentation) is the value of the
> > > > >> > >> >> >> > consumer.registrationEndpoint property defined in
> > > > streams.propertiesThe
> > > > >> > >> >> >> > camelContext.xml specifies an endpoint with the id
> > > > >> > >> >> >> > consumerRegistrationEndpoint to have a uri equal to
> the
> > > > propertyThe
> > > > >> > >> >> >> > consumerRegistrationEndpoint routes from uri
> > > > direct:publisher register with
> > > > >> > >> >> >> > the bean activityRegistrationProcessor nested in
> > between
> > > > the routeThe
> > > > >> > >> >> >> > streams-eip-applicationContext contains a bean with
> > the id
> > > > >> > >> >> >> > activityRegistrationProcessor created for the class
> > > > >> > >> >> >> > ActivityPublisherRegistrationProcessorThe exchange
> will
> > > > enter the "process"
> > > > >> > >> >> >> > method of the class
> > > > ActivityPublisherRegisitrationProcessor and that this
> > > > >> > >> >> >> > is because this class implements the "Processor"
> > interface
> > > > provided by
> > > > >> > >> >> >> > CamelThe direct:add-publisher-route takes the
> exchange
> > > > output from the
> > > > >> > >> >> >> > "process" method and routes it to the
> > > > activityRegistrationProcessor
> > > > >> > >> >> >> > "register" method. The bean
> > activityRegistrationProcessor
> > > > is defined in the
> > > > >> > >> >> >> > streams-eip-osgi-component-import.xmlThe output from
> > this
> > > > method is then
> > > > >> > >> >> >> > sent to the "createNewRouteForConsumer" method of
> > > > activityConsumerRouter.
> > > > >> > >> >> >> > This method creates a new route for the newly
> > registered
> > > > publisher using
> > > > >> > >> >> >> > the private static final class
> > DynamicConsumerRouteBuilder
> > > > which is
> > > > >> > >> >> >> > required to extend RouteBuilder which is provided by
> > > > Camel. This
> > > > >> > >> >> >> > DynamicConsumerRouteBuilder contains several methods:
> > > > >> > >> >> >> > getConsumerReceiveMethod() (which corresponds to
> @Value
> > > > >> > >> >> >> > ${consumer.receiveMethod} which corresponds to
> > "receive"),
> > > > >> > >> >> >> > getConsumerSplitMethod() (@Value
> > > > ${consumer.splitMethod},"split"), and
> > > > >> > >> >> >> > getConsumerActivityQUri() (@Value
> > ${consumerActivityQUri},
> > > > >> > >> >> >> > "direct:activityQ"). This is different than the
> > > > camelContext.xml in that
> > > > >> > >> >> >> > the route is being created programmatically. What
> this
> > is
> > > > doing is routing
> > > > >> > >> >> >> > input from the inroute url (which Camel does
> > automatically
> > > > through the
> > > > >> > >> >> >> > configure method which is required to be overridden),
> > to
> > > > the "receive"
> > > > >> > >> >> >> > method of ActivityConsumer, then to the "split"
> method
> > of
> > > > ActivityConsumer,
> > > > >> > >> >> >> > and then to the "direct:activityQ" which if you look
> > back
> > > > in the
> > > > >> > >> >> >> > camelContext.xml routes to the
> > "activemq:queue:activities"
> > > > which then
> > > > >> > >> >> >> > routes to "receiveExchange"
> > > > >> > >> >> >> > This is the process to register a publisher. The
> > process
> > > > for registering a
> > > > >> > >> >> >> > subscriber is relatively the same though it involves
> > > > separate classes with
> > > > >> > >> >> >> > their own private static final RouteBuilder class.
> > From my
> > > > perspective, the
> > > > >> > >> >> >> > two most difficult things with setting this project
> up
> > > > were understanding
> > > > >> > >> >> >> > that the "process" method of the class that
> implements
> > > > "Processor" is the
> > > > >> > >> >> >> > entry point and the DynamicConsumerRouteBuilder
> creates
> > > > the second entry
> > > > >> > >> >> >> > point (The 5th and last points). This made the
> project
> > > > very, VERY hard to
> > > > >> > >> >> >> > understand.
> > > > >> > >> >> >> > In addition to simplicity of design, the mvn clean
> > install
> > > > of the web
> > > > >> > >> >> >> > service project is much faster and small scale
> activity
> > > > publishing is also
> > > > >> > >> >> >> > faster (see my email about load testing). These are
> > minor
> > > > points though as
> > > > >> > >> >> >> > compilation has no effect on deployment. OSGi does
> add
> > the
> > > > benefit of
> > > > >> > >> >> >> > modularized programming which is valuable, though I
> > think
> > > > the added
> > > > >> > >> >> >> > complexity of Camel merits moving the project away
> from
> > > > this paradigm.
> > > > >> > >> >> >> >
> > > > >> > >> >> >>
> > > > >> > >> >> >> I agree that the project is pretty difficult to
> > understand
> > > > ATM.  I think
> > > > >> > >> >> >> what we need to do is think about what the
> > responsibilities
> > > > of the code are
> > > > >> > >> >> >> and allow for different implementations that are not so
> > > > tightly coupled as
> > > > >> > >> >> >> they are now.  For instance, having worked with Storm
> to
> > > > ingest millions of
> > > > >> > >> >> >> activities a day, I personally would like to see
> streams
> > be
> > > > responsible for
> > > > >> > >> >> >> defining an over-arching orchestration model that can
> be
> > > > implemented within
> > > > >> > >> >> >> a single war or on top of a distributed system.  This
> > would
> > > > look something
> > > > >> > >> >> >> like the follows:
> > > > >> > >> >> >>
> > > > >> > >> >> >> |__ Streams-Core  (Base classes, interfaces, utilities,
> > > > extensions, etc)
> > > > >> > >> >> >> |
> > > > >> > >> >> >> |__ Streams-Storm (Storm implementation of the core)
> > > > >> > >> >> >> |
> > > > >> > >> >> >> |__ Streams-Camel (Camel implementation of the core)
> > > > >> > >> >> >> |
> > > > >> > >> >> >> |__ Streams-WS (Web service implementation)
> > > > >> > >> >> >>
> > > > >> > >> >> >>
> > > > >> > >> >> >> > Danny
> > > > >> > >> >> >> >
> > > > >> > >> >> >> >
> > > > >> > >> >> >> > > From: dsullivan7@hotmail.com
> > > > >> > >> >> >> > > To: dev@streams.incubator.apache.org
> > > > >> > >> >> >> > > Subject: [DISCUSS] Switching Streams from Camel
> > > > deployment to .war
> > > > >> > >> >> >> > deployment
> > > > >> > >> >> >> > > Date: Tue, 29 Oct 2013 11:07:39 -0400
> > > > >> > >> >> >> > >
> > > > >> > >> >> >> > > The discussion thread for switching Streams from
> > > > Camel/osgi/Servicemix
> > > > >> > >> >> >> > to a single .war deployment
> > > > >> > >> >> >> >
> > > > >> > >> >> >> >
> > > > >> > >> >> >
> > > > >> > >> >
> > > > >> > >
> > > > >>
> > > > >
> > > >
> >
> >
>

Re: [DISCUSS] Switching Streams from Camel deployment to .war deployment

[Matt Franklin <m....@gmail.com>]

On Thu, Oct 31, 2013 at 5:04 PM, Danny Sullivan <ds...@hotmail.com>wrote:

> I'm not quite following this, so I apologize. What I'm trying to do is
> programmatically make a request to a jar running on a separate jvm and get
> the response from that call all within the same method. Similar to this
> http request:
>
> HttpGet httpget = new HttpGet();
>
> httpget.setURI(new URI("www.streams-persistence.com"));
>
> CloseableHttpResponse response = httpclient.execute(httpget);
>
> //do stuff with the response...
>
>
>
>
> I imagine that this would translate to something in Camel similar to this:
>
>
> <route>
>
> <from uri="bean:subscriberService?method=getAllSubscribers"/>
>
> <inOut
> uri="addressToStreamsPersistenceJar?method=selectAllSubscribersFromDatabase"/>
>
> </route>
>

Camel uses AMQP as a messaging system so that you don't have to create HTTP
requests between sources, though it is possible to do so.  In Storm, we
usually try to use a buffer system like Kafka to do the same.


>
>
> But it is unclear what the actual implementation would be.
>
>
> This actually brings me to another suggestion. Would there be a big
> performance impact to have communication between the software components
> occur between http? Say the 5 software components I outlined earlier were
> packaged as 5 separate wars. These wars could communicate with each other
> via get a post requests. This sounds unconventional offhand so I'd like to
> hear some thoughts on it.
>

It is most certainly possible, but IMO probably not the best option for
success.  Protocols like PubSubHubbub use HTTP for a message transport.


> -Danny
>
> > Date: Thu, 31 Oct 2013 13:28:41 -0700
> > Subject: Re: [DISCUSS] Switching Streams from Camel deployment to .war
> deployment
> > From: chris@cxtsoftware.com
> > To: dev@streams.incubator.apache.org
> >
> > Or Content Enricher [2]
> >
> > [2] http://camel.apache.org/content-enricher.html
> >
> >
> > On Thu, Oct 31, 2013 at 12:37 PM, Jason Letourneau
> > <jl...@gmail.com>wrote:
> >
> > > check out the link here[1]
> > >
> > >
> > > [1]http://camel.apache.org/request-reply.html
> > >
> > > On Thu, Oct 31, 2013 at 2:32 PM, Danny Sullivan <
> dsullivan7@hotmail.com>
> > > wrote:
> > > > I have a quick Camel question that I arrived at in the
> implementation of
> > > these new components:
> > > >
> > > > Lets say I have a method in streams-activity.jar that needs all
> > > subscribers in the database. This would require a call to the
> > > streams-persistence.jar. So far, I've seen camel used mostly for
> passing
> > > data through the application, but not for making a single
> request-reponse
> > > from within a method. How can I use Camel to get a list of all
> subscribers
> > > in the streams-persistence.jar from the streams-activity.jar?
> > > >
> > > >> From: dsullivan7@hotmail.com
> > > >> To: dev@streams.incubator.apache.org
> > > >> Subject: RE: [DISCUSS] Switching Streams from Camel deployment to
> .war
> > > deployment
> > > >> Date: Thu, 31 Oct 2013 08:40:44 -0400
> > > >>
> > > >> Excellent, I'll write up something as a proof of concept and we can
> > > discuss further to make sure everything is vanilla.
> > > >>
> > > >> > Date: Wed, 30 Oct 2013 17:01:28 -0400
> > > >> > Subject: Re: [DISCUSS] Switching Streams from Camel deployment to
> > > .war deployment
> > > >> > From: jletourneau80@gmail.com
> > > >> > To: dev@streams.incubator.apache.org
> > > >> >
> > > >> > That sounds pretty promising to me.
> > > >> >
> > > >> > On Wed, Oct 30, 2013 at 2:38 PM, Danny Sullivan <
> > > dsullivan7@hotmail.com> wrote:
> > > >> > > Thanks for the feedback. You have an interesting point about the
> > > url linking to a separate processing space. Let me tie my answer into
> your
> > > last question about "advocating for the simplicity at registration to
> give
> > > up flexibility at registration, but retaining the inner "guts" of
> > > EIP/messaging". Consider a new architecture:
> > > >> > >
> > > >> > > streams-web.war: single entry point to application, but
> functions
> > > ONLY as an entry point. From here Camel routes the incoming requests
> to 4
> > > separate jarssubscriber-registration.jar: subscriber registration
> > > publisher-registration.jar: publisher registrationactivity.jar: returns
> > > activity (also contains subscriber warehouse and storm activity
> > > aggregator)publish.jar: publishes activitystreams-cassandra.jar: the
> above
> > > 4 jars would all have a hook into this jar which would function as a
> hook
> > > onto the database. Each jar would have camel route output to this jar.
> > > >> > >
> > > >> > > In this implementation, Camel would no longer be the entry and
> exit
> > > point of a client to the application, but would handle the
> communication
> > > between components. The flow of activity through the application would
> be
> > > method based in each jar. This would allow deployment on up to 6
> different
> > > process spaces. However, this does not address that there is a single
> > > server entry point, but I'm not sure if it was a concern in the first
> place.
> > > >> > >
> > > >> > > My argument, at its basis, is that we should move away from
> using
> > > Camel as the entry point to the application. I would be happy to
> maintain
> > > messaging between components.
> > > >> > >
> > > >> > >> Date: Wed, 30 Oct 2013 12:32:55 -0400
> > > >> > >> Subject: Re: [DISCUSS] Switching Streams from Camel deployment
> to
> > > .war deployment
> > > >> > >> From: jletourneau80@gmail.com
> > > >> > >> To: dev@streams.incubator.apache.org
> > > >> > >>
> > > >> > >> An interesting use case that I am holding onto is the ability
> for
> > > >> > >> publishers to register via a single URL (registration
> endpoint),
> > > but
> > > >> > >> be sent a URL back to post to a different process space for
> actual
> > > >> > >> publishing.  The same is true on the subscriber front.
>  Currently,
> > > the
> > > >> > >> Camel/EIP infrastructure abstracts this because different
> > > components
> > > >> > >> deployed in different process spaces handling the route
> creation
> > > can
> > > >> > >> just be bolted onto a running Streams instance without new
> > > subs/pubs
> > > >> > >> behaving any differently than existing.  This seems to be a
> > > >> > >> potentially critical scaling point.  Is there a way to do this
> with
> > > >> > >> the Spring solution?
> > > >> > >>
> > > >> > >> The persistence point is a good one, though I would classify
> that
> > > as
> > > >> > >> "not implemented" vs "not possible" (not that you were).
> > > >> > >>
> > > >> > >> I'm not married to Camel, I just like the EIP approach to
> building
> > > >> > >> something that is ultimately a messaging system.  There are
> known
> > > >> > >> patterns that solve at least a subset of the problems Streams
> is
> > > >> > >> trying to solve and implementations that can handle the load
> and
> > > I'll
> > > >> > >> reiterate flexibility == complexity almost always.
> > > >> > >>
> > > >> > >> It comes right back to the central question: Do you want
> > > flexibility
> > > >> > >> or simplicity?  It doesn't have to be black and white either I
> > > don't
> > > >> > >> think...
> > > >> > >>
> > > >> > >> More pointedly: Where should we give up flexibility for
> > > simplicity?  I
> > > >> > >> read that Danny is advocating for the simplicity at
> registration to
> > > >> > >> give up flexibility at registration, but retaining the inner
> > > "guts" of
> > > >> > >> EIP/messaging?  Thoughts?
> > > >> > >>
> > > >> > >> On Wed, Oct 30, 2013 at 11:37 AM, Danny Sullivan <
> > > dsullivan7@hotmail.com> wrote:
> > > >> > >> > My argument is not for the IoC pattern as that can be (and
> has
> > > been) implemented alongside Camel. My main argument is that the syntax
> at
> > > the entry point is not only familiar but much simpler. This wouldn't
> be a
> > > very strong argument if the Camel implementation wasn't much more
> > > complicated but I feel that it is the case. Also, looking toward the
> > > future, if the server is restarted, in-routes are lost in Camel. The
> way to
> > > curb this is to persist the dynamic routes that Camel creates, and
> then on
> > > start up pull every one of these routes and recreate a dynamic route
> for
> > > each one. Not only is this much easier to implement using the Spring
> web
> > > implementation, but it already has been implemented and you can try it
> by
> > > checking out the webservice branch, registering a subscriber,
> restarting
> > > tomcat, and using the same url you had before. This will allow
> subscribers
> > > to hang on to their urls once they register. (the same is true for
> > > publishers: you can post via the same url after restarting tomcat)
> > > >> > >> >
> > > >> > >> >> Date: Wed, 30 Oct 2013 11:00:21 -0400
> > > >> > >> >> Subject: Re: [DISCUSS] Switching Streams from Camel
> deployment
> > > to .war deployment
> > > >> > >> >> From: jletourneau80@gmail.com
> > > >> > >> >> To: dev@streams.incubator.apache.org
> > > >> > >> >>
> > > >> > >> >> To be fair, while the current implementation is heavily
> > > camel-based,
> > > >> > >> >> all of the interfaces related to Streams functionality are
> not.
> > >  The
> > > >> > >> >> current model maps to what Matt has outlined in my opinion,
> > > though
> > > >> > >> >> packing names etc. probably don't follow that exact pattern.
> > > >> > >> >>
> > > >> > >> >> With regards to the complexity and different components in
> the
> > > >> > >> >> registration process, this was a cut at the abstraction
> based
> > > on the
> > > >> > >> >> assumption that different implementations may be plugged in
> and
> > > in
> > > >> > >> >> fact may live on different processor space (ie. a polling
> > > publisher vs
> > > >> > >> >> a push publisher may be instantiated on different servers
> but
> > > the
> > > >> > >> >> registration URL is staticly defined).
> > > >> > >> >>
> > > >> > >> >> Is the main argument I am seeing for Spring  the
> familiarity of
> > > its
> > > >> > >> >> IoC pattern implementation and syntax at the entry point?
> > > >> > >> >>
> > > >> > >> >> On Wed, Oct 30, 2013 at 10:53 AM, Danny Sullivan <
> > > dsullivan7@hotmail.com> wrote:
> > > >> > >> >> > Could you clarify whether the same entry points would
> exist
> > > for the camel implementation of the core (implementing the "process"
> > > method/ using a DynammicRouteBuilder) or would the webservice be the
> sole
> > > entry point to Streams and after it enters would it hand it off to
> Camel?
> > > And what would be the entry point for the Storm implementation?
> > > >> > >> >> > -Danny
> > > >> > >> >> >
> > > >> > >> >> >> Date: Wed, 30 Oct 2013 10:13:04 -0400
> > > >> > >> >> >> Subject: Re: [DISCUSS] Switching Streams from Camel
> > > deployment to .war deployment
> > > >> > >> >> >> From: m.ben.franklin@gmail.com
> > > >> > >> >> >> To: dev@streams.incubator.apache.org
> > > >> > >> >> >>
> > > >> > >> >> >> On Tue, Oct 29, 2013 at 2:55 PM, Danny Sullivan <
> > > dsullivan7@hotmail.com>wrote:
> > > >> > >> >> >>
> > > >> > >> >> >> > My case for switching from OSGi is for simplicity in
> > > design. To follow the
> > > >> > >> >> >> > path of an activity through Streams in the webservice,
> > > there is one main
> > > >> > >> >> >> > things the developer needs to understand:
> > > >> > >> >> >> > The @RequestMapping annotation specifies the HTTP entry
> > > point
> > > >> > >> >> >> > There are 4 @RequestMapping annotations that
> correspond to
> > > each of the 4
> > > >> > >> >> >> > ways a user enters the application: registering a
> > > publisher, registering a
> > > >> > >> >> >> > subscriber, publishing activity, and getting an
> activity
> > > stream. Where
> > > >> > >> >> >> > these are located in the source code can be found by
> > > searching for the
> > > >> > >> >> >> > paths specified in the documentation (search for
> > > "/publisherRegister",
> > > >> > >> >> >> > "/subscriberRegister", "/publishActivity",
> "/getActivity"
> > > which will all
> > > >> > >> >> >> > lead you to StreamsWebController.java). From the
> methods
> > > that process
> > > >> > >> >> >> > requests, the flow through the application is through
> > > methods which can be
> > > >> > >> >> >> > understood by most Java programmers.
> > > >> > >> >> >> > The flow of activities through the current trunk
> branch is
> > > understood as
> > > >> > >> >> >> > follows:
> > > >> > >> >> >> > The string "/publisher/register" (the entry point to
> > > register a publisher
> > > >> > >> >> >> > specified in the documentation) is the value of the
> > > >> > >> >> >> > consumer.registrationEndpoint property defined in
> > > streams.propertiesThe
> > > >> > >> >> >> > camelContext.xml specifies an endpoint with the id
> > > >> > >> >> >> > consumerRegistrationEndpoint to have a uri equal to the
> > > propertyThe
> > > >> > >> >> >> > consumerRegistrationEndpoint routes from uri
> > > direct:publisher register with
> > > >> > >> >> >> > the bean activityRegistrationProcessor nested in
> between
> > > the routeThe
> > > >> > >> >> >> > streams-eip-applicationContext contains a bean with
> the id
> > > >> > >> >> >> > activityRegistrationProcessor created for the class
> > > >> > >> >> >> > ActivityPublisherRegistrationProcessorThe exchange will
> > > enter the "process"
> > > >> > >> >> >> > method of the class
> > > ActivityPublisherRegisitrationProcessor and that this
> > > >> > >> >> >> > is because this class implements the "Processor"
> interface
> > > provided by
> > > >> > >> >> >> > CamelThe direct:add-publisher-route takes the exchange
> > > output from the
> > > >> > >> >> >> > "process" method and routes it to the
> > > activityRegistrationProcessor
> > > >> > >> >> >> > "register" method. The bean
> activityRegistrationProcessor
> > > is defined in the
> > > >> > >> >> >> > streams-eip-osgi-component-import.xmlThe output from
> this
> > > method is then
> > > >> > >> >> >> > sent to the "createNewRouteForConsumer" method of
> > > activityConsumerRouter.
> > > >> > >> >> >> > This method creates a new route for the newly
> registered
> > > publisher using
> > > >> > >> >> >> > the private static final class
> DynamicConsumerRouteBuilder
> > > which is
> > > >> > >> >> >> > required to extend RouteBuilder which is provided by
> > > Camel. This
> > > >> > >> >> >> > DynamicConsumerRouteBuilder contains several methods:
> > > >> > >> >> >> > getConsumerReceiveMethod() (which corresponds to @Value
> > > >> > >> >> >> > ${consumer.receiveMethod} which corresponds to
> "receive"),
> > > >> > >> >> >> > getConsumerSplitMethod() (@Value
> > > ${consumer.splitMethod},"split"), and
> > > >> > >> >> >> > getConsumerActivityQUri() (@Value
> ${consumerActivityQUri},
> > > >> > >> >> >> > "direct:activityQ"). This is different than the
> > > camelContext.xml in that
> > > >> > >> >> >> > the route is being created programmatically. What this
> is
> > > doing is routing
> > > >> > >> >> >> > input from the inroute url (which Camel does
> automatically
> > > through the
> > > >> > >> >> >> > configure method which is required to be overridden),
> to
> > > the "receive"
> > > >> > >> >> >> > method of ActivityConsumer, then to the "split" method
> of
> > > ActivityConsumer,
> > > >> > >> >> >> > and then to the "direct:activityQ" which if you look
> back
> > > in the
> > > >> > >> >> >> > camelContext.xml routes to the
> "activemq:queue:activities"
> > > which then
> > > >> > >> >> >> > routes to "receiveExchange"
> > > >> > >> >> >> > This is the process to register a publisher. The
> process
> > > for registering a
> > > >> > >> >> >> > subscriber is relatively the same though it involves
> > > separate classes with
> > > >> > >> >> >> > their own private static final RouteBuilder class.
> From my
> > > perspective, the
> > > >> > >> >> >> > two most difficult things with setting this project up
> > > were understanding
> > > >> > >> >> >> > that the "process" method of the class that implements
> > > "Processor" is the
> > > >> > >> >> >> > entry point and the DynamicConsumerRouteBuilder creates
> > > the second entry
> > > >> > >> >> >> > point (The 5th and last points). This made the project
> > > very, VERY hard to
> > > >> > >> >> >> > understand.
> > > >> > >> >> >> > In addition to simplicity of design, the mvn clean
> install
> > > of the web
> > > >> > >> >> >> > service project is much faster and small scale activity
> > > publishing is also
> > > >> > >> >> >> > faster (see my email about load testing). These are
> minor
> > > points though as
> > > >> > >> >> >> > compilation has no effect on deployment. OSGi does add
> the
> > > benefit of
> > > >> > >> >> >> > modularized programming which is valuable, though I
> think
> > > the added
> > > >> > >> >> >> > complexity of Camel merits moving the project away from
> > > this paradigm.
> > > >> > >> >> >> >
> > > >> > >> >> >>
> > > >> > >> >> >> I agree that the project is pretty difficult to
> understand
> > > ATM.  I think
> > > >> > >> >> >> what we need to do is think about what the
> responsibilities
> > > of the code are
> > > >> > >> >> >> and allow for different implementations that are not so
> > > tightly coupled as
> > > >> > >> >> >> they are now.  For instance, having worked with Storm to
> > > ingest millions of
> > > >> > >> >> >> activities a day, I personally would like to see streams
> be
> > > responsible for
> > > >> > >> >> >> defining an over-arching orchestration model that can be
> > > implemented within
> > > >> > >> >> >> a single war or on top of a distributed system.  This
> would
> > > look something
> > > >> > >> >> >> like the follows:
> > > >> > >> >> >>
> > > >> > >> >> >> |__ Streams-Core  (Base classes, interfaces, utilities,
> > > extensions, etc)
> > > >> > >> >> >> |
> > > >> > >> >> >> |__ Streams-Storm (Storm implementation of the core)
> > > >> > >> >> >> |
> > > >> > >> >> >> |__ Streams-Camel (Camel implementation of the core)
> > > >> > >> >> >> |
> > > >> > >> >> >> |__ Streams-WS (Web service implementation)
> > > >> > >> >> >>
> > > >> > >> >> >>
> > > >> > >> >> >> > Danny
> > > >> > >> >> >> >
> > > >> > >> >> >> >
> > > >> > >> >> >> > > From: dsullivan7@hotmail.com
> > > >> > >> >> >> > > To: dev@streams.incubator.apache.org
> > > >> > >> >> >> > > Subject: [DISCUSS] Switching Streams from Camel
> > > deployment to .war
> > > >> > >> >> >> > deployment
> > > >> > >> >> >> > > Date: Tue, 29 Oct 2013 11:07:39 -0400
> > > >> > >> >> >> > >
> > > >> > >> >> >> > > The discussion thread for switching Streams from
> > > Camel/osgi/Servicemix
> > > >> > >> >> >> > to a single .war deployment
> > > >> > >> >> >> >
> > > >> > >> >> >> >
> > > >> > >> >> >
> > > >> > >> >
> > > >> > >
> > > >>
> > > >
> > >
>
>

RE: [DISCUSS] Switching Streams from Camel deployment to .war deployment

[Danny Sullivan <ds...@hotmail.com>]

I'm not quite following this, so I apologize. What I'm trying to do is programmatically make a request to a jar running on a separate jvm and get the response from that call all within the same method. Similar to this http request:

HttpGet httpget = new HttpGet();

httpget.setURI(new URI("www.streams-persistence.com"));

CloseableHttpResponse response = httpclient.execute(httpget);

//do stuff with the response...




I imagine that this would translate to something in Camel similar to this:


<route>

<from uri="bean:subscriberService?method=getAllSubscribers"/>

<inOut uri="addressToStreamsPersistenceJar?method=selectAllSubscribersFromDatabase"/>

</route>


But it is unclear what the actual implementation would be.


This actually brings me to another suggestion. Would there be a big performance impact to have communication between the software components occur between http? Say the 5 software components I outlined earlier were packaged as 5 separate wars. These wars could communicate with each other via get a post requests. This sounds unconventional offhand so I'd like to hear some thoughts on it.
-Danny 

> Date: Thu, 31 Oct 2013 13:28:41 -0700
> Subject: Re: [DISCUSS] Switching Streams from Camel deployment to .war deployment
> From: chris@cxtsoftware.com
> To: dev@streams.incubator.apache.org
> 
> Or Content Enricher [2]
> 
> [2] http://camel.apache.org/content-enricher.html
> 
> 
> On Thu, Oct 31, 2013 at 12:37 PM, Jason Letourneau
> <jl...@gmail.com>wrote:
> 
> > check out the link here[1]
> >
> >
> > [1]http://camel.apache.org/request-reply.html
> >
> > On Thu, Oct 31, 2013 at 2:32 PM, Danny Sullivan <ds...@hotmail.com>
> > wrote:
> > > I have a quick Camel question that I arrived at in the implementation of
> > these new components:
> > >
> > > Lets say I have a method in streams-activity.jar that needs all
> > subscribers in the database. This would require a call to the
> > streams-persistence.jar. So far, I've seen camel used mostly for passing
> > data through the application, but not for making a single request-reponse
> > from within a method. How can I use Camel to get a list of all subscribers
> > in the streams-persistence.jar from the streams-activity.jar?
> > >
> > >> From: dsullivan7@hotmail.com
> > >> To: dev@streams.incubator.apache.org
> > >> Subject: RE: [DISCUSS] Switching Streams from Camel deployment to .war
> > deployment
> > >> Date: Thu, 31 Oct 2013 08:40:44 -0400
> > >>
> > >> Excellent, I'll write up something as a proof of concept and we can
> > discuss further to make sure everything is vanilla.
> > >>
> > >> > Date: Wed, 30 Oct 2013 17:01:28 -0400
> > >> > Subject: Re: [DISCUSS] Switching Streams from Camel deployment to
> > .war deployment
> > >> > From: jletourneau80@gmail.com
> > >> > To: dev@streams.incubator.apache.org
> > >> >
> > >> > That sounds pretty promising to me.
> > >> >
> > >> > On Wed, Oct 30, 2013 at 2:38 PM, Danny Sullivan <
> > dsullivan7@hotmail.com> wrote:
> > >> > > Thanks for the feedback. You have an interesting point about the
> > url linking to a separate processing space. Let me tie my answer into your
> > last question about "advocating for the simplicity at registration to give
> > up flexibility at registration, but retaining the inner "guts" of
> > EIP/messaging". Consider a new architecture:
> > >> > >
> > >> > > streams-web.war: single entry point to application, but functions
> > ONLY as an entry point. From here Camel routes the incoming requests to 4
> > separate jarssubscriber-registration.jar: subscriber registration
> > publisher-registration.jar: publisher registrationactivity.jar: returns
> > activity (also contains subscriber warehouse and storm activity
> > aggregator)publish.jar: publishes activitystreams-cassandra.jar: the above
> > 4 jars would all have a hook into this jar which would function as a hook
> > onto the database. Each jar would have camel route output to this jar.
> > >> > >
> > >> > > In this implementation, Camel would no longer be the entry and exit
> > point of a client to the application, but would handle the communication
> > between components. The flow of activity through the application would be
> > method based in each jar. This would allow deployment on up to 6 different
> > process spaces. However, this does not address that there is a single
> > server entry point, but I'm not sure if it was a concern in the first place.
> > >> > >
> > >> > > My argument, at its basis, is that we should move away from using
> > Camel as the entry point to the application. I would be happy to maintain
> > messaging between components.
> > >> > >
> > >> > >> Date: Wed, 30 Oct 2013 12:32:55 -0400
> > >> > >> Subject: Re: [DISCUSS] Switching Streams from Camel deployment to
> > .war deployment
> > >> > >> From: jletourneau80@gmail.com
> > >> > >> To: dev@streams.incubator.apache.org
> > >> > >>
> > >> > >> An interesting use case that I am holding onto is the ability for
> > >> > >> publishers to register via a single URL (registration endpoint),
> > but
> > >> > >> be sent a URL back to post to a different process space for actual
> > >> > >> publishing.  The same is true on the subscriber front.  Currently,
> > the
> > >> > >> Camel/EIP infrastructure abstracts this because different
> > components
> > >> > >> deployed in different process spaces handling the route creation
> > can
> > >> > >> just be bolted onto a running Streams instance without new
> > subs/pubs
> > >> > >> behaving any differently than existing.  This seems to be a
> > >> > >> potentially critical scaling point.  Is there a way to do this with
> > >> > >> the Spring solution?
> > >> > >>
> > >> > >> The persistence point is a good one, though I would classify that
> > as
> > >> > >> "not implemented" vs "not possible" (not that you were).
> > >> > >>
> > >> > >> I'm not married to Camel, I just like the EIP approach to building
> > >> > >> something that is ultimately a messaging system.  There are known
> > >> > >> patterns that solve at least a subset of the problems Streams is
> > >> > >> trying to solve and implementations that can handle the load and
> > I'll
> > >> > >> reiterate flexibility == complexity almost always.
> > >> > >>
> > >> > >> It comes right back to the central question: Do you want
> > flexibility
> > >> > >> or simplicity?  It doesn't have to be black and white either I
> > don't
> > >> > >> think...
> > >> > >>
> > >> > >> More pointedly: Where should we give up flexibility for
> > simplicity?  I
> > >> > >> read that Danny is advocating for the simplicity at registration to
> > >> > >> give up flexibility at registration, but retaining the inner
> > "guts" of
> > >> > >> EIP/messaging?  Thoughts?
> > >> > >>
> > >> > >> On Wed, Oct 30, 2013 at 11:37 AM, Danny Sullivan <
> > dsullivan7@hotmail.com> wrote:
> > >> > >> > My argument is not for the IoC pattern as that can be (and has
> > been) implemented alongside Camel. My main argument is that the syntax at
> > the entry point is not only familiar but much simpler. This wouldn't be a
> > very strong argument if the Camel implementation wasn't much more
> > complicated but I feel that it is the case. Also, looking toward the
> > future, if the server is restarted, in-routes are lost in Camel. The way to
> > curb this is to persist the dynamic routes that Camel creates, and then on
> > start up pull every one of these routes and recreate a dynamic route for
> > each one. Not only is this much easier to implement using the Spring web
> > implementation, but it already has been implemented and you can try it by
> > checking out the webservice branch, registering a subscriber, restarting
> > tomcat, and using the same url you had before. This will allow subscribers
> > to hang on to their urls once they register. (the same is true for
> > publishers: you can post via the same url after restarting tomcat)
> > >> > >> >
> > >> > >> >> Date: Wed, 30 Oct 2013 11:00:21 -0400
> > >> > >> >> Subject: Re: [DISCUSS] Switching Streams from Camel deployment
> > to .war deployment
> > >> > >> >> From: jletourneau80@gmail.com
> > >> > >> >> To: dev@streams.incubator.apache.org
> > >> > >> >>
> > >> > >> >> To be fair, while the current implementation is heavily
> > camel-based,
> > >> > >> >> all of the interfaces related to Streams functionality are not.
> >  The
> > >> > >> >> current model maps to what Matt has outlined in my opinion,
> > though
> > >> > >> >> packing names etc. probably don't follow that exact pattern.
> > >> > >> >>
> > >> > >> >> With regards to the complexity and different components in the
> > >> > >> >> registration process, this was a cut at the abstraction based
> > on the
> > >> > >> >> assumption that different implementations may be plugged in and
> > in
> > >> > >> >> fact may live on different processor space (ie. a polling
> > publisher vs
> > >> > >> >> a push publisher may be instantiated on different servers but
> > the
> > >> > >> >> registration URL is staticly defined).
> > >> > >> >>
> > >> > >> >> Is the main argument I am seeing for Spring  the familiarity of
> > its
> > >> > >> >> IoC pattern implementation and syntax at the entry point?
> > >> > >> >>
> > >> > >> >> On Wed, Oct 30, 2013 at 10:53 AM, Danny Sullivan <
> > dsullivan7@hotmail.com> wrote:
> > >> > >> >> > Could you clarify whether the same entry points would exist
> > for the camel implementation of the core (implementing the "process"
> > method/ using a DynammicRouteBuilder) or would the webservice be the sole
> > entry point to Streams and after it enters would it hand it off to Camel?
> > And what would be the entry point for the Storm implementation?
> > >> > >> >> > -Danny
> > >> > >> >> >
> > >> > >> >> >> Date: Wed, 30 Oct 2013 10:13:04 -0400
> > >> > >> >> >> Subject: Re: [DISCUSS] Switching Streams from Camel
> > deployment to .war deployment
> > >> > >> >> >> From: m.ben.franklin@gmail.com
> > >> > >> >> >> To: dev@streams.incubator.apache.org
> > >> > >> >> >>
> > >> > >> >> >> On Tue, Oct 29, 2013 at 2:55 PM, Danny Sullivan <
> > dsullivan7@hotmail.com>wrote:
> > >> > >> >> >>
> > >> > >> >> >> > My case for switching from OSGi is for simplicity in
> > design. To follow the
> > >> > >> >> >> > path of an activity through Streams in the webservice,
> > there is one main
> > >> > >> >> >> > things the developer needs to understand:
> > >> > >> >> >> > The @RequestMapping annotation specifies the HTTP entry
> > point
> > >> > >> >> >> > There are 4 @RequestMapping annotations that correspond to
> > each of the 4
> > >> > >> >> >> > ways a user enters the application: registering a
> > publisher, registering a
> > >> > >> >> >> > subscriber, publishing activity, and getting an activity
> > stream. Where
> > >> > >> >> >> > these are located in the source code can be found by
> > searching for the
> > >> > >> >> >> > paths specified in the documentation (search for
> > "/publisherRegister",
> > >> > >> >> >> > "/subscriberRegister", "/publishActivity", "/getActivity"
> > which will all
> > >> > >> >> >> > lead you to StreamsWebController.java). From the methods
> > that process
> > >> > >> >> >> > requests, the flow through the application is through
> > methods which can be
> > >> > >> >> >> > understood by most Java programmers.
> > >> > >> >> >> > The flow of activities through the current trunk branch is
> > understood as
> > >> > >> >> >> > follows:
> > >> > >> >> >> > The string "/publisher/register" (the entry point to
> > register a publisher
> > >> > >> >> >> > specified in the documentation) is the value of the
> > >> > >> >> >> > consumer.registrationEndpoint property defined in
> > streams.propertiesThe
> > >> > >> >> >> > camelContext.xml specifies an endpoint with the id
> > >> > >> >> >> > consumerRegistrationEndpoint to have a uri equal to the
> > propertyThe
> > >> > >> >> >> > consumerRegistrationEndpoint routes from uri
> > direct:publisher register with
> > >> > >> >> >> > the bean activityRegistrationProcessor nested in between
> > the routeThe
> > >> > >> >> >> > streams-eip-applicationContext contains a bean with the id
> > >> > >> >> >> > activityRegistrationProcessor created for the class
> > >> > >> >> >> > ActivityPublisherRegistrationProcessorThe exchange will
> > enter the "process"
> > >> > >> >> >> > method of the class
> > ActivityPublisherRegisitrationProcessor and that this
> > >> > >> >> >> > is because this class implements the "Processor" interface
> > provided by
> > >> > >> >> >> > CamelThe direct:add-publisher-route takes the exchange
> > output from the
> > >> > >> >> >> > "process" method and routes it to the
> > activityRegistrationProcessor
> > >> > >> >> >> > "register" method. The bean activityRegistrationProcessor
> > is defined in the
> > >> > >> >> >> > streams-eip-osgi-component-import.xmlThe output from this
> > method is then
> > >> > >> >> >> > sent to the "createNewRouteForConsumer" method of
> > activityConsumerRouter.
> > >> > >> >> >> > This method creates a new route for the newly registered
> > publisher using
> > >> > >> >> >> > the private static final class DynamicConsumerRouteBuilder
> > which is
> > >> > >> >> >> > required to extend RouteBuilder which is provided by
> > Camel. This
> > >> > >> >> >> > DynamicConsumerRouteBuilder contains several methods:
> > >> > >> >> >> > getConsumerReceiveMethod() (which corresponds to @Value
> > >> > >> >> >> > ${consumer.receiveMethod} which corresponds to "receive"),
> > >> > >> >> >> > getConsumerSplitMethod() (@Value
> > ${consumer.splitMethod},"split"), and
> > >> > >> >> >> > getConsumerActivityQUri() (@Value ${consumerActivityQUri},
> > >> > >> >> >> > "direct:activityQ"). This is different than the
> > camelContext.xml in that
> > >> > >> >> >> > the route is being created programmatically. What this is
> > doing is routing
> > >> > >> >> >> > input from the inroute url (which Camel does automatically
> > through the
> > >> > >> >> >> > configure method which is required to be overridden), to
> > the "receive"
> > >> > >> >> >> > method of ActivityConsumer, then to the "split" method of
> > ActivityConsumer,
> > >> > >> >> >> > and then to the "direct:activityQ" which if you look back
> > in the
> > >> > >> >> >> > camelContext.xml routes to the "activemq:queue:activities"
> > which then
> > >> > >> >> >> > routes to "receiveExchange"
> > >> > >> >> >> > This is the process to register a publisher. The process
> > for registering a
> > >> > >> >> >> > subscriber is relatively the same though it involves
> > separate classes with
> > >> > >> >> >> > their own private static final RouteBuilder class. From my
> > perspective, the
> > >> > >> >> >> > two most difficult things with setting this project up
> > were understanding
> > >> > >> >> >> > that the "process" method of the class that implements
> > "Processor" is the
> > >> > >> >> >> > entry point and the DynamicConsumerRouteBuilder creates
> > the second entry
> > >> > >> >> >> > point (The 5th and last points). This made the project
> > very, VERY hard to
> > >> > >> >> >> > understand.
> > >> > >> >> >> > In addition to simplicity of design, the mvn clean install
> > of the web
> > >> > >> >> >> > service project is much faster and small scale activity
> > publishing is also
> > >> > >> >> >> > faster (see my email about load testing). These are minor
> > points though as
> > >> > >> >> >> > compilation has no effect on deployment. OSGi does add the
> > benefit of
> > >> > >> >> >> > modularized programming which is valuable, though I think
> > the added
> > >> > >> >> >> > complexity of Camel merits moving the project away from
> > this paradigm.
> > >> > >> >> >> >
> > >> > >> >> >>
> > >> > >> >> >> I agree that the project is pretty difficult to understand
> > ATM.  I think
> > >> > >> >> >> what we need to do is think about what the responsibilities
> > of the code are
> > >> > >> >> >> and allow for different implementations that are not so
> > tightly coupled as
> > >> > >> >> >> they are now.  For instance, having worked with Storm to
> > ingest millions of
> > >> > >> >> >> activities a day, I personally would like to see streams be
> > responsible for
> > >> > >> >> >> defining an over-arching orchestration model that can be
> > implemented within
> > >> > >> >> >> a single war or on top of a distributed system.  This would
> > look something
> > >> > >> >> >> like the follows:
> > >> > >> >> >>
> > >> > >> >> >> |__ Streams-Core  (Base classes, interfaces, utilities,
> > extensions, etc)
> > >> > >> >> >> |
> > >> > >> >> >> |__ Streams-Storm (Storm implementation of the core)
> > >> > >> >> >> |
> > >> > >> >> >> |__ Streams-Camel (Camel implementation of the core)
> > >> > >> >> >> |
> > >> > >> >> >> |__ Streams-WS (Web service implementation)
> > >> > >> >> >>
> > >> > >> >> >>
> > >> > >> >> >> > Danny
> > >> > >> >> >> >
> > >> > >> >> >> >
> > >> > >> >> >> > > From: dsullivan7@hotmail.com
> > >> > >> >> >> > > To: dev@streams.incubator.apache.org
> > >> > >> >> >> > > Subject: [DISCUSS] Switching Streams from Camel
> > deployment to .war
> > >> > >> >> >> > deployment
> > >> > >> >> >> > > Date: Tue, 29 Oct 2013 11:07:39 -0400
> > >> > >> >> >> > >
> > >> > >> >> >> > > The discussion thread for switching Streams from
> > Camel/osgi/Servicemix
> > >> > >> >> >> > to a single .war deployment
> > >> > >> >> >> >
> > >> > >> >> >> >
> > >> > >> >> >
> > >> > >> >
> > >> > >
> > >>
> > >
> >
 		 	   		  

Re: [DISCUSS] Switching Streams from Camel deployment to .war deployment

[Chris Geer <ch...@cxtsoftware.com>]

Or Content Enricher [2]

[2] http://camel.apache.org/content-enricher.html


On Thu, Oct 31, 2013 at 12:37 PM, Jason Letourneau
<jl...@gmail.com>wrote:

> check out the link here[1]
>
>
> [1]http://camel.apache.org/request-reply.html
>
> On Thu, Oct 31, 2013 at 2:32 PM, Danny Sullivan <ds...@hotmail.com>
> wrote:
> > I have a quick Camel question that I arrived at in the implementation of
> these new components:
> >
> > Lets say I have a method in streams-activity.jar that needs all
> subscribers in the database. This would require a call to the
> streams-persistence.jar. So far, I've seen camel used mostly for passing
> data through the application, but not for making a single request-reponse
> from within a method. How can I use Camel to get a list of all subscribers
> in the streams-persistence.jar from the streams-activity.jar?
> >
> >> From: dsullivan7@hotmail.com
> >> To: dev@streams.incubator.apache.org
> >> Subject: RE: [DISCUSS] Switching Streams from Camel deployment to .war
> deployment
> >> Date: Thu, 31 Oct 2013 08:40:44 -0400
> >>
> >> Excellent, I'll write up something as a proof of concept and we can
> discuss further to make sure everything is vanilla.
> >>
> >> > Date: Wed, 30 Oct 2013 17:01:28 -0400
> >> > Subject: Re: [DISCUSS] Switching Streams from Camel deployment to
> .war deployment
> >> > From: jletourneau80@gmail.com
> >> > To: dev@streams.incubator.apache.org
> >> >
> >> > That sounds pretty promising to me.
> >> >
> >> > On Wed, Oct 30, 2013 at 2:38 PM, Danny Sullivan <
> dsullivan7@hotmail.com> wrote:
> >> > > Thanks for the feedback. You have an interesting point about the
> url linking to a separate processing space. Let me tie my answer into your
> last question about "advocating for the simplicity at registration to give
> up flexibility at registration, but retaining the inner "guts" of
> EIP/messaging". Consider a new architecture:
> >> > >
> >> > > streams-web.war: single entry point to application, but functions
> ONLY as an entry point. From here Camel routes the incoming requests to 4
> separate jarssubscriber-registration.jar: subscriber registration
> publisher-registration.jar: publisher registrationactivity.jar: returns
> activity (also contains subscriber warehouse and storm activity
> aggregator)publish.jar: publishes activitystreams-cassandra.jar: the above
> 4 jars would all have a hook into this jar which would function as a hook
> onto the database. Each jar would have camel route output to this jar.
> >> > >
> >> > > In this implementation, Camel would no longer be the entry and exit
> point of a client to the application, but would handle the communication
> between components. The flow of activity through the application would be
> method based in each jar. This would allow deployment on up to 6 different
> process spaces. However, this does not address that there is a single
> server entry point, but I'm not sure if it was a concern in the first place.
> >> > >
> >> > > My argument, at its basis, is that we should move away from using
> Camel as the entry point to the application. I would be happy to maintain
> messaging between components.
> >> > >
> >> > >> Date: Wed, 30 Oct 2013 12:32:55 -0400
> >> > >> Subject: Re: [DISCUSS] Switching Streams from Camel deployment to
> .war deployment
> >> > >> From: jletourneau80@gmail.com
> >> > >> To: dev@streams.incubator.apache.org
> >> > >>
> >> > >> An interesting use case that I am holding onto is the ability for
> >> > >> publishers to register via a single URL (registration endpoint),
> but
> >> > >> be sent a URL back to post to a different process space for actual
> >> > >> publishing.  The same is true on the subscriber front.  Currently,
> the
> >> > >> Camel/EIP infrastructure abstracts this because different
> components
> >> > >> deployed in different process spaces handling the route creation
> can
> >> > >> just be bolted onto a running Streams instance without new
> subs/pubs
> >> > >> behaving any differently than existing.  This seems to be a
> >> > >> potentially critical scaling point.  Is there a way to do this with
> >> > >> the Spring solution?
> >> > >>
> >> > >> The persistence point is a good one, though I would classify that
> as
> >> > >> "not implemented" vs "not possible" (not that you were).
> >> > >>
> >> > >> I'm not married to Camel, I just like the EIP approach to building
> >> > >> something that is ultimately a messaging system.  There are known
> >> > >> patterns that solve at least a subset of the problems Streams is
> >> > >> trying to solve and implementations that can handle the load and
> I'll
> >> > >> reiterate flexibility == complexity almost always.
> >> > >>
> >> > >> It comes right back to the central question: Do you want
> flexibility
> >> > >> or simplicity?  It doesn't have to be black and white either I
> don't
> >> > >> think...
> >> > >>
> >> > >> More pointedly: Where should we give up flexibility for
> simplicity?  I
> >> > >> read that Danny is advocating for the simplicity at registration to
> >> > >> give up flexibility at registration, but retaining the inner
> "guts" of
> >> > >> EIP/messaging?  Thoughts?
> >> > >>
> >> > >> On Wed, Oct 30, 2013 at 11:37 AM, Danny Sullivan <
> dsullivan7@hotmail.com> wrote:
> >> > >> > My argument is not for the IoC pattern as that can be (and has
> been) implemented alongside Camel. My main argument is that the syntax at
> the entry point is not only familiar but much simpler. This wouldn't be a
> very strong argument if the Camel implementation wasn't much more
> complicated but I feel that it is the case. Also, looking toward the
> future, if the server is restarted, in-routes are lost in Camel. The way to
> curb this is to persist the dynamic routes that Camel creates, and then on
> start up pull every one of these routes and recreate a dynamic route for
> each one. Not only is this much easier to implement using the Spring web
> implementation, but it already has been implemented and you can try it by
> checking out the webservice branch, registering a subscriber, restarting
> tomcat, and using the same url you had before. This will allow subscribers
> to hang on to their urls once they register. (the same is true for
> publishers: you can post via the same url after restarting tomcat)
> >> > >> >
> >> > >> >> Date: Wed, 30 Oct 2013 11:00:21 -0400
> >> > >> >> Subject: Re: [DISCUSS] Switching Streams from Camel deployment
> to .war deployment
> >> > >> >> From: jletourneau80@gmail.com
> >> > >> >> To: dev@streams.incubator.apache.org
> >> > >> >>
> >> > >> >> To be fair, while the current implementation is heavily
> camel-based,
> >> > >> >> all of the interfaces related to Streams functionality are not.
>  The
> >> > >> >> current model maps to what Matt has outlined in my opinion,
> though
> >> > >> >> packing names etc. probably don't follow that exact pattern.
> >> > >> >>
> >> > >> >> With regards to the complexity and different components in the
> >> > >> >> registration process, this was a cut at the abstraction based
> on the
> >> > >> >> assumption that different implementations may be plugged in and
> in
> >> > >> >> fact may live on different processor space (ie. a polling
> publisher vs
> >> > >> >> a push publisher may be instantiated on different servers but
> the
> >> > >> >> registration URL is staticly defined).
> >> > >> >>
> >> > >> >> Is the main argument I am seeing for Spring  the familiarity of
> its
> >> > >> >> IoC pattern implementation and syntax at the entry point?
> >> > >> >>
> >> > >> >> On Wed, Oct 30, 2013 at 10:53 AM, Danny Sullivan <
> dsullivan7@hotmail.com> wrote:
> >> > >> >> > Could you clarify whether the same entry points would exist
> for the camel implementation of the core (implementing the "process"
> method/ using a DynammicRouteBuilder) or would the webservice be the sole
> entry point to Streams and after it enters would it hand it off to Camel?
> And what would be the entry point for the Storm implementation?
> >> > >> >> > -Danny
> >> > >> >> >
> >> > >> >> >> Date: Wed, 30 Oct 2013 10:13:04 -0400
> >> > >> >> >> Subject: Re: [DISCUSS] Switching Streams from Camel
> deployment to .war deployment
> >> > >> >> >> From: m.ben.franklin@gmail.com
> >> > >> >> >> To: dev@streams.incubator.apache.org
> >> > >> >> >>
> >> > >> >> >> On Tue, Oct 29, 2013 at 2:55 PM, Danny Sullivan <
> dsullivan7@hotmail.com>wrote:
> >> > >> >> >>
> >> > >> >> >> > My case for switching from OSGi is for simplicity in
> design. To follow the
> >> > >> >> >> > path of an activity through Streams in the webservice,
> there is one main
> >> > >> >> >> > things the developer needs to understand:
> >> > >> >> >> > The @RequestMapping annotation specifies the HTTP entry
> point
> >> > >> >> >> > There are 4 @RequestMapping annotations that correspond to
> each of the 4
> >> > >> >> >> > ways a user enters the application: registering a
> publisher, registering a
> >> > >> >> >> > subscriber, publishing activity, and getting an activity
> stream. Where
> >> > >> >> >> > these are located in the source code can be found by
> searching for the
> >> > >> >> >> > paths specified in the documentation (search for
> "/publisherRegister",
> >> > >> >> >> > "/subscriberRegister", "/publishActivity", "/getActivity"
> which will all
> >> > >> >> >> > lead you to StreamsWebController.java). From the methods
> that process
> >> > >> >> >> > requests, the flow through the application is through
> methods which can be
> >> > >> >> >> > understood by most Java programmers.
> >> > >> >> >> > The flow of activities through the current trunk branch is
> understood as
> >> > >> >> >> > follows:
> >> > >> >> >> > The string "/publisher/register" (the entry point to
> register a publisher
> >> > >> >> >> > specified in the documentation) is the value of the
> >> > >> >> >> > consumer.registrationEndpoint property defined in
> streams.propertiesThe
> >> > >> >> >> > camelContext.xml specifies an endpoint with the id
> >> > >> >> >> > consumerRegistrationEndpoint to have a uri equal to the
> propertyThe
> >> > >> >> >> > consumerRegistrationEndpoint routes from uri
> direct:publisher register with
> >> > >> >> >> > the bean activityRegistrationProcessor nested in between
> the routeThe
> >> > >> >> >> > streams-eip-applicationContext contains a bean with the id
> >> > >> >> >> > activityRegistrationProcessor created for the class
> >> > >> >> >> > ActivityPublisherRegistrationProcessorThe exchange will
> enter the "process"
> >> > >> >> >> > method of the class
> ActivityPublisherRegisitrationProcessor and that this
> >> > >> >> >> > is because this class implements the "Processor" interface
> provided by
> >> > >> >> >> > CamelThe direct:add-publisher-route takes the exchange
> output from the
> >> > >> >> >> > "process" method and routes it to the
> activityRegistrationProcessor
> >> > >> >> >> > "register" method. The bean activityRegistrationProcessor
> is defined in the
> >> > >> >> >> > streams-eip-osgi-component-import.xmlThe output from this
> method is then
> >> > >> >> >> > sent to the "createNewRouteForConsumer" method of
> activityConsumerRouter.
> >> > >> >> >> > This method creates a new route for the newly registered
> publisher using
> >> > >> >> >> > the private static final class DynamicConsumerRouteBuilder
> which is
> >> > >> >> >> > required to extend RouteBuilder which is provided by
> Camel. This
> >> > >> >> >> > DynamicConsumerRouteBuilder contains several methods:
> >> > >> >> >> > getConsumerReceiveMethod() (which corresponds to @Value
> >> > >> >> >> > ${consumer.receiveMethod} which corresponds to "receive"),
> >> > >> >> >> > getConsumerSplitMethod() (@Value
> ${consumer.splitMethod},"split"), and
> >> > >> >> >> > getConsumerActivityQUri() (@Value ${consumerActivityQUri},
> >> > >> >> >> > "direct:activityQ"). This is different than the
> camelContext.xml in that
> >> > >> >> >> > the route is being created programmatically. What this is
> doing is routing
> >> > >> >> >> > input from the inroute url (which Camel does automatically
> through the
> >> > >> >> >> > configure method which is required to be overridden), to
> the "receive"
> >> > >> >> >> > method of ActivityConsumer, then to the "split" method of
> ActivityConsumer,
> >> > >> >> >> > and then to the "direct:activityQ" which if you look back
> in the
> >> > >> >> >> > camelContext.xml routes to the "activemq:queue:activities"
> which then
> >> > >> >> >> > routes to "receiveExchange"
> >> > >> >> >> > This is the process to register a publisher. The process
> for registering a
> >> > >> >> >> > subscriber is relatively the same though it involves
> separate classes with
> >> > >> >> >> > their own private static final RouteBuilder class. From my
> perspective, the
> >> > >> >> >> > two most difficult things with setting this project up
> were understanding
> >> > >> >> >> > that the "process" method of the class that implements
> "Processor" is the
> >> > >> >> >> > entry point and the DynamicConsumerRouteBuilder creates
> the second entry
> >> > >> >> >> > point (The 5th and last points). This made the project
> very, VERY hard to
> >> > >> >> >> > understand.
> >> > >> >> >> > In addition to simplicity of design, the mvn clean install
> of the web
> >> > >> >> >> > service project is much faster and small scale activity
> publishing is also
> >> > >> >> >> > faster (see my email about load testing). These are minor
> points though as
> >> > >> >> >> > compilation has no effect on deployment. OSGi does add the
> benefit of
> >> > >> >> >> > modularized programming which is valuable, though I think
> the added
> >> > >> >> >> > complexity of Camel merits moving the project away from
> this paradigm.
> >> > >> >> >> >
> >> > >> >> >>
> >> > >> >> >> I agree that the project is pretty difficult to understand
> ATM.  I think
> >> > >> >> >> what we need to do is think about what the responsibilities
> of the code are
> >> > >> >> >> and allow for different implementations that are not so
> tightly coupled as
> >> > >> >> >> they are now.  For instance, having worked with Storm to
> ingest millions of
> >> > >> >> >> activities a day, I personally would like to see streams be
> responsible for
> >> > >> >> >> defining an over-arching orchestration model that can be
> implemented within
> >> > >> >> >> a single war or on top of a distributed system.  This would
> look something
> >> > >> >> >> like the follows:
> >> > >> >> >>
> >> > >> >> >> |__ Streams-Core  (Base classes, interfaces, utilities,
> extensions, etc)
> >> > >> >> >> |
> >> > >> >> >> |__ Streams-Storm (Storm implementation of the core)
> >> > >> >> >> |
> >> > >> >> >> |__ Streams-Camel (Camel implementation of the core)
> >> > >> >> >> |
> >> > >> >> >> |__ Streams-WS (Web service implementation)
> >> > >> >> >>
> >> > >> >> >>
> >> > >> >> >> > Danny
> >> > >> >> >> >
> >> > >> >> >> >
> >> > >> >> >> > > From: dsullivan7@hotmail.com
> >> > >> >> >> > > To: dev@streams.incubator.apache.org
> >> > >> >> >> > > Subject: [DISCUSS] Switching Streams from Camel
> deployment to .war
> >> > >> >> >> > deployment
> >> > >> >> >> > > Date: Tue, 29 Oct 2013 11:07:39 -0400
> >> > >> >> >> > >
> >> > >> >> >> > > The discussion thread for switching Streams from
> Camel/osgi/Servicemix
> >> > >> >> >> > to a single .war deployment
> >> > >> >> >> >
> >> > >> >> >> >
> >> > >> >> >
> >> > >> >
> >> > >
> >>
> >
>

Re: [DISCUSS] Switching Streams from Camel deployment to .war deployment

[Jason Letourneau <jl...@gmail.com>]

check out the link here[1]


[1]http://camel.apache.org/request-reply.html

On Thu, Oct 31, 2013 at 2:32 PM, Danny Sullivan <ds...@hotmail.com> wrote:
> I have a quick Camel question that I arrived at in the implementation of these new components:
>
> Lets say I have a method in streams-activity.jar that needs all subscribers in the database. This would require a call to the streams-persistence.jar. So far, I've seen camel used mostly for passing data through the application, but not for making a single request-reponse from within a method. How can I use Camel to get a list of all subscribers in the streams-persistence.jar from the streams-activity.jar?
>
>> From: dsullivan7@hotmail.com
>> To: dev@streams.incubator.apache.org
>> Subject: RE: [DISCUSS] Switching Streams from Camel deployment to .war deployment
>> Date: Thu, 31 Oct 2013 08:40:44 -0400
>>
>> Excellent, I'll write up something as a proof of concept and we can discuss further to make sure everything is vanilla.
>>
>> > Date: Wed, 30 Oct 2013 17:01:28 -0400
>> > Subject: Re: [DISCUSS] Switching Streams from Camel deployment to .war deployment
>> > From: jletourneau80@gmail.com
>> > To: dev@streams.incubator.apache.org
>> >
>> > That sounds pretty promising to me.
>> >
>> > On Wed, Oct 30, 2013 at 2:38 PM, Danny Sullivan <ds...@hotmail.com> wrote:
>> > > Thanks for the feedback. You have an interesting point about the url linking to a separate processing space. Let me tie my answer into your last question about "advocating for the simplicity at registration to give up flexibility at registration, but retaining the inner "guts" of EIP/messaging". Consider a new architecture:
>> > >
>> > > streams-web.war: single entry point to application, but functions ONLY as an entry point. From here Camel routes the incoming requests to 4 separate jarssubscriber-registration.jar: subscriber registration publisher-registration.jar: publisher registrationactivity.jar: returns activity (also contains subscriber warehouse and storm activity aggregator)publish.jar: publishes activitystreams-cassandra.jar: the above 4 jars would all have a hook into this jar which would function as a hook onto the database. Each jar would have camel route output to this jar.
>> > >
>> > > In this implementation, Camel would no longer be the entry and exit point of a client to the application, but would handle the communication between components. The flow of activity through the application would be method based in each jar. This would allow deployment on up to 6 different process spaces. However, this does not address that there is a single server entry point, but I'm not sure if it was a concern in the first place.
>> > >
>> > > My argument, at its basis, is that we should move away from using Camel as the entry point to the application. I would be happy to maintain messaging between components.
>> > >
>> > >> Date: Wed, 30 Oct 2013 12:32:55 -0400
>> > >> Subject: Re: [DISCUSS] Switching Streams from Camel deployment to .war deployment
>> > >> From: jletourneau80@gmail.com
>> > >> To: dev@streams.incubator.apache.org
>> > >>
>> > >> An interesting use case that I am holding onto is the ability for
>> > >> publishers to register via a single URL (registration endpoint), but
>> > >> be sent a URL back to post to a different process space for actual
>> > >> publishing.  The same is true on the subscriber front.  Currently, the
>> > >> Camel/EIP infrastructure abstracts this because different components
>> > >> deployed in different process spaces handling the route creation can
>> > >> just be bolted onto a running Streams instance without new subs/pubs
>> > >> behaving any differently than existing.  This seems to be a
>> > >> potentially critical scaling point.  Is there a way to do this with
>> > >> the Spring solution?
>> > >>
>> > >> The persistence point is a good one, though I would classify that as
>> > >> "not implemented" vs "not possible" (not that you were).
>> > >>
>> > >> I'm not married to Camel, I just like the EIP approach to building
>> > >> something that is ultimately a messaging system.  There are known
>> > >> patterns that solve at least a subset of the problems Streams is
>> > >> trying to solve and implementations that can handle the load and I'll
>> > >> reiterate flexibility == complexity almost always.
>> > >>
>> > >> It comes right back to the central question: Do you want flexibility
>> > >> or simplicity?  It doesn't have to be black and white either I don't
>> > >> think...
>> > >>
>> > >> More pointedly: Where should we give up flexibility for simplicity?  I
>> > >> read that Danny is advocating for the simplicity at registration to
>> > >> give up flexibility at registration, but retaining the inner "guts" of
>> > >> EIP/messaging?  Thoughts?
>> > >>
>> > >> On Wed, Oct 30, 2013 at 11:37 AM, Danny Sullivan <ds...@hotmail.com> wrote:
>> > >> > My argument is not for the IoC pattern as that can be (and has been) implemented alongside Camel. My main argument is that the syntax at the entry point is not only familiar but much simpler. This wouldn't be a very strong argument if the Camel implementation wasn't much more complicated but I feel that it is the case. Also, looking toward the future, if the server is restarted, in-routes are lost in Camel. The way to curb this is to persist the dynamic routes that Camel creates, and then on start up pull every one of these routes and recreate a dynamic route for each one. Not only is this much easier to implement using the Spring web implementation, but it already has been implemented and you can try it by checking out the webservice branch, registering a subscriber, restarting tomcat, and using the same url you had before. This will allow subscribers to hang on to their urls once they register. (the same is true for publishers: you can post via the same url after restarting tomcat)
>> > >> >
>> > >> >> Date: Wed, 30 Oct 2013 11:00:21 -0400
>> > >> >> Subject: Re: [DISCUSS] Switching Streams from Camel deployment to .war deployment
>> > >> >> From: jletourneau80@gmail.com
>> > >> >> To: dev@streams.incubator.apache.org
>> > >> >>
>> > >> >> To be fair, while the current implementation is heavily camel-based,
>> > >> >> all of the interfaces related to Streams functionality are not.  The
>> > >> >> current model maps to what Matt has outlined in my opinion, though
>> > >> >> packing names etc. probably don't follow that exact pattern.
>> > >> >>
>> > >> >> With regards to the complexity and different components in the
>> > >> >> registration process, this was a cut at the abstraction based on the
>> > >> >> assumption that different implementations may be plugged in and in
>> > >> >> fact may live on different processor space (ie. a polling publisher vs
>> > >> >> a push publisher may be instantiated on different servers but the
>> > >> >> registration URL is staticly defined).
>> > >> >>
>> > >> >> Is the main argument I am seeing for Spring  the familiarity of its
>> > >> >> IoC pattern implementation and syntax at the entry point?
>> > >> >>
>> > >> >> On Wed, Oct 30, 2013 at 10:53 AM, Danny Sullivan <ds...@hotmail.com> wrote:
>> > >> >> > Could you clarify whether the same entry points would exist for the camel implementation of the core (implementing the "process" method/ using a DynammicRouteBuilder) or would the webservice be the sole entry point to Streams and after it enters would it hand it off to Camel? And what would be the entry point for the Storm implementation?
>> > >> >> > -Danny
>> > >> >> >
>> > >> >> >> Date: Wed, 30 Oct 2013 10:13:04 -0400
>> > >> >> >> Subject: Re: [DISCUSS] Switching Streams from Camel deployment to .war deployment
>> > >> >> >> From: m.ben.franklin@gmail.com
>> > >> >> >> To: dev@streams.incubator.apache.org
>> > >> >> >>
>> > >> >> >> On Tue, Oct 29, 2013 at 2:55 PM, Danny Sullivan <ds...@hotmail.com>wrote:
>> > >> >> >>
>> > >> >> >> > My case for switching from OSGi is for simplicity in design. To follow the
>> > >> >> >> > path of an activity through Streams in the webservice, there is one main
>> > >> >> >> > things the developer needs to understand:
>> > >> >> >> > The @RequestMapping annotation specifies the HTTP entry point
>> > >> >> >> > There are 4 @RequestMapping annotations that correspond to each of the 4
>> > >> >> >> > ways a user enters the application: registering a publisher, registering a
>> > >> >> >> > subscriber, publishing activity, and getting an activity stream. Where
>> > >> >> >> > these are located in the source code can be found by searching for the
>> > >> >> >> > paths specified in the documentation (search for "/publisherRegister",
>> > >> >> >> > "/subscriberRegister", "/publishActivity", "/getActivity" which will all
>> > >> >> >> > lead you to StreamsWebController.java). From the methods that process
>> > >> >> >> > requests, the flow through the application is through methods which can be
>> > >> >> >> > understood by most Java programmers.
>> > >> >> >> > The flow of activities through the current trunk branch is understood as
>> > >> >> >> > follows:
>> > >> >> >> > The string "/publisher/register" (the entry point to register a publisher
>> > >> >> >> > specified in the documentation) is the value of the
>> > >> >> >> > consumer.registrationEndpoint property defined in streams.propertiesThe
>> > >> >> >> > camelContext.xml specifies an endpoint with the id
>> > >> >> >> > consumerRegistrationEndpoint to have a uri equal to the propertyThe
>> > >> >> >> > consumerRegistrationEndpoint routes from uri direct:publisher register with
>> > >> >> >> > the bean activityRegistrationProcessor nested in between the routeThe
>> > >> >> >> > streams-eip-applicationContext contains a bean with the id
>> > >> >> >> > activityRegistrationProcessor created for the class
>> > >> >> >> > ActivityPublisherRegistrationProcessorThe exchange will enter the "process"
>> > >> >> >> > method of the class ActivityPublisherRegisitrationProcessor and that this
>> > >> >> >> > is because this class implements the "Processor" interface provided by
>> > >> >> >> > CamelThe direct:add-publisher-route takes the exchange output from the
>> > >> >> >> > "process" method and routes it to the activityRegistrationProcessor
>> > >> >> >> > "register" method. The bean activityRegistrationProcessor is defined in the
>> > >> >> >> > streams-eip-osgi-component-import.xmlThe output from this method is then
>> > >> >> >> > sent to the "createNewRouteForConsumer" method of activityConsumerRouter.
>> > >> >> >> > This method creates a new route for the newly registered publisher using
>> > >> >> >> > the private static final class DynamicConsumerRouteBuilder which is
>> > >> >> >> > required to extend RouteBuilder which is provided by Camel. This
>> > >> >> >> > DynamicConsumerRouteBuilder contains several methods:
>> > >> >> >> > getConsumerReceiveMethod() (which corresponds to @Value
>> > >> >> >> > ${consumer.receiveMethod} which corresponds to "receive"),
>> > >> >> >> > getConsumerSplitMethod() (@Value ${consumer.splitMethod},"split"), and
>> > >> >> >> > getConsumerActivityQUri() (@Value ${consumerActivityQUri},
>> > >> >> >> > "direct:activityQ"). This is different than the camelContext.xml in that
>> > >> >> >> > the route is being created programmatically. What this is doing is routing
>> > >> >> >> > input from the inroute url (which Camel does automatically through the
>> > >> >> >> > configure method which is required to be overridden), to the "receive"
>> > >> >> >> > method of ActivityConsumer, then to the "split" method of ActivityConsumer,
>> > >> >> >> > and then to the "direct:activityQ" which if you look back in the
>> > >> >> >> > camelContext.xml routes to the "activemq:queue:activities" which then
>> > >> >> >> > routes to "receiveExchange"
>> > >> >> >> > This is the process to register a publisher. The process for registering a
>> > >> >> >> > subscriber is relatively the same though it involves separate classes with
>> > >> >> >> > their own private static final RouteBuilder class. From my perspective, the
>> > >> >> >> > two most difficult things with setting this project up were understanding
>> > >> >> >> > that the "process" method of the class that implements "Processor" is the
>> > >> >> >> > entry point and the DynamicConsumerRouteBuilder creates the second entry
>> > >> >> >> > point (The 5th and last points). This made the project very, VERY hard to
>> > >> >> >> > understand.
>> > >> >> >> > In addition to simplicity of design, the mvn clean install of the web
>> > >> >> >> > service project is much faster and small scale activity publishing is also
>> > >> >> >> > faster (see my email about load testing). These are minor points though as
>> > >> >> >> > compilation has no effect on deployment. OSGi does add the benefit of
>> > >> >> >> > modularized programming which is valuable, though I think the added
>> > >> >> >> > complexity of Camel merits moving the project away from this paradigm.
>> > >> >> >> >
>> > >> >> >>
>> > >> >> >> I agree that the project is pretty difficult to understand ATM.  I think
>> > >> >> >> what we need to do is think about what the responsibilities of the code are
>> > >> >> >> and allow for different implementations that are not so tightly coupled as
>> > >> >> >> they are now.  For instance, having worked with Storm to ingest millions of
>> > >> >> >> activities a day, I personally would like to see streams be responsible for
>> > >> >> >> defining an over-arching orchestration model that can be implemented within
>> > >> >> >> a single war or on top of a distributed system.  This would look something
>> > >> >> >> like the follows:
>> > >> >> >>
>> > >> >> >> |__ Streams-Core  (Base classes, interfaces, utilities, extensions, etc)
>> > >> >> >> |
>> > >> >> >> |__ Streams-Storm (Storm implementation of the core)
>> > >> >> >> |
>> > >> >> >> |__ Streams-Camel (Camel implementation of the core)
>> > >> >> >> |
>> > >> >> >> |__ Streams-WS (Web service implementation)
>> > >> >> >>
>> > >> >> >>
>> > >> >> >> > Danny
>> > >> >> >> >
>> > >> >> >> >
>> > >> >> >> > > From: dsullivan7@hotmail.com
>> > >> >> >> > > To: dev@streams.incubator.apache.org
>> > >> >> >> > > Subject: [DISCUSS] Switching Streams from Camel deployment to .war
>> > >> >> >> > deployment
>> > >> >> >> > > Date: Tue, 29 Oct 2013 11:07:39 -0400
>> > >> >> >> > >
>> > >> >> >> > > The discussion thread for switching Streams from Camel/osgi/Servicemix
>> > >> >> >> > to a single .war deployment
>> > >> >> >> >
>> > >> >> >> >
>> > >> >> >
>> > >> >
>> > >
>>
>

RE: [DISCUSS] Switching Streams from Camel deployment to .war deployment

[Danny Sullivan <ds...@hotmail.com>]

I have a quick Camel question that I arrived at in the implementation of these new components:

Lets say I have a method in streams-activity.jar that needs all subscribers in the database. This would require a call to the streams-persistence.jar. So far, I've seen camel used mostly for passing data through the application, but not for making a single request-reponse from within a method. How can I use Camel to get a list of all subscribers in the streams-persistence.jar from the streams-activity.jar? 

> From: dsullivan7@hotmail.com
> To: dev@streams.incubator.apache.org
> Subject: RE: [DISCUSS] Switching Streams from Camel deployment to .war deployment
> Date: Thu, 31 Oct 2013 08:40:44 -0400
> 
> Excellent, I'll write up something as a proof of concept and we can discuss further to make sure everything is vanilla.
> 
> > Date: Wed, 30 Oct 2013 17:01:28 -0400
> > Subject: Re: [DISCUSS] Switching Streams from Camel deployment to .war deployment
> > From: jletourneau80@gmail.com
> > To: dev@streams.incubator.apache.org
> > 
> > That sounds pretty promising to me.
> > 
> > On Wed, Oct 30, 2013 at 2:38 PM, Danny Sullivan <ds...@hotmail.com> wrote:
> > > Thanks for the feedback. You have an interesting point about the url linking to a separate processing space. Let me tie my answer into your last question about "advocating for the simplicity at registration to give up flexibility at registration, but retaining the inner "guts" of EIP/messaging". Consider a new architecture:
> > >
> > > streams-web.war: single entry point to application, but functions ONLY as an entry point. From here Camel routes the incoming requests to 4 separate jarssubscriber-registration.jar: subscriber registration publisher-registration.jar: publisher registrationactivity.jar: returns activity (also contains subscriber warehouse and storm activity aggregator)publish.jar: publishes activitystreams-cassandra.jar: the above 4 jars would all have a hook into this jar which would function as a hook onto the database. Each jar would have camel route output to this jar.
> > >
> > > In this implementation, Camel would no longer be the entry and exit point of a client to the application, but would handle the communication between components. The flow of activity through the application would be method based in each jar. This would allow deployment on up to 6 different process spaces. However, this does not address that there is a single server entry point, but I'm not sure if it was a concern in the first place.
> > >
> > > My argument, at its basis, is that we should move away from using Camel as the entry point to the application. I would be happy to maintain messaging between components.
> > >
> > >> Date: Wed, 30 Oct 2013 12:32:55 -0400
> > >> Subject: Re: [DISCUSS] Switching Streams from Camel deployment to .war deployment
> > >> From: jletourneau80@gmail.com
> > >> To: dev@streams.incubator.apache.org
> > >>
> > >> An interesting use case that I am holding onto is the ability for
> > >> publishers to register via a single URL (registration endpoint), but
> > >> be sent a URL back to post to a different process space for actual
> > >> publishing.  The same is true on the subscriber front.  Currently, the
> > >> Camel/EIP infrastructure abstracts this because different components
> > >> deployed in different process spaces handling the route creation can
> > >> just be bolted onto a running Streams instance without new subs/pubs
> > >> behaving any differently than existing.  This seems to be a
> > >> potentially critical scaling point.  Is there a way to do this with
> > >> the Spring solution?
> > >>
> > >> The persistence point is a good one, though I would classify that as
> > >> "not implemented" vs "not possible" (not that you were).
> > >>
> > >> I'm not married to Camel, I just like the EIP approach to building
> > >> something that is ultimately a messaging system.  There are known
> > >> patterns that solve at least a subset of the problems Streams is
> > >> trying to solve and implementations that can handle the load and I'll
> > >> reiterate flexibility == complexity almost always.
> > >>
> > >> It comes right back to the central question: Do you want flexibility
> > >> or simplicity?  It doesn't have to be black and white either I don't
> > >> think...
> > >>
> > >> More pointedly: Where should we give up flexibility for simplicity?  I
> > >> read that Danny is advocating for the simplicity at registration to
> > >> give up flexibility at registration, but retaining the inner "guts" of
> > >> EIP/messaging?  Thoughts?
> > >>
> > >> On Wed, Oct 30, 2013 at 11:37 AM, Danny Sullivan <ds...@hotmail.com> wrote:
> > >> > My argument is not for the IoC pattern as that can be (and has been) implemented alongside Camel. My main argument is that the syntax at the entry point is not only familiar but much simpler. This wouldn't be a very strong argument if the Camel implementation wasn't much more complicated but I feel that it is the case. Also, looking toward the future, if the server is restarted, in-routes are lost in Camel. The way to curb this is to persist the dynamic routes that Camel creates, and then on start up pull every one of these routes and recreate a dynamic route for each one. Not only is this much easier to implement using the Spring web implementation, but it already has been implemented and you can try it by checking out the webservice branch, registering a subscriber, restarting tomcat, and using the same url you had before. This will allow subscribers to hang on to their urls once they register. (the same is true for publishers: you can post via the same url after restarting tomcat)
> > >> >
> > >> >> Date: Wed, 30 Oct 2013 11:00:21 -0400
> > >> >> Subject: Re: [DISCUSS] Switching Streams from Camel deployment to .war deployment
> > >> >> From: jletourneau80@gmail.com
> > >> >> To: dev@streams.incubator.apache.org
> > >> >>
> > >> >> To be fair, while the current implementation is heavily camel-based,
> > >> >> all of the interfaces related to Streams functionality are not.  The
> > >> >> current model maps to what Matt has outlined in my opinion, though
> > >> >> packing names etc. probably don't follow that exact pattern.
> > >> >>
> > >> >> With regards to the complexity and different components in the
> > >> >> registration process, this was a cut at the abstraction based on the
> > >> >> assumption that different implementations may be plugged in and in
> > >> >> fact may live on different processor space (ie. a polling publisher vs
> > >> >> a push publisher may be instantiated on different servers but the
> > >> >> registration URL is staticly defined).
> > >> >>
> > >> >> Is the main argument I am seeing for Spring  the familiarity of its
> > >> >> IoC pattern implementation and syntax at the entry point?
> > >> >>
> > >> >> On Wed, Oct 30, 2013 at 10:53 AM, Danny Sullivan <ds...@hotmail.com> wrote:
> > >> >> > Could you clarify whether the same entry points would exist for the camel implementation of the core (implementing the "process" method/ using a DynammicRouteBuilder) or would the webservice be the sole entry point to Streams and after it enters would it hand it off to Camel? And what would be the entry point for the Storm implementation?
> > >> >> > -Danny
> > >> >> >
> > >> >> >> Date: Wed, 30 Oct 2013 10:13:04 -0400
> > >> >> >> Subject: Re: [DISCUSS] Switching Streams from Camel deployment to .war deployment
> > >> >> >> From: m.ben.franklin@gmail.com
> > >> >> >> To: dev@streams.incubator.apache.org
> > >> >> >>
> > >> >> >> On Tue, Oct 29, 2013 at 2:55 PM, Danny Sullivan <ds...@hotmail.com>wrote:
> > >> >> >>
> > >> >> >> > My case for switching from OSGi is for simplicity in design. To follow the
> > >> >> >> > path of an activity through Streams in the webservice, there is one main
> > >> >> >> > things the developer needs to understand:
> > >> >> >> > The @RequestMapping annotation specifies the HTTP entry point
> > >> >> >> > There are 4 @RequestMapping annotations that correspond to each of the 4
> > >> >> >> > ways a user enters the application: registering a publisher, registering a
> > >> >> >> > subscriber, publishing activity, and getting an activity stream. Where
> > >> >> >> > these are located in the source code can be found by searching for the
> > >> >> >> > paths specified in the documentation (search for "/publisherRegister",
> > >> >> >> > "/subscriberRegister", "/publishActivity", "/getActivity" which will all
> > >> >> >> > lead you to StreamsWebController.java). From the methods that process
> > >> >> >> > requests, the flow through the application is through methods which can be
> > >> >> >> > understood by most Java programmers.
> > >> >> >> > The flow of activities through the current trunk branch is understood as
> > >> >> >> > follows:
> > >> >> >> > The string "/publisher/register" (the entry point to register a publisher
> > >> >> >> > specified in the documentation) is the value of the
> > >> >> >> > consumer.registrationEndpoint property defined in streams.propertiesThe
> > >> >> >> > camelContext.xml specifies an endpoint with the id
> > >> >> >> > consumerRegistrationEndpoint to have a uri equal to the propertyThe
> > >> >> >> > consumerRegistrationEndpoint routes from uri direct:publisher register with
> > >> >> >> > the bean activityRegistrationProcessor nested in between the routeThe
> > >> >> >> > streams-eip-applicationContext contains a bean with the id
> > >> >> >> > activityRegistrationProcessor created for the class
> > >> >> >> > ActivityPublisherRegistrationProcessorThe exchange will enter the "process"
> > >> >> >> > method of the class ActivityPublisherRegisitrationProcessor and that this
> > >> >> >> > is because this class implements the "Processor" interface provided by
> > >> >> >> > CamelThe direct:add-publisher-route takes the exchange output from the
> > >> >> >> > "process" method and routes it to the activityRegistrationProcessor
> > >> >> >> > "register" method. The bean activityRegistrationProcessor is defined in the
> > >> >> >> > streams-eip-osgi-component-import.xmlThe output from this method is then
> > >> >> >> > sent to the "createNewRouteForConsumer" method of activityConsumerRouter.
> > >> >> >> > This method creates a new route for the newly registered publisher using
> > >> >> >> > the private static final class DynamicConsumerRouteBuilder which is
> > >> >> >> > required to extend RouteBuilder which is provided by Camel. This
> > >> >> >> > DynamicConsumerRouteBuilder contains several methods:
> > >> >> >> > getConsumerReceiveMethod() (which corresponds to @Value
> > >> >> >> > ${consumer.receiveMethod} which corresponds to "receive"),
> > >> >> >> > getConsumerSplitMethod() (@Value ${consumer.splitMethod},"split"), and
> > >> >> >> > getConsumerActivityQUri() (@Value ${consumerActivityQUri},
> > >> >> >> > "direct:activityQ"). This is different than the camelContext.xml in that
> > >> >> >> > the route is being created programmatically. What this is doing is routing
> > >> >> >> > input from the inroute url (which Camel does automatically through the
> > >> >> >> > configure method which is required to be overridden), to the "receive"
> > >> >> >> > method of ActivityConsumer, then to the "split" method of ActivityConsumer,
> > >> >> >> > and then to the "direct:activityQ" which if you look back in the
> > >> >> >> > camelContext.xml routes to the "activemq:queue:activities" which then
> > >> >> >> > routes to "receiveExchange"
> > >> >> >> > This is the process to register a publisher. The process for registering a
> > >> >> >> > subscriber is relatively the same though it involves separate classes with
> > >> >> >> > their own private static final RouteBuilder class. From my perspective, the
> > >> >> >> > two most difficult things with setting this project up were understanding
> > >> >> >> > that the "process" method of the class that implements "Processor" is the
> > >> >> >> > entry point and the DynamicConsumerRouteBuilder creates the second entry
> > >> >> >> > point (The 5th and last points). This made the project very, VERY hard to
> > >> >> >> > understand.
> > >> >> >> > In addition to simplicity of design, the mvn clean install of the web
> > >> >> >> > service project is much faster and small scale activity publishing is also
> > >> >> >> > faster (see my email about load testing). These are minor points though as
> > >> >> >> > compilation has no effect on deployment. OSGi does add the benefit of
> > >> >> >> > modularized programming which is valuable, though I think the added
> > >> >> >> > complexity of Camel merits moving the project away from this paradigm.
> > >> >> >> >
> > >> >> >>
> > >> >> >> I agree that the project is pretty difficult to understand ATM.  I think
> > >> >> >> what we need to do is think about what the responsibilities of the code are
> > >> >> >> and allow for different implementations that are not so tightly coupled as
> > >> >> >> they are now.  For instance, having worked with Storm to ingest millions of
> > >> >> >> activities a day, I personally would like to see streams be responsible for
> > >> >> >> defining an over-arching orchestration model that can be implemented within
> > >> >> >> a single war or on top of a distributed system.  This would look something
> > >> >> >> like the follows:
> > >> >> >>
> > >> >> >> |__ Streams-Core  (Base classes, interfaces, utilities, extensions, etc)
> > >> >> >> |
> > >> >> >> |__ Streams-Storm (Storm implementation of the core)
> > >> >> >> |
> > >> >> >> |__ Streams-Camel (Camel implementation of the core)
> > >> >> >> |
> > >> >> >> |__ Streams-WS (Web service implementation)
> > >> >> >>
> > >> >> >>
> > >> >> >> > Danny
> > >> >> >> >
> > >> >> >> >
> > >> >> >> > > From: dsullivan7@hotmail.com
> > >> >> >> > > To: dev@streams.incubator.apache.org
> > >> >> >> > > Subject: [DISCUSS] Switching Streams from Camel deployment to .war
> > >> >> >> > deployment
> > >> >> >> > > Date: Tue, 29 Oct 2013 11:07:39 -0400
> > >> >> >> > >
> > >> >> >> > > The discussion thread for switching Streams from Camel/osgi/Servicemix
> > >> >> >> > to a single .war deployment
> > >> >> >> >
> > >> >> >> >
> > >> >> >
> > >> >
> > >
>  		 	   		  
 		 	   		  

RE: [DISCUSS] Switching Streams from Camel deployment to .war deployment

[Danny Sullivan <ds...@hotmail.com>]

Excellent, I'll write up something as a proof of concept and we can discuss further to make sure everything is vanilla.

> Date: Wed, 30 Oct 2013 17:01:28 -0400
> Subject: Re: [DISCUSS] Switching Streams from Camel deployment to .war deployment
> From: jletourneau80@gmail.com
> To: dev@streams.incubator.apache.org
> 
> That sounds pretty promising to me.
> 
> On Wed, Oct 30, 2013 at 2:38 PM, Danny Sullivan <ds...@hotmail.com> wrote:
> > Thanks for the feedback. You have an interesting point about the url linking to a separate processing space. Let me tie my answer into your last question about "advocating for the simplicity at registration to give up flexibility at registration, but retaining the inner "guts" of EIP/messaging". Consider a new architecture:
> >
> > streams-web.war: single entry point to application, but functions ONLY as an entry point. From here Camel routes the incoming requests to 4 separate jarssubscriber-registration.jar: subscriber registration publisher-registration.jar: publisher registrationactivity.jar: returns activity (also contains subscriber warehouse and storm activity aggregator)publish.jar: publishes activitystreams-cassandra.jar: the above 4 jars would all have a hook into this jar which would function as a hook onto the database. Each jar would have camel route output to this jar.
> >
> > In this implementation, Camel would no longer be the entry and exit point of a client to the application, but would handle the communication between components. The flow of activity through the application would be method based in each jar. This would allow deployment on up to 6 different process spaces. However, this does not address that there is a single server entry point, but I'm not sure if it was a concern in the first place.
> >
> > My argument, at its basis, is that we should move away from using Camel as the entry point to the application. I would be happy to maintain messaging between components.
> >
> >> Date: Wed, 30 Oct 2013 12:32:55 -0400
> >> Subject: Re: [DISCUSS] Switching Streams from Camel deployment to .war deployment
> >> From: jletourneau80@gmail.com
> >> To: dev@streams.incubator.apache.org
> >>
> >> An interesting use case that I am holding onto is the ability for
> >> publishers to register via a single URL (registration endpoint), but
> >> be sent a URL back to post to a different process space for actual
> >> publishing.  The same is true on the subscriber front.  Currently, the
> >> Camel/EIP infrastructure abstracts this because different components
> >> deployed in different process spaces handling the route creation can
> >> just be bolted onto a running Streams instance without new subs/pubs
> >> behaving any differently than existing.  This seems to be a
> >> potentially critical scaling point.  Is there a way to do this with
> >> the Spring solution?
> >>
> >> The persistence point is a good one, though I would classify that as
> >> "not implemented" vs "not possible" (not that you were).
> >>
> >> I'm not married to Camel, I just like the EIP approach to building
> >> something that is ultimately a messaging system.  There are known
> >> patterns that solve at least a subset of the problems Streams is
> >> trying to solve and implementations that can handle the load and I'll
> >> reiterate flexibility == complexity almost always.
> >>
> >> It comes right back to the central question: Do you want flexibility
> >> or simplicity?  It doesn't have to be black and white either I don't
> >> think...
> >>
> >> More pointedly: Where should we give up flexibility for simplicity?  I
> >> read that Danny is advocating for the simplicity at registration to
> >> give up flexibility at registration, but retaining the inner "guts" of
> >> EIP/messaging?  Thoughts?
> >>
> >> On Wed, Oct 30, 2013 at 11:37 AM, Danny Sullivan <ds...@hotmail.com> wrote:
> >> > My argument is not for the IoC pattern as that can be (and has been) implemented alongside Camel. My main argument is that the syntax at the entry point is not only familiar but much simpler. This wouldn't be a very strong argument if the Camel implementation wasn't much more complicated but I feel that it is the case. Also, looking toward the future, if the server is restarted, in-routes are lost in Camel. The way to curb this is to persist the dynamic routes that Camel creates, and then on start up pull every one of these routes and recreate a dynamic route for each one. Not only is this much easier to implement using the Spring web implementation, but it already has been implemented and you can try it by checking out the webservice branch, registering a subscriber, restarting tomcat, and using the same url you had before. This will allow subscribers to hang on to their urls once they register. (the same is true for publishers: you can post via the same url after restarting tomcat)
> >> >
> >> >> Date: Wed, 30 Oct 2013 11:00:21 -0400
> >> >> Subject: Re: [DISCUSS] Switching Streams from Camel deployment to .war deployment
> >> >> From: jletourneau80@gmail.com
> >> >> To: dev@streams.incubator.apache.org
> >> >>
> >> >> To be fair, while the current implementation is heavily camel-based,
> >> >> all of the interfaces related to Streams functionality are not.  The
> >> >> current model maps to what Matt has outlined in my opinion, though
> >> >> packing names etc. probably don't follow that exact pattern.
> >> >>
> >> >> With regards to the complexity and different components in the
> >> >> registration process, this was a cut at the abstraction based on the
> >> >> assumption that different implementations may be plugged in and in
> >> >> fact may live on different processor space (ie. a polling publisher vs
> >> >> a push publisher may be instantiated on different servers but the
> >> >> registration URL is staticly defined).
> >> >>
> >> >> Is the main argument I am seeing for Spring  the familiarity of its
> >> >> IoC pattern implementation and syntax at the entry point?
> >> >>
> >> >> On Wed, Oct 30, 2013 at 10:53 AM, Danny Sullivan <ds...@hotmail.com> wrote:
> >> >> > Could you clarify whether the same entry points would exist for the camel implementation of the core (implementing the "process" method/ using a DynammicRouteBuilder) or would the webservice be the sole entry point to Streams and after it enters would it hand it off to Camel? And what would be the entry point for the Storm implementation?
> >> >> > -Danny
> >> >> >
> >> >> >> Date: Wed, 30 Oct 2013 10:13:04 -0400
> >> >> >> Subject: Re: [DISCUSS] Switching Streams from Camel deployment to .war deployment
> >> >> >> From: m.ben.franklin@gmail.com
> >> >> >> To: dev@streams.incubator.apache.org
> >> >> >>
> >> >> >> On Tue, Oct 29, 2013 at 2:55 PM, Danny Sullivan <ds...@hotmail.com>wrote:
> >> >> >>
> >> >> >> > My case for switching from OSGi is for simplicity in design. To follow the
> >> >> >> > path of an activity through Streams in the webservice, there is one main
> >> >> >> > things the developer needs to understand:
> >> >> >> > The @RequestMapping annotation specifies the HTTP entry point
> >> >> >> > There are 4 @RequestMapping annotations that correspond to each of the 4
> >> >> >> > ways a user enters the application: registering a publisher, registering a
> >> >> >> > subscriber, publishing activity, and getting an activity stream. Where
> >> >> >> > these are located in the source code can be found by searching for the
> >> >> >> > paths specified in the documentation (search for "/publisherRegister",
> >> >> >> > "/subscriberRegister", "/publishActivity", "/getActivity" which will all
> >> >> >> > lead you to StreamsWebController.java). From the methods that process
> >> >> >> > requests, the flow through the application is through methods which can be
> >> >> >> > understood by most Java programmers.
> >> >> >> > The flow of activities through the current trunk branch is understood as
> >> >> >> > follows:
> >> >> >> > The string "/publisher/register" (the entry point to register a publisher
> >> >> >> > specified in the documentation) is the value of the
> >> >> >> > consumer.registrationEndpoint property defined in streams.propertiesThe
> >> >> >> > camelContext.xml specifies an endpoint with the id
> >> >> >> > consumerRegistrationEndpoint to have a uri equal to the propertyThe
> >> >> >> > consumerRegistrationEndpoint routes from uri direct:publisher register with
> >> >> >> > the bean activityRegistrationProcessor nested in between the routeThe
> >> >> >> > streams-eip-applicationContext contains a bean with the id
> >> >> >> > activityRegistrationProcessor created for the class
> >> >> >> > ActivityPublisherRegistrationProcessorThe exchange will enter the "process"
> >> >> >> > method of the class ActivityPublisherRegisitrationProcessor and that this
> >> >> >> > is because this class implements the "Processor" interface provided by
> >> >> >> > CamelThe direct:add-publisher-route takes the exchange output from the
> >> >> >> > "process" method and routes it to the activityRegistrationProcessor
> >> >> >> > "register" method. The bean activityRegistrationProcessor is defined in the
> >> >> >> > streams-eip-osgi-component-import.xmlThe output from this method is then
> >> >> >> > sent to the "createNewRouteForConsumer" method of activityConsumerRouter.
> >> >> >> > This method creates a new route for the newly registered publisher using
> >> >> >> > the private static final class DynamicConsumerRouteBuilder which is
> >> >> >> > required to extend RouteBuilder which is provided by Camel. This
> >> >> >> > DynamicConsumerRouteBuilder contains several methods:
> >> >> >> > getConsumerReceiveMethod() (which corresponds to @Value
> >> >> >> > ${consumer.receiveMethod} which corresponds to "receive"),
> >> >> >> > getConsumerSplitMethod() (@Value ${consumer.splitMethod},"split"), and
> >> >> >> > getConsumerActivityQUri() (@Value ${consumerActivityQUri},
> >> >> >> > "direct:activityQ"). This is different than the camelContext.xml in that
> >> >> >> > the route is being created programmatically. What this is doing is routing
> >> >> >> > input from the inroute url (which Camel does automatically through the
> >> >> >> > configure method which is required to be overridden), to the "receive"
> >> >> >> > method of ActivityConsumer, then to the "split" method of ActivityConsumer,
> >> >> >> > and then to the "direct:activityQ" which if you look back in the
> >> >> >> > camelContext.xml routes to the "activemq:queue:activities" which then
> >> >> >> > routes to "receiveExchange"
> >> >> >> > This is the process to register a publisher. The process for registering a
> >> >> >> > subscriber is relatively the same though it involves separate classes with
> >> >> >> > their own private static final RouteBuilder class. From my perspective, the
> >> >> >> > two most difficult things with setting this project up were understanding
> >> >> >> > that the "process" method of the class that implements "Processor" is the
> >> >> >> > entry point and the DynamicConsumerRouteBuilder creates the second entry
> >> >> >> > point (The 5th and last points). This made the project very, VERY hard to
> >> >> >> > understand.
> >> >> >> > In addition to simplicity of design, the mvn clean install of the web
> >> >> >> > service project is much faster and small scale activity publishing is also
> >> >> >> > faster (see my email about load testing). These are minor points though as
> >> >> >> > compilation has no effect on deployment. OSGi does add the benefit of
> >> >> >> > modularized programming which is valuable, though I think the added
> >> >> >> > complexity of Camel merits moving the project away from this paradigm.
> >> >> >> >
> >> >> >>
> >> >> >> I agree that the project is pretty difficult to understand ATM.  I think
> >> >> >> what we need to do is think about what the responsibilities of the code are
> >> >> >> and allow for different implementations that are not so tightly coupled as
> >> >> >> they are now.  For instance, having worked with Storm to ingest millions of
> >> >> >> activities a day, I personally would like to see streams be responsible for
> >> >> >> defining an over-arching orchestration model that can be implemented within
> >> >> >> a single war or on top of a distributed system.  This would look something
> >> >> >> like the follows:
> >> >> >>
> >> >> >> |__ Streams-Core  (Base classes, interfaces, utilities, extensions, etc)
> >> >> >> |
> >> >> >> |__ Streams-Storm (Storm implementation of the core)
> >> >> >> |
> >> >> >> |__ Streams-Camel (Camel implementation of the core)
> >> >> >> |
> >> >> >> |__ Streams-WS (Web service implementation)
> >> >> >>
> >> >> >>
> >> >> >> > Danny
> >> >> >> >
> >> >> >> >
> >> >> >> > > From: dsullivan7@hotmail.com
> >> >> >> > > To: dev@streams.incubator.apache.org
> >> >> >> > > Subject: [DISCUSS] Switching Streams from Camel deployment to .war
> >> >> >> > deployment
> >> >> >> > > Date: Tue, 29 Oct 2013 11:07:39 -0400
> >> >> >> > >
> >> >> >> > > The discussion thread for switching Streams from Camel/osgi/Servicemix
> >> >> >> > to a single .war deployment
> >> >> >> >
> >> >> >> >
> >> >> >
> >> >
> >
 		 	   		  

Re: [DISCUSS] Switching Streams from Camel deployment to .war deployment

[Jason Letourneau <jl...@gmail.com>]

That sounds pretty promising to me.

On Wed, Oct 30, 2013 at 2:38 PM, Danny Sullivan <ds...@hotmail.com> wrote:
> Thanks for the feedback. You have an interesting point about the url linking to a separate processing space. Let me tie my answer into your last question about "advocating for the simplicity at registration to give up flexibility at registration, but retaining the inner "guts" of EIP/messaging". Consider a new architecture:
>
> streams-web.war: single entry point to application, but functions ONLY as an entry point. From here Camel routes the incoming requests to 4 separate jarssubscriber-registration.jar: subscriber registration publisher-registration.jar: publisher registrationactivity.jar: returns activity (also contains subscriber warehouse and storm activity aggregator)publish.jar: publishes activitystreams-cassandra.jar: the above 4 jars would all have a hook into this jar which would function as a hook onto the database. Each jar would have camel route output to this jar.
>
> In this implementation, Camel would no longer be the entry and exit point of a client to the application, but would handle the communication between components. The flow of activity through the application would be method based in each jar. This would allow deployment on up to 6 different process spaces. However, this does not address that there is a single server entry point, but I'm not sure if it was a concern in the first place.
>
> My argument, at its basis, is that we should move away from using Camel as the entry point to the application. I would be happy to maintain messaging between components.
>
>> Date: Wed, 30 Oct 2013 12:32:55 -0400
>> Subject: Re: [DISCUSS] Switching Streams from Camel deployment to .war deployment
>> From: jletourneau80@gmail.com
>> To: dev@streams.incubator.apache.org
>>
>> An interesting use case that I am holding onto is the ability for
>> publishers to register via a single URL (registration endpoint), but
>> be sent a URL back to post to a different process space for actual
>> publishing.  The same is true on the subscriber front.  Currently, the
>> Camel/EIP infrastructure abstracts this because different components
>> deployed in different process spaces handling the route creation can
>> just be bolted onto a running Streams instance without new subs/pubs
>> behaving any differently than existing.  This seems to be a
>> potentially critical scaling point.  Is there a way to do this with
>> the Spring solution?
>>
>> The persistence point is a good one, though I would classify that as
>> "not implemented" vs "not possible" (not that you were).
>>
>> I'm not married to Camel, I just like the EIP approach to building
>> something that is ultimately a messaging system.  There are known
>> patterns that solve at least a subset of the problems Streams is
>> trying to solve and implementations that can handle the load and I'll
>> reiterate flexibility == complexity almost always.
>>
>> It comes right back to the central question: Do you want flexibility
>> or simplicity?  It doesn't have to be black and white either I don't
>> think...
>>
>> More pointedly: Where should we give up flexibility for simplicity?  I
>> read that Danny is advocating for the simplicity at registration to
>> give up flexibility at registration, but retaining the inner "guts" of
>> EIP/messaging?  Thoughts?
>>
>> On Wed, Oct 30, 2013 at 11:37 AM, Danny Sullivan <ds...@hotmail.com> wrote:
>> > My argument is not for the IoC pattern as that can be (and has been) implemented alongside Camel. My main argument is that the syntax at the entry point is not only familiar but much simpler. This wouldn't be a very strong argument if the Camel implementation wasn't much more complicated but I feel that it is the case. Also, looking toward the future, if the server is restarted, in-routes are lost in Camel. The way to curb this is to persist the dynamic routes that Camel creates, and then on start up pull every one of these routes and recreate a dynamic route for each one. Not only is this much easier to implement using the Spring web implementation, but it already has been implemented and you can try it by checking out the webservice branch, registering a subscriber, restarting tomcat, and using the same url you had before. This will allow subscribers to hang on to their urls once they register. (the same is true for publishers: you can post via the same url after restarting tomcat)
>> >
>> >> Date: Wed, 30 Oct 2013 11:00:21 -0400
>> >> Subject: Re: [DISCUSS] Switching Streams from Camel deployment to .war deployment
>> >> From: jletourneau80@gmail.com
>> >> To: dev@streams.incubator.apache.org
>> >>
>> >> To be fair, while the current implementation is heavily camel-based,
>> >> all of the interfaces related to Streams functionality are not.  The
>> >> current model maps to what Matt has outlined in my opinion, though
>> >> packing names etc. probably don't follow that exact pattern.
>> >>
>> >> With regards to the complexity and different components in the
>> >> registration process, this was a cut at the abstraction based on the
>> >> assumption that different implementations may be plugged in and in
>> >> fact may live on different processor space (ie. a polling publisher vs
>> >> a push publisher may be instantiated on different servers but the
>> >> registration URL is staticly defined).
>> >>
>> >> Is the main argument I am seeing for Spring  the familiarity of its
>> >> IoC pattern implementation and syntax at the entry point?
>> >>
>> >> On Wed, Oct 30, 2013 at 10:53 AM, Danny Sullivan <ds...@hotmail.com> wrote:
>> >> > Could you clarify whether the same entry points would exist for the camel implementation of the core (implementing the "process" method/ using a DynammicRouteBuilder) or would the webservice be the sole entry point to Streams and after it enters would it hand it off to Camel? And what would be the entry point for the Storm implementation?
>> >> > -Danny
>> >> >
>> >> >> Date: Wed, 30 Oct 2013 10:13:04 -0400
>> >> >> Subject: Re: [DISCUSS] Switching Streams from Camel deployment to .war deployment
>> >> >> From: m.ben.franklin@gmail.com
>> >> >> To: dev@streams.incubator.apache.org
>> >> >>
>> >> >> On Tue, Oct 29, 2013 at 2:55 PM, Danny Sullivan <ds...@hotmail.com>wrote:
>> >> >>
>> >> >> > My case for switching from OSGi is for simplicity in design. To follow the
>> >> >> > path of an activity through Streams in the webservice, there is one main
>> >> >> > things the developer needs to understand:
>> >> >> > The @RequestMapping annotation specifies the HTTP entry point
>> >> >> > There are 4 @RequestMapping annotations that correspond to each of the 4
>> >> >> > ways a user enters the application: registering a publisher, registering a
>> >> >> > subscriber, publishing activity, and getting an activity stream. Where
>> >> >> > these are located in the source code can be found by searching for the
>> >> >> > paths specified in the documentation (search for "/publisherRegister",
>> >> >> > "/subscriberRegister", "/publishActivity", "/getActivity" which will all
>> >> >> > lead you to StreamsWebController.java). From the methods that process
>> >> >> > requests, the flow through the application is through methods which can be
>> >> >> > understood by most Java programmers.
>> >> >> > The flow of activities through the current trunk branch is understood as
>> >> >> > follows:
>> >> >> > The string "/publisher/register" (the entry point to register a publisher
>> >> >> > specified in the documentation) is the value of the
>> >> >> > consumer.registrationEndpoint property defined in streams.propertiesThe
>> >> >> > camelContext.xml specifies an endpoint with the id
>> >> >> > consumerRegistrationEndpoint to have a uri equal to the propertyThe
>> >> >> > consumerRegistrationEndpoint routes from uri direct:publisher register with
>> >> >> > the bean activityRegistrationProcessor nested in between the routeThe
>> >> >> > streams-eip-applicationContext contains a bean with the id
>> >> >> > activityRegistrationProcessor created for the class
>> >> >> > ActivityPublisherRegistrationProcessorThe exchange will enter the "process"
>> >> >> > method of the class ActivityPublisherRegisitrationProcessor and that this
>> >> >> > is because this class implements the "Processor" interface provided by
>> >> >> > CamelThe direct:add-publisher-route takes the exchange output from the
>> >> >> > "process" method and routes it to the activityRegistrationProcessor
>> >> >> > "register" method. The bean activityRegistrationProcessor is defined in the
>> >> >> > streams-eip-osgi-component-import.xmlThe output from this method is then
>> >> >> > sent to the "createNewRouteForConsumer" method of activityConsumerRouter.
>> >> >> > This method creates a new route for the newly registered publisher using
>> >> >> > the private static final class DynamicConsumerRouteBuilder which is
>> >> >> > required to extend RouteBuilder which is provided by Camel. This
>> >> >> > DynamicConsumerRouteBuilder contains several methods:
>> >> >> > getConsumerReceiveMethod() (which corresponds to @Value
>> >> >> > ${consumer.receiveMethod} which corresponds to "receive"),
>> >> >> > getConsumerSplitMethod() (@Value ${consumer.splitMethod},"split"), and
>> >> >> > getConsumerActivityQUri() (@Value ${consumerActivityQUri},
>> >> >> > "direct:activityQ"). This is different than the camelContext.xml in that
>> >> >> > the route is being created programmatically. What this is doing is routing
>> >> >> > input from the inroute url (which Camel does automatically through the
>> >> >> > configure method which is required to be overridden), to the "receive"
>> >> >> > method of ActivityConsumer, then to the "split" method of ActivityConsumer,
>> >> >> > and then to the "direct:activityQ" which if you look back in the
>> >> >> > camelContext.xml routes to the "activemq:queue:activities" which then
>> >> >> > routes to "receiveExchange"
>> >> >> > This is the process to register a publisher. The process for registering a
>> >> >> > subscriber is relatively the same though it involves separate classes with
>> >> >> > their own private static final RouteBuilder class. From my perspective, the
>> >> >> > two most difficult things with setting this project up were understanding
>> >> >> > that the "process" method of the class that implements "Processor" is the
>> >> >> > entry point and the DynamicConsumerRouteBuilder creates the second entry
>> >> >> > point (The 5th and last points). This made the project very, VERY hard to
>> >> >> > understand.
>> >> >> > In addition to simplicity of design, the mvn clean install of the web
>> >> >> > service project is much faster and small scale activity publishing is also
>> >> >> > faster (see my email about load testing). These are minor points though as
>> >> >> > compilation has no effect on deployment. OSGi does add the benefit of
>> >> >> > modularized programming which is valuable, though I think the added
>> >> >> > complexity of Camel merits moving the project away from this paradigm.
>> >> >> >
>> >> >>
>> >> >> I agree that the project is pretty difficult to understand ATM.  I think
>> >> >> what we need to do is think about what the responsibilities of the code are
>> >> >> and allow for different implementations that are not so tightly coupled as
>> >> >> they are now.  For instance, having worked with Storm to ingest millions of
>> >> >> activities a day, I personally would like to see streams be responsible for
>> >> >> defining an over-arching orchestration model that can be implemented within
>> >> >> a single war or on top of a distributed system.  This would look something
>> >> >> like the follows:
>> >> >>
>> >> >> |__ Streams-Core  (Base classes, interfaces, utilities, extensions, etc)
>> >> >> |
>> >> >> |__ Streams-Storm (Storm implementation of the core)
>> >> >> |
>> >> >> |__ Streams-Camel (Camel implementation of the core)
>> >> >> |
>> >> >> |__ Streams-WS (Web service implementation)
>> >> >>
>> >> >>
>> >> >> > Danny
>> >> >> >
>> >> >> >
>> >> >> > > From: dsullivan7@hotmail.com
>> >> >> > > To: dev@streams.incubator.apache.org
>> >> >> > > Subject: [DISCUSS] Switching Streams from Camel deployment to .war
>> >> >> > deployment
>> >> >> > > Date: Tue, 29 Oct 2013 11:07:39 -0400
>> >> >> > >
>> >> >> > > The discussion thread for switching Streams from Camel/osgi/Servicemix
>> >> >> > to a single .war deployment
>> >> >> >
>> >> >> >
>> >> >
>> >
>

RE: [DISCUSS] Switching Streams from Camel deployment to .war deployment

[Danny Sullivan <ds...@hotmail.com>]

Thanks for the feedback. You have an interesting point about the url linking to a separate processing space. Let me tie my answer into your last question about "advocating for the simplicity at registration to give up flexibility at registration, but retaining the inner "guts" of EIP/messaging". Consider a new architecture:

streams-web.war: single entry point to application, but functions ONLY as an entry point. From here Camel routes the incoming requests to 4 separate jarssubscriber-registration.jar: subscriber registration publisher-registration.jar: publisher registrationactivity.jar: returns activity (also contains subscriber warehouse and storm activity aggregator)publish.jar: publishes activitystreams-cassandra.jar: the above 4 jars would all have a hook into this jar which would function as a hook onto the database. Each jar would have camel route output to this jar.

In this implementation, Camel would no longer be the entry and exit point of a client to the application, but would handle the communication between components. The flow of activity through the application would be method based in each jar. This would allow deployment on up to 6 different process spaces. However, this does not address that there is a single server entry point, but I'm not sure if it was a concern in the first place.

My argument, at its basis, is that we should move away from using Camel as the entry point to the application. I would be happy to maintain messaging between components.

> Date: Wed, 30 Oct 2013 12:32:55 -0400
> Subject: Re: [DISCUSS] Switching Streams from Camel deployment to .war deployment
> From: jletourneau80@gmail.com
> To: dev@streams.incubator.apache.org
> 
> An interesting use case that I am holding onto is the ability for
> publishers to register via a single URL (registration endpoint), but
> be sent a URL back to post to a different process space for actual
> publishing.  The same is true on the subscriber front.  Currently, the
> Camel/EIP infrastructure abstracts this because different components
> deployed in different process spaces handling the route creation can
> just be bolted onto a running Streams instance without new subs/pubs
> behaving any differently than existing.  This seems to be a
> potentially critical scaling point.  Is there a way to do this with
> the Spring solution?
> 
> The persistence point is a good one, though I would classify that as
> "not implemented" vs "not possible" (not that you were).
> 
> I'm not married to Camel, I just like the EIP approach to building
> something that is ultimately a messaging system.  There are known
> patterns that solve at least a subset of the problems Streams is
> trying to solve and implementations that can handle the load and I'll
> reiterate flexibility == complexity almost always.
> 
> It comes right back to the central question: Do you want flexibility
> or simplicity?  It doesn't have to be black and white either I don't
> think...
> 
> More pointedly: Where should we give up flexibility for simplicity?  I
> read that Danny is advocating for the simplicity at registration to
> give up flexibility at registration, but retaining the inner "guts" of
> EIP/messaging?  Thoughts?
> 
> On Wed, Oct 30, 2013 at 11:37 AM, Danny Sullivan <ds...@hotmail.com> wrote:
> > My argument is not for the IoC pattern as that can be (and has been) implemented alongside Camel. My main argument is that the syntax at the entry point is not only familiar but much simpler. This wouldn't be a very strong argument if the Camel implementation wasn't much more complicated but I feel that it is the case. Also, looking toward the future, if the server is restarted, in-routes are lost in Camel. The way to curb this is to persist the dynamic routes that Camel creates, and then on start up pull every one of these routes and recreate a dynamic route for each one. Not only is this much easier to implement using the Spring web implementation, but it already has been implemented and you can try it by checking out the webservice branch, registering a subscriber, restarting tomcat, and using the same url you had before. This will allow subscribers to hang on to their urls once they register. (the same is true for publishers: you can post via the same url after restarting tomcat)
> >
> >> Date: Wed, 30 Oct 2013 11:00:21 -0400
> >> Subject: Re: [DISCUSS] Switching Streams from Camel deployment to .war deployment
> >> From: jletourneau80@gmail.com
> >> To: dev@streams.incubator.apache.org
> >>
> >> To be fair, while the current implementation is heavily camel-based,
> >> all of the interfaces related to Streams functionality are not.  The
> >> current model maps to what Matt has outlined in my opinion, though
> >> packing names etc. probably don't follow that exact pattern.
> >>
> >> With regards to the complexity and different components in the
> >> registration process, this was a cut at the abstraction based on the
> >> assumption that different implementations may be plugged in and in
> >> fact may live on different processor space (ie. a polling publisher vs
> >> a push publisher may be instantiated on different servers but the
> >> registration URL is staticly defined).
> >>
> >> Is the main argument I am seeing for Spring  the familiarity of its
> >> IoC pattern implementation and syntax at the entry point?
> >>
> >> On Wed, Oct 30, 2013 at 10:53 AM, Danny Sullivan <ds...@hotmail.com> wrote:
> >> > Could you clarify whether the same entry points would exist for the camel implementation of the core (implementing the "process" method/ using a DynammicRouteBuilder) or would the webservice be the sole entry point to Streams and after it enters would it hand it off to Camel? And what would be the entry point for the Storm implementation?
> >> > -Danny
> >> >
> >> >> Date: Wed, 30 Oct 2013 10:13:04 -0400
> >> >> Subject: Re: [DISCUSS] Switching Streams from Camel deployment to .war deployment
> >> >> From: m.ben.franklin@gmail.com
> >> >> To: dev@streams.incubator.apache.org
> >> >>
> >> >> On Tue, Oct 29, 2013 at 2:55 PM, Danny Sullivan <ds...@hotmail.com>wrote:
> >> >>
> >> >> > My case for switching from OSGi is for simplicity in design. To follow the
> >> >> > path of an activity through Streams in the webservice, there is one main
> >> >> > things the developer needs to understand:
> >> >> > The @RequestMapping annotation specifies the HTTP entry point
> >> >> > There are 4 @RequestMapping annotations that correspond to each of the 4
> >> >> > ways a user enters the application: registering a publisher, registering a
> >> >> > subscriber, publishing activity, and getting an activity stream. Where
> >> >> > these are located in the source code can be found by searching for the
> >> >> > paths specified in the documentation (search for "/publisherRegister",
> >> >> > "/subscriberRegister", "/publishActivity", "/getActivity" which will all
> >> >> > lead you to StreamsWebController.java). From the methods that process
> >> >> > requests, the flow through the application is through methods which can be
> >> >> > understood by most Java programmers.
> >> >> > The flow of activities through the current trunk branch is understood as
> >> >> > follows:
> >> >> > The string "/publisher/register" (the entry point to register a publisher
> >> >> > specified in the documentation) is the value of the
> >> >> > consumer.registrationEndpoint property defined in streams.propertiesThe
> >> >> > camelContext.xml specifies an endpoint with the id
> >> >> > consumerRegistrationEndpoint to have a uri equal to the propertyThe
> >> >> > consumerRegistrationEndpoint routes from uri direct:publisher register with
> >> >> > the bean activityRegistrationProcessor nested in between the routeThe
> >> >> > streams-eip-applicationContext contains a bean with the id
> >> >> > activityRegistrationProcessor created for the class
> >> >> > ActivityPublisherRegistrationProcessorThe exchange will enter the "process"
> >> >> > method of the class ActivityPublisherRegisitrationProcessor and that this
> >> >> > is because this class implements the "Processor" interface provided by
> >> >> > CamelThe direct:add-publisher-route takes the exchange output from the
> >> >> > "process" method and routes it to the activityRegistrationProcessor
> >> >> > "register" method. The bean activityRegistrationProcessor is defined in the
> >> >> > streams-eip-osgi-component-import.xmlThe output from this method is then
> >> >> > sent to the "createNewRouteForConsumer" method of activityConsumerRouter.
> >> >> > This method creates a new route for the newly registered publisher using
> >> >> > the private static final class DynamicConsumerRouteBuilder which is
> >> >> > required to extend RouteBuilder which is provided by Camel. This
> >> >> > DynamicConsumerRouteBuilder contains several methods:
> >> >> > getConsumerReceiveMethod() (which corresponds to @Value
> >> >> > ${consumer.receiveMethod} which corresponds to "receive"),
> >> >> > getConsumerSplitMethod() (@Value ${consumer.splitMethod},"split"), and
> >> >> > getConsumerActivityQUri() (@Value ${consumerActivityQUri},
> >> >> > "direct:activityQ"). This is different than the camelContext.xml in that
> >> >> > the route is being created programmatically. What this is doing is routing
> >> >> > input from the inroute url (which Camel does automatically through the
> >> >> > configure method which is required to be overridden), to the "receive"
> >> >> > method of ActivityConsumer, then to the "split" method of ActivityConsumer,
> >> >> > and then to the "direct:activityQ" which if you look back in the
> >> >> > camelContext.xml routes to the "activemq:queue:activities" which then
> >> >> > routes to "receiveExchange"
> >> >> > This is the process to register a publisher. The process for registering a
> >> >> > subscriber is relatively the same though it involves separate classes with
> >> >> > their own private static final RouteBuilder class. From my perspective, the
> >> >> > two most difficult things with setting this project up were understanding
> >> >> > that the "process" method of the class that implements "Processor" is the
> >> >> > entry point and the DynamicConsumerRouteBuilder creates the second entry
> >> >> > point (The 5th and last points). This made the project very, VERY hard to
> >> >> > understand.
> >> >> > In addition to simplicity of design, the mvn clean install of the web
> >> >> > service project is much faster and small scale activity publishing is also
> >> >> > faster (see my email about load testing). These are minor points though as
> >> >> > compilation has no effect on deployment. OSGi does add the benefit of
> >> >> > modularized programming which is valuable, though I think the added
> >> >> > complexity of Camel merits moving the project away from this paradigm.
> >> >> >
> >> >>
> >> >> I agree that the project is pretty difficult to understand ATM.  I think
> >> >> what we need to do is think about what the responsibilities of the code are
> >> >> and allow for different implementations that are not so tightly coupled as
> >> >> they are now.  For instance, having worked with Storm to ingest millions of
> >> >> activities a day, I personally would like to see streams be responsible for
> >> >> defining an over-arching orchestration model that can be implemented within
> >> >> a single war or on top of a distributed system.  This would look something
> >> >> like the follows:
> >> >>
> >> >> |__ Streams-Core  (Base classes, interfaces, utilities, extensions, etc)
> >> >> |
> >> >> |__ Streams-Storm (Storm implementation of the core)
> >> >> |
> >> >> |__ Streams-Camel (Camel implementation of the core)
> >> >> |
> >> >> |__ Streams-WS (Web service implementation)
> >> >>
> >> >>
> >> >> > Danny
> >> >> >
> >> >> >
> >> >> > > From: dsullivan7@hotmail.com
> >> >> > > To: dev@streams.incubator.apache.org
> >> >> > > Subject: [DISCUSS] Switching Streams from Camel deployment to .war
> >> >> > deployment
> >> >> > > Date: Tue, 29 Oct 2013 11:07:39 -0400
> >> >> > >
> >> >> > > The discussion thread for switching Streams from Camel/osgi/Servicemix
> >> >> > to a single .war deployment
> >> >> >
> >> >> >
> >> >
> >
 		 	   		  

Re: [DISCUSS] Switching Streams from Camel deployment to .war deployment

[Jason Letourneau <jl...@gmail.com>]

An interesting use case that I am holding onto is the ability for
publishers to register via a single URL (registration endpoint), but
be sent a URL back to post to a different process space for actual
publishing.  The same is true on the subscriber front.  Currently, the
Camel/EIP infrastructure abstracts this because different components
deployed in different process spaces handling the route creation can
just be bolted onto a running Streams instance without new subs/pubs
behaving any differently than existing.  This seems to be a
potentially critical scaling point.  Is there a way to do this with
the Spring solution?

The persistence point is a good one, though I would classify that as
"not implemented" vs "not possible" (not that you were).

I'm not married to Camel, I just like the EIP approach to building
something that is ultimately a messaging system.  There are known
patterns that solve at least a subset of the problems Streams is
trying to solve and implementations that can handle the load and I'll
reiterate flexibility == complexity almost always.

It comes right back to the central question: Do you want flexibility
or simplicity?  It doesn't have to be black and white either I don't
think...

More pointedly: Where should we give up flexibility for simplicity?  I
read that Danny is advocating for the simplicity at registration to
give up flexibility at registration, but retaining the inner "guts" of
EIP/messaging?  Thoughts?

On Wed, Oct 30, 2013 at 11:37 AM, Danny Sullivan <ds...@hotmail.com> wrote:
> My argument is not for the IoC pattern as that can be (and has been) implemented alongside Camel. My main argument is that the syntax at the entry point is not only familiar but much simpler. This wouldn't be a very strong argument if the Camel implementation wasn't much more complicated but I feel that it is the case. Also, looking toward the future, if the server is restarted, in-routes are lost in Camel. The way to curb this is to persist the dynamic routes that Camel creates, and then on start up pull every one of these routes and recreate a dynamic route for each one. Not only is this much easier to implement using the Spring web implementation, but it already has been implemented and you can try it by checking out the webservice branch, registering a subscriber, restarting tomcat, and using the same url you had before. This will allow subscribers to hang on to their urls once they register. (the same is true for publishers: you can post via the same url after restarting tomcat)
>
>> Date: Wed, 30 Oct 2013 11:00:21 -0400
>> Subject: Re: [DISCUSS] Switching Streams from Camel deployment to .war deployment
>> From: jletourneau80@gmail.com
>> To: dev@streams.incubator.apache.org
>>
>> To be fair, while the current implementation is heavily camel-based,
>> all of the interfaces related to Streams functionality are not.  The
>> current model maps to what Matt has outlined in my opinion, though
>> packing names etc. probably don't follow that exact pattern.
>>
>> With regards to the complexity and different components in the
>> registration process, this was a cut at the abstraction based on the
>> assumption that different implementations may be plugged in and in
>> fact may live on different processor space (ie. a polling publisher vs
>> a push publisher may be instantiated on different servers but the
>> registration URL is staticly defined).
>>
>> Is the main argument I am seeing for Spring  the familiarity of its
>> IoC pattern implementation and syntax at the entry point?
>>
>> On Wed, Oct 30, 2013 at 10:53 AM, Danny Sullivan <ds...@hotmail.com> wrote:
>> > Could you clarify whether the same entry points would exist for the camel implementation of the core (implementing the "process" method/ using a DynammicRouteBuilder) or would the webservice be the sole entry point to Streams and after it enters would it hand it off to Camel? And what would be the entry point for the Storm implementation?
>> > -Danny
>> >
>> >> Date: Wed, 30 Oct 2013 10:13:04 -0400
>> >> Subject: Re: [DISCUSS] Switching Streams from Camel deployment to .war deployment
>> >> From: m.ben.franklin@gmail.com
>> >> To: dev@streams.incubator.apache.org
>> >>
>> >> On Tue, Oct 29, 2013 at 2:55 PM, Danny Sullivan <ds...@hotmail.com>wrote:
>> >>
>> >> > My case for switching from OSGi is for simplicity in design. To follow the
>> >> > path of an activity through Streams in the webservice, there is one main
>> >> > things the developer needs to understand:
>> >> > The @RequestMapping annotation specifies the HTTP entry point
>> >> > There are 4 @RequestMapping annotations that correspond to each of the 4
>> >> > ways a user enters the application: registering a publisher, registering a
>> >> > subscriber, publishing activity, and getting an activity stream. Where
>> >> > these are located in the source code can be found by searching for the
>> >> > paths specified in the documentation (search for "/publisherRegister",
>> >> > "/subscriberRegister", "/publishActivity", "/getActivity" which will all
>> >> > lead you to StreamsWebController.java). From the methods that process
>> >> > requests, the flow through the application is through methods which can be
>> >> > understood by most Java programmers.
>> >> > The flow of activities through the current trunk branch is understood as
>> >> > follows:
>> >> > The string "/publisher/register" (the entry point to register a publisher
>> >> > specified in the documentation) is the value of the
>> >> > consumer.registrationEndpoint property defined in streams.propertiesThe
>> >> > camelContext.xml specifies an endpoint with the id
>> >> > consumerRegistrationEndpoint to have a uri equal to the propertyThe
>> >> > consumerRegistrationEndpoint routes from uri direct:publisher register with
>> >> > the bean activityRegistrationProcessor nested in between the routeThe
>> >> > streams-eip-applicationContext contains a bean with the id
>> >> > activityRegistrationProcessor created for the class
>> >> > ActivityPublisherRegistrationProcessorThe exchange will enter the "process"
>> >> > method of the class ActivityPublisherRegisitrationProcessor and that this
>> >> > is because this class implements the "Processor" interface provided by
>> >> > CamelThe direct:add-publisher-route takes the exchange output from the
>> >> > "process" method and routes it to the activityRegistrationProcessor
>> >> > "register" method. The bean activityRegistrationProcessor is defined in the
>> >> > streams-eip-osgi-component-import.xmlThe output from this method is then
>> >> > sent to the "createNewRouteForConsumer" method of activityConsumerRouter.
>> >> > This method creates a new route for the newly registered publisher using
>> >> > the private static final class DynamicConsumerRouteBuilder which is
>> >> > required to extend RouteBuilder which is provided by Camel. This
>> >> > DynamicConsumerRouteBuilder contains several methods:
>> >> > getConsumerReceiveMethod() (which corresponds to @Value
>> >> > ${consumer.receiveMethod} which corresponds to "receive"),
>> >> > getConsumerSplitMethod() (@Value ${consumer.splitMethod},"split"), and
>> >> > getConsumerActivityQUri() (@Value ${consumerActivityQUri},
>> >> > "direct:activityQ"). This is different than the camelContext.xml in that
>> >> > the route is being created programmatically. What this is doing is routing
>> >> > input from the inroute url (which Camel does automatically through the
>> >> > configure method which is required to be overridden), to the "receive"
>> >> > method of ActivityConsumer, then to the "split" method of ActivityConsumer,
>> >> > and then to the "direct:activityQ" which if you look back in the
>> >> > camelContext.xml routes to the "activemq:queue:activities" which then
>> >> > routes to "receiveExchange"
>> >> > This is the process to register a publisher. The process for registering a
>> >> > subscriber is relatively the same though it involves separate classes with
>> >> > their own private static final RouteBuilder class. From my perspective, the
>> >> > two most difficult things with setting this project up were understanding
>> >> > that the "process" method of the class that implements "Processor" is the
>> >> > entry point and the DynamicConsumerRouteBuilder creates the second entry
>> >> > point (The 5th and last points). This made the project very, VERY hard to
>> >> > understand.
>> >> > In addition to simplicity of design, the mvn clean install of the web
>> >> > service project is much faster and small scale activity publishing is also
>> >> > faster (see my email about load testing). These are minor points though as
>> >> > compilation has no effect on deployment. OSGi does add the benefit of
>> >> > modularized programming which is valuable, though I think the added
>> >> > complexity of Camel merits moving the project away from this paradigm.
>> >> >
>> >>
>> >> I agree that the project is pretty difficult to understand ATM.  I think
>> >> what we need to do is think about what the responsibilities of the code are
>> >> and allow for different implementations that are not so tightly coupled as
>> >> they are now.  For instance, having worked with Storm to ingest millions of
>> >> activities a day, I personally would like to see streams be responsible for
>> >> defining an over-arching orchestration model that can be implemented within
>> >> a single war or on top of a distributed system.  This would look something
>> >> like the follows:
>> >>
>> >> |__ Streams-Core  (Base classes, interfaces, utilities, extensions, etc)
>> >> |
>> >> |__ Streams-Storm (Storm implementation of the core)
>> >> |
>> >> |__ Streams-Camel (Camel implementation of the core)
>> >> |
>> >> |__ Streams-WS (Web service implementation)
>> >>
>> >>
>> >> > Danny
>> >> >
>> >> >
>> >> > > From: dsullivan7@hotmail.com
>> >> > > To: dev@streams.incubator.apache.org
>> >> > > Subject: [DISCUSS] Switching Streams from Camel deployment to .war
>> >> > deployment
>> >> > > Date: Tue, 29 Oct 2013 11:07:39 -0400
>> >> > >
>> >> > > The discussion thread for switching Streams from Camel/osgi/Servicemix
>> >> > to a single .war deployment
>> >> >
>> >> >
>> >
>

RE: [DISCUSS] Switching Streams from Camel deployment to .war deployment

[Danny Sullivan <ds...@hotmail.com>]

My argument is not for the IoC pattern as that can be (and has been) implemented alongside Camel. My main argument is that the syntax at the entry point is not only familiar but much simpler. This wouldn't be a very strong argument if the Camel implementation wasn't much more complicated but I feel that it is the case. Also, looking toward the future, if the server is restarted, in-routes are lost in Camel. The way to curb this is to persist the dynamic routes that Camel creates, and then on start up pull every one of these routes and recreate a dynamic route for each one. Not only is this much easier to implement using the Spring web implementation, but it already has been implemented and you can try it by checking out the webservice branch, registering a subscriber, restarting tomcat, and using the same url you had before. This will allow subscribers to hang on to their urls once they register. (the same is true for publishers: you can post via the same url after restarting tomcat)     

> Date: Wed, 30 Oct 2013 11:00:21 -0400
> Subject: Re: [DISCUSS] Switching Streams from Camel deployment to .war deployment
> From: jletourneau80@gmail.com
> To: dev@streams.incubator.apache.org
> 
> To be fair, while the current implementation is heavily camel-based,
> all of the interfaces related to Streams functionality are not.  The
> current model maps to what Matt has outlined in my opinion, though
> packing names etc. probably don't follow that exact pattern.
> 
> With regards to the complexity and different components in the
> registration process, this was a cut at the abstraction based on the
> assumption that different implementations may be plugged in and in
> fact may live on different processor space (ie. a polling publisher vs
> a push publisher may be instantiated on different servers but the
> registration URL is staticly defined).
> 
> Is the main argument I am seeing for Spring  the familiarity of its
> IoC pattern implementation and syntax at the entry point?
> 
> On Wed, Oct 30, 2013 at 10:53 AM, Danny Sullivan <ds...@hotmail.com> wrote:
> > Could you clarify whether the same entry points would exist for the camel implementation of the core (implementing the "process" method/ using a DynammicRouteBuilder) or would the webservice be the sole entry point to Streams and after it enters would it hand it off to Camel? And what would be the entry point for the Storm implementation?
> > -Danny
> >
> >> Date: Wed, 30 Oct 2013 10:13:04 -0400
> >> Subject: Re: [DISCUSS] Switching Streams from Camel deployment to .war deployment
> >> From: m.ben.franklin@gmail.com
> >> To: dev@streams.incubator.apache.org
> >>
> >> On Tue, Oct 29, 2013 at 2:55 PM, Danny Sullivan <ds...@hotmail.com>wrote:
> >>
> >> > My case for switching from OSGi is for simplicity in design. To follow the
> >> > path of an activity through Streams in the webservice, there is one main
> >> > things the developer needs to understand:
> >> > The @RequestMapping annotation specifies the HTTP entry point
> >> > There are 4 @RequestMapping annotations that correspond to each of the 4
> >> > ways a user enters the application: registering a publisher, registering a
> >> > subscriber, publishing activity, and getting an activity stream. Where
> >> > these are located in the source code can be found by searching for the
> >> > paths specified in the documentation (search for "/publisherRegister",
> >> > "/subscriberRegister", "/publishActivity", "/getActivity" which will all
> >> > lead you to StreamsWebController.java). From the methods that process
> >> > requests, the flow through the application is through methods which can be
> >> > understood by most Java programmers.
> >> > The flow of activities through the current trunk branch is understood as
> >> > follows:
> >> > The string "/publisher/register" (the entry point to register a publisher
> >> > specified in the documentation) is the value of the
> >> > consumer.registrationEndpoint property defined in streams.propertiesThe
> >> > camelContext.xml specifies an endpoint with the id
> >> > consumerRegistrationEndpoint to have a uri equal to the propertyThe
> >> > consumerRegistrationEndpoint routes from uri direct:publisher register with
> >> > the bean activityRegistrationProcessor nested in between the routeThe
> >> > streams-eip-applicationContext contains a bean with the id
> >> > activityRegistrationProcessor created for the class
> >> > ActivityPublisherRegistrationProcessorThe exchange will enter the "process"
> >> > method of the class ActivityPublisherRegisitrationProcessor and that this
> >> > is because this class implements the "Processor" interface provided by
> >> > CamelThe direct:add-publisher-route takes the exchange output from the
> >> > "process" method and routes it to the activityRegistrationProcessor
> >> > "register" method. The bean activityRegistrationProcessor is defined in the
> >> > streams-eip-osgi-component-import.xmlThe output from this method is then
> >> > sent to the "createNewRouteForConsumer" method of activityConsumerRouter.
> >> > This method creates a new route for the newly registered publisher using
> >> > the private static final class DynamicConsumerRouteBuilder which is
> >> > required to extend RouteBuilder which is provided by Camel. This
> >> > DynamicConsumerRouteBuilder contains several methods:
> >> > getConsumerReceiveMethod() (which corresponds to @Value
> >> > ${consumer.receiveMethod} which corresponds to "receive"),
> >> > getConsumerSplitMethod() (@Value ${consumer.splitMethod},"split"), and
> >> > getConsumerActivityQUri() (@Value ${consumerActivityQUri},
> >> > "direct:activityQ"). This is different than the camelContext.xml in that
> >> > the route is being created programmatically. What this is doing is routing
> >> > input from the inroute url (which Camel does automatically through the
> >> > configure method which is required to be overridden), to the "receive"
> >> > method of ActivityConsumer, then to the "split" method of ActivityConsumer,
> >> > and then to the "direct:activityQ" which if you look back in the
> >> > camelContext.xml routes to the "activemq:queue:activities" which then
> >> > routes to "receiveExchange"
> >> > This is the process to register a publisher. The process for registering a
> >> > subscriber is relatively the same though it involves separate classes with
> >> > their own private static final RouteBuilder class. From my perspective, the
> >> > two most difficult things with setting this project up were understanding
> >> > that the "process" method of the class that implements "Processor" is the
> >> > entry point and the DynamicConsumerRouteBuilder creates the second entry
> >> > point (The 5th and last points). This made the project very, VERY hard to
> >> > understand.
> >> > In addition to simplicity of design, the mvn clean install of the web
> >> > service project is much faster and small scale activity publishing is also
> >> > faster (see my email about load testing). These are minor points though as
> >> > compilation has no effect on deployment. OSGi does add the benefit of
> >> > modularized programming which is valuable, though I think the added
> >> > complexity of Camel merits moving the project away from this paradigm.
> >> >
> >>
> >> I agree that the project is pretty difficult to understand ATM.  I think
> >> what we need to do is think about what the responsibilities of the code are
> >> and allow for different implementations that are not so tightly coupled as
> >> they are now.  For instance, having worked with Storm to ingest millions of
> >> activities a day, I personally would like to see streams be responsible for
> >> defining an over-arching orchestration model that can be implemented within
> >> a single war or on top of a distributed system.  This would look something
> >> like the follows:
> >>
> >> |__ Streams-Core  (Base classes, interfaces, utilities, extensions, etc)
> >> |
> >> |__ Streams-Storm (Storm implementation of the core)
> >> |
> >> |__ Streams-Camel (Camel implementation of the core)
> >> |
> >> |__ Streams-WS (Web service implementation)
> >>
> >>
> >> > Danny
> >> >
> >> >
> >> > > From: dsullivan7@hotmail.com
> >> > > To: dev@streams.incubator.apache.org
> >> > > Subject: [DISCUSS] Switching Streams from Camel deployment to .war
> >> > deployment
> >> > > Date: Tue, 29 Oct 2013 11:07:39 -0400
> >> > >
> >> > > The discussion thread for switching Streams from Camel/osgi/Servicemix
> >> > to a single .war deployment
> >> >
> >> >
> >
 		 	   		  

Re: [DISCUSS] Switching Streams from Camel deployment to .war deployment

[Jason Letourneau <jl...@gmail.com>]

To be fair, while the current implementation is heavily camel-based,
all of the interfaces related to Streams functionality are not.  The
current model maps to what Matt has outlined in my opinion, though
packing names etc. probably don't follow that exact pattern.

With regards to the complexity and different components in the
registration process, this was a cut at the abstraction based on the
assumption that different implementations may be plugged in and in
fact may live on different processor space (ie. a polling publisher vs
a push publisher may be instantiated on different servers but the
registration URL is staticly defined).

Is the main argument I am seeing for Spring  the familiarity of its
IoC pattern implementation and syntax at the entry point?

On Wed, Oct 30, 2013 at 10:53 AM, Danny Sullivan <ds...@hotmail.com> wrote:
> Could you clarify whether the same entry points would exist for the camel implementation of the core (implementing the "process" method/ using a DynammicRouteBuilder) or would the webservice be the sole entry point to Streams and after it enters would it hand it off to Camel? And what would be the entry point for the Storm implementation?
> -Danny
>
>> Date: Wed, 30 Oct 2013 10:13:04 -0400
>> Subject: Re: [DISCUSS] Switching Streams from Camel deployment to .war deployment
>> From: m.ben.franklin@gmail.com
>> To: dev@streams.incubator.apache.org
>>
>> On Tue, Oct 29, 2013 at 2:55 PM, Danny Sullivan <ds...@hotmail.com>wrote:
>>
>> > My case for switching from OSGi is for simplicity in design. To follow the
>> > path of an activity through Streams in the webservice, there is one main
>> > things the developer needs to understand:
>> > The @RequestMapping annotation specifies the HTTP entry point
>> > There are 4 @RequestMapping annotations that correspond to each of the 4
>> > ways a user enters the application: registering a publisher, registering a
>> > subscriber, publishing activity, and getting an activity stream. Where
>> > these are located in the source code can be found by searching for the
>> > paths specified in the documentation (search for "/publisherRegister",
>> > "/subscriberRegister", "/publishActivity", "/getActivity" which will all
>> > lead you to StreamsWebController.java). From the methods that process
>> > requests, the flow through the application is through methods which can be
>> > understood by most Java programmers.
>> > The flow of activities through the current trunk branch is understood as
>> > follows:
>> > The string "/publisher/register" (the entry point to register a publisher
>> > specified in the documentation) is the value of the
>> > consumer.registrationEndpoint property defined in streams.propertiesThe
>> > camelContext.xml specifies an endpoint with the id
>> > consumerRegistrationEndpoint to have a uri equal to the propertyThe
>> > consumerRegistrationEndpoint routes from uri direct:publisher register with
>> > the bean activityRegistrationProcessor nested in between the routeThe
>> > streams-eip-applicationContext contains a bean with the id
>> > activityRegistrationProcessor created for the class
>> > ActivityPublisherRegistrationProcessorThe exchange will enter the "process"
>> > method of the class ActivityPublisherRegisitrationProcessor and that this
>> > is because this class implements the "Processor" interface provided by
>> > CamelThe direct:add-publisher-route takes the exchange output from the
>> > "process" method and routes it to the activityRegistrationProcessor
>> > "register" method. The bean activityRegistrationProcessor is defined in the
>> > streams-eip-osgi-component-import.xmlThe output from this method is then
>> > sent to the "createNewRouteForConsumer" method of activityConsumerRouter.
>> > This method creates a new route for the newly registered publisher using
>> > the private static final class DynamicConsumerRouteBuilder which is
>> > required to extend RouteBuilder which is provided by Camel. This
>> > DynamicConsumerRouteBuilder contains several methods:
>> > getConsumerReceiveMethod() (which corresponds to @Value
>> > ${consumer.receiveMethod} which corresponds to "receive"),
>> > getConsumerSplitMethod() (@Value ${consumer.splitMethod},"split"), and
>> > getConsumerActivityQUri() (@Value ${consumerActivityQUri},
>> > "direct:activityQ"). This is different than the camelContext.xml in that
>> > the route is being created programmatically. What this is doing is routing
>> > input from the inroute url (which Camel does automatically through the
>> > configure method which is required to be overridden), to the "receive"
>> > method of ActivityConsumer, then to the "split" method of ActivityConsumer,
>> > and then to the "direct:activityQ" which if you look back in the
>> > camelContext.xml routes to the "activemq:queue:activities" which then
>> > routes to "receiveExchange"
>> > This is the process to register a publisher. The process for registering a
>> > subscriber is relatively the same though it involves separate classes with
>> > their own private static final RouteBuilder class. From my perspective, the
>> > two most difficult things with setting this project up were understanding
>> > that the "process" method of the class that implements "Processor" is the
>> > entry point and the DynamicConsumerRouteBuilder creates the second entry
>> > point (The 5th and last points). This made the project very, VERY hard to
>> > understand.
>> > In addition to simplicity of design, the mvn clean install of the web
>> > service project is much faster and small scale activity publishing is also
>> > faster (see my email about load testing). These are minor points though as
>> > compilation has no effect on deployment. OSGi does add the benefit of
>> > modularized programming which is valuable, though I think the added
>> > complexity of Camel merits moving the project away from this paradigm.
>> >
>>
>> I agree that the project is pretty difficult to understand ATM.  I think
>> what we need to do is think about what the responsibilities of the code are
>> and allow for different implementations that are not so tightly coupled as
>> they are now.  For instance, having worked with Storm to ingest millions of
>> activities a day, I personally would like to see streams be responsible for
>> defining an over-arching orchestration model that can be implemented within
>> a single war or on top of a distributed system.  This would look something
>> like the follows:
>>
>> |__ Streams-Core  (Base classes, interfaces, utilities, extensions, etc)
>> |
>> |__ Streams-Storm (Storm implementation of the core)
>> |
>> |__ Streams-Camel (Camel implementation of the core)
>> |
>> |__ Streams-WS (Web service implementation)
>>
>>
>> > Danny
>> >
>> >
>> > > From: dsullivan7@hotmail.com
>> > > To: dev@streams.incubator.apache.org
>> > > Subject: [DISCUSS] Switching Streams from Camel deployment to .war
>> > deployment
>> > > Date: Tue, 29 Oct 2013 11:07:39 -0400
>> > >
>> > > The discussion thread for switching Streams from Camel/osgi/Servicemix
>> > to a single .war deployment
>> >
>> >
>

RE: [DISCUSS] Switching Streams from Camel deployment to .war deployment

[Danny Sullivan <ds...@hotmail.com>]

Could you clarify whether the same entry points would exist for the camel implementation of the core (implementing the "process" method/ using a DynammicRouteBuilder) or would the webservice be the sole entry point to Streams and after it enters would it hand it off to Camel? And what would be the entry point for the Storm implementation?
-Danny

> Date: Wed, 30 Oct 2013 10:13:04 -0400
> Subject: Re: [DISCUSS] Switching Streams from Camel deployment to .war deployment
> From: m.ben.franklin@gmail.com
> To: dev@streams.incubator.apache.org
> 
> On Tue, Oct 29, 2013 at 2:55 PM, Danny Sullivan <ds...@hotmail.com>wrote:
> 
> > My case for switching from OSGi is for simplicity in design. To follow the
> > path of an activity through Streams in the webservice, there is one main
> > things the developer needs to understand:
> > The @RequestMapping annotation specifies the HTTP entry point
> > There are 4 @RequestMapping annotations that correspond to each of the 4
> > ways a user enters the application: registering a publisher, registering a
> > subscriber, publishing activity, and getting an activity stream. Where
> > these are located in the source code can be found by searching for the
> > paths specified in the documentation (search for "/publisherRegister",
> > "/subscriberRegister", "/publishActivity", "/getActivity" which will all
> > lead you to StreamsWebController.java). From the methods that process
> > requests, the flow through the application is through methods which can be
> > understood by most Java programmers.
> > The flow of activities through the current trunk branch is understood as
> > follows:
> > The string "/publisher/register" (the entry point to register a publisher
> > specified in the documentation) is the value of the
> > consumer.registrationEndpoint property defined in streams.propertiesThe
> > camelContext.xml specifies an endpoint with the id
> > consumerRegistrationEndpoint to have a uri equal to the propertyThe
> > consumerRegistrationEndpoint routes from uri direct:publisher register with
> > the bean activityRegistrationProcessor nested in between the routeThe
> > streams-eip-applicationContext contains a bean with the id
> > activityRegistrationProcessor created for the class
> > ActivityPublisherRegistrationProcessorThe exchange will enter the "process"
> > method of the class ActivityPublisherRegisitrationProcessor and that this
> > is because this class implements the "Processor" interface provided by
> > CamelThe direct:add-publisher-route takes the exchange output from the
> > "process" method and routes it to the activityRegistrationProcessor
> > "register" method. The bean activityRegistrationProcessor is defined in the
> > streams-eip-osgi-component-import.xmlThe output from this method is then
> > sent to the "createNewRouteForConsumer" method of activityConsumerRouter.
> > This method creates a new route for the newly registered publisher using
> > the private static final class DynamicConsumerRouteBuilder which is
> > required to extend RouteBuilder which is provided by Camel. This
> > DynamicConsumerRouteBuilder contains several methods:
> > getConsumerReceiveMethod() (which corresponds to @Value
> > ${consumer.receiveMethod} which corresponds to "receive"),
> > getConsumerSplitMethod() (@Value ${consumer.splitMethod},"split"), and
> > getConsumerActivityQUri() (@Value ${consumerActivityQUri},
> > "direct:activityQ"). This is different than the camelContext.xml in that
> > the route is being created programmatically. What this is doing is routing
> > input from the inroute url (which Camel does automatically through the
> > configure method which is required to be overridden), to the "receive"
> > method of ActivityConsumer, then to the "split" method of ActivityConsumer,
> > and then to the "direct:activityQ" which if you look back in the
> > camelContext.xml routes to the "activemq:queue:activities" which then
> > routes to "receiveExchange"
> > This is the process to register a publisher. The process for registering a
> > subscriber is relatively the same though it involves separate classes with
> > their own private static final RouteBuilder class. From my perspective, the
> > two most difficult things with setting this project up were understanding
> > that the "process" method of the class that implements "Processor" is the
> > entry point and the DynamicConsumerRouteBuilder creates the second entry
> > point (The 5th and last points). This made the project very, VERY hard to
> > understand.
> > In addition to simplicity of design, the mvn clean install of the web
> > service project is much faster and small scale activity publishing is also
> > faster (see my email about load testing). These are minor points though as
> > compilation has no effect on deployment. OSGi does add the benefit of
> > modularized programming which is valuable, though I think the added
> > complexity of Camel merits moving the project away from this paradigm.
> >
> 
> I agree that the project is pretty difficult to understand ATM.  I think
> what we need to do is think about what the responsibilities of the code are
> and allow for different implementations that are not so tightly coupled as
> they are now.  For instance, having worked with Storm to ingest millions of
> activities a day, I personally would like to see streams be responsible for
> defining an over-arching orchestration model that can be implemented within
> a single war or on top of a distributed system.  This would look something
> like the follows:
> 
> |__ Streams-Core  (Base classes, interfaces, utilities, extensions, etc)
> |
> |__ Streams-Storm (Storm implementation of the core)
> |
> |__ Streams-Camel (Camel implementation of the core)
> |
> |__ Streams-WS (Web service implementation)
> 
> 
> > Danny
> >
> >
> > > From: dsullivan7@hotmail.com
> > > To: dev@streams.incubator.apache.org
> > > Subject: [DISCUSS] Switching Streams from Camel deployment to .war
> > deployment
> > > Date: Tue, 29 Oct 2013 11:07:39 -0400
> > >
> > > The discussion thread for switching Streams from Camel/osgi/Servicemix
> > to a single .war deployment
> >
> >
 		 	   		  

Re: [DISCUSS] Switching Streams from Camel deployment to .war deployment

[Matt Franklin <m....@gmail.com>]

On Tue, Oct 29, 2013 at 2:55 PM, Danny Sullivan <ds...@hotmail.com>wrote:

> My case for switching from OSGi is for simplicity in design. To follow the
> path of an activity through Streams in the webservice, there is one main
> things the developer needs to understand:
> The @RequestMapping annotation specifies the HTTP entry point
> There are 4 @RequestMapping annotations that correspond to each of the 4
> ways a user enters the application: registering a publisher, registering a
> subscriber, publishing activity, and getting an activity stream. Where
> these are located in the source code can be found by searching for the
> paths specified in the documentation (search for "/publisherRegister",
> "/subscriberRegister", "/publishActivity", "/getActivity" which will all
> lead you to StreamsWebController.java). From the methods that process
> requests, the flow through the application is through methods which can be
> understood by most Java programmers.
> The flow of activities through the current trunk branch is understood as
> follows:
> The string "/publisher/register" (the entry point to register a publisher
> specified in the documentation) is the value of the
> consumer.registrationEndpoint property defined in streams.propertiesThe
> camelContext.xml specifies an endpoint with the id
> consumerRegistrationEndpoint to have a uri equal to the propertyThe
> consumerRegistrationEndpoint routes from uri direct:publisher register with
> the bean activityRegistrationProcessor nested in between the routeThe
> streams-eip-applicationContext contains a bean with the id
> activityRegistrationProcessor created for the class
> ActivityPublisherRegistrationProcessorThe exchange will enter the "process"
> method of the class ActivityPublisherRegisitrationProcessor and that this
> is because this class implements the "Processor" interface provided by
> CamelThe direct:add-publisher-route takes the exchange output from the
> "process" method and routes it to the activityRegistrationProcessor
> "register" method. The bean activityRegistrationProcessor is defined in the
> streams-eip-osgi-component-import.xmlThe output from this method is then
> sent to the "createNewRouteForConsumer" method of activityConsumerRouter.
> This method creates a new route for the newly registered publisher using
> the private static final class DynamicConsumerRouteBuilder which is
> required to extend RouteBuilder which is provided by Camel. This
> DynamicConsumerRouteBuilder contains several methods:
> getConsumerReceiveMethod() (which corresponds to @Value
> ${consumer.receiveMethod} which corresponds to "receive"),
> getConsumerSplitMethod() (@Value ${consumer.splitMethod},"split"), and
> getConsumerActivityQUri() (@Value ${consumerActivityQUri},
> "direct:activityQ"). This is different than the camelContext.xml in that
> the route is being created programmatically. What this is doing is routing
> input from the inroute url (which Camel does automatically through the
> configure method which is required to be overridden), to the "receive"
> method of ActivityConsumer, then to the "split" method of ActivityConsumer,
> and then to the "direct:activityQ" which if you look back in the
> camelContext.xml routes to the "activemq:queue:activities" which then
> routes to "receiveExchange"
> This is the process to register a publisher. The process for registering a
> subscriber is relatively the same though it involves separate classes with
> their own private static final RouteBuilder class. From my perspective, the
> two most difficult things with setting this project up were understanding
> that the "process" method of the class that implements "Processor" is the
> entry point and the DynamicConsumerRouteBuilder creates the second entry
> point (The 5th and last points). This made the project very, VERY hard to
> understand.
> In addition to simplicity of design, the mvn clean install of the web
> service project is much faster and small scale activity publishing is also
> faster (see my email about load testing). These are minor points though as
> compilation has no effect on deployment. OSGi does add the benefit of
> modularized programming which is valuable, though I think the added
> complexity of Camel merits moving the project away from this paradigm.
>

I agree that the project is pretty difficult to understand ATM.  I think
what we need to do is think about what the responsibilities of the code are
and allow for different implementations that are not so tightly coupled as
they are now.  For instance, having worked with Storm to ingest millions of
activities a day, I personally would like to see streams be responsible for
defining an over-arching orchestration model that can be implemented within
a single war or on top of a distributed system.  This would look something
like the follows:

|__ Streams-Core  (Base classes, interfaces, utilities, extensions, etc)
|
|__ Streams-Storm (Storm implementation of the core)
|
|__ Streams-Camel (Camel implementation of the core)
|
|__ Streams-WS (Web service implementation)


> Danny
>
>
> > From: dsullivan7@hotmail.com
> > To: dev@streams.incubator.apache.org
> > Subject: [DISCUSS] Switching Streams from Camel deployment to .war
> deployment
> > Date: Tue, 29 Oct 2013 11:07:39 -0400
> >
> > The discussion thread for switching Streams from Camel/osgi/Servicemix
> to a single .war deployment
>
>

RE: [DISCUSS] Switching Streams from Camel deployment to .war deployment

[Danny Sullivan <ds...@hotmail.com>]

My case for switching from OSGi is for simplicity in design. To follow the path of an activity through Streams in the webservice, there is one main things the developer needs to understand: 
The @RequestMapping annotation specifies the HTTP entry point  
There are 4 @RequestMapping annotations that correspond to each of the 4 ways a user enters the application: registering a publisher, registering a subscriber, publishing activity, and getting an activity stream. Where these are located in the source code can be found by searching for the paths specified in the documentation (search for "/publisherRegister", "/subscriberRegister", "/publishActivity", "/getActivity" which will all lead you to StreamsWebController.java). From the methods that process requests, the flow through the application is through methods which can be understood by most Java programmers.
The flow of activities through the current trunk branch is understood as follows:
The string "/publisher/register" (the entry point to register a publisher specified in the documentation) is the value of the consumer.registrationEndpoint property defined in streams.propertiesThe camelContext.xml specifies an endpoint with the id consumerRegistrationEndpoint to have a uri equal to the propertyThe consumerRegistrationEndpoint routes from uri direct:publisher register with the bean activityRegistrationProcessor nested in between the routeThe streams-eip-applicationContext contains a bean with the id activityRegistrationProcessor created for the class ActivityPublisherRegistrationProcessorThe exchange will enter the "process" method of the class ActivityPublisherRegisitrationProcessor and that this is because this class implements the "Processor" interface provided by CamelThe direct:add-publisher-route takes the exchange output from the "process" method and routes it to the activityRegistrationProcessor "register" method. The bean activityRegistrationProcessor is defined in the streams-eip-osgi-component-import.xmlThe output from this method is then sent to the "createNewRouteForConsumer" method of activityConsumerRouter. This method creates a new route for the newly registered publisher using the private static final class DynamicConsumerRouteBuilder which is required to extend RouteBuilder which is provided by Camel. This DynamicConsumerRouteBuilder contains several methods: getConsumerReceiveMethod() (which corresponds to @Value ${consumer.receiveMethod} which corresponds to "receive"), getConsumerSplitMethod() (@Value ${consumer.splitMethod},"split"), and getConsumerActivityQUri() (@Value ${consumerActivityQUri}, "direct:activityQ"). This is different than the camelContext.xml in that the route is being created programmatically. What this is doing is routing input from the inroute url (which Camel does automatically through the configure method which is required to be overridden), to the "receive" method of ActivityConsumer, then to the "split" method of ActivityConsumer, and then to the "direct:activityQ" which if you look back in the camelContext.xml routes to the "activemq:queue:activities" which then routes to "receiveExchange" 
This is the process to register a publisher. The process for registering a subscriber is relatively the same though it involves separate classes with their own private static final RouteBuilder class. From my perspective, the two most difficult things with setting this project up were understanding that the "process" method of the class that implements "Processor" is the entry point and the DynamicConsumerRouteBuilder creates the second entry point (The 5th and last points). This made the project very, VERY hard to understand. 
In addition to simplicity of design, the mvn clean install of the web service project is much faster and small scale activity publishing is also faster (see my email about load testing). These are minor points though as compilation has no effect on deployment. OSGi does add the benefit of modularized programming which is valuable, though I think the added complexity of Camel merits moving the project away from this paradigm.
Danny  


> From: dsullivan7@hotmail.com
> To: dev@streams.incubator.apache.org
> Subject: [DISCUSS] Switching Streams from Camel deployment to .war deployment
> Date: Tue, 29 Oct 2013 11:07:39 -0400
> 
> The discussion thread for switching Streams from Camel/osgi/Servicemix to a single .war deployment