You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@streams.apache.org by Danny Sullivan <ds...@hotmail.com> on 2013/10/03 22:50:35 UTC
Using Camel for Routing
Hey all,
I've been considering taking a stab at moving streams from it's current configuration with Camel to a Spring Web Service. Concerns over failover and load balancing would be mitigated by the Cassandra DB (running on 9160) and Storm (which depends on an instance of Zookeeper on 2181). Before I start lancing any windmills I thought it'd be nice to get input on why the design choice was made to base the application around Camel?
I think that tracking the flow of an activity through the application will be much easier if we use Spring MVC. I'll set up a new branch using the MVC architecture and I'll keep the community updated as I go along.
Danny
Re: Using Camel for Routing
Posted by Chris Geer <ch...@cxtsoftware.com>.
On Thu, Oct 3, 2013 at 5:35 PM, Danny Sullivan <ds...@hotmail.com>wrote:
> Hey Chris,
> Thanks for the response. The way Streams is currently implemented, Camel
> provides both entry and exit endpoints to the application. These are either
> dynamic (when a publisher or subscriber is registered) or defined in the
> camel context. Camel also provides routing from component to component once
> publishers (ActivityConsumers) and subscribers (ActivitySubscribers) are
> created. Because these classes are serialized so that they can be routed in
> Camel, it made sense to me to keep these objects as they are and pass them
> from class to class via method calls (as opposed of using "from" and "to"
> in the camel context).
Ya, I agree the existing architecture has it quirks. I would personally
like to see something a lot more loosely coupled and message based (not
method based). To me, a publish-subscribe (or at least a queue based async
models) fits streams much better than a request/response model of method
calls. I need to get some time to document my idea and get some feedback on
it.
> The reason I chose a Spring Web Service as opposed to any other library is
> because Spring is already in use in the application and it is what is
> familiar to me. With the use of a Spring MVC all of the entry and exit
> points will be in one Web Controller class and the user can follow an
> activity all the way through the application.
>
I don't think adding Cassandra and Zookeeper will add too much ease of use
> issues to the application. To incorporate storm (and zookeeper), all that's
> needed is a dependency in the pom, there is no added software that is
> needed by the user. Cassandra on the the other hand does require the user
> to download the Cassandra software, so you're correct in that regard. I'd
> really like to embed Cassandra into the application and is one of the tasks
> I'd like to work on.
Is there a reason that you would recommend CFX as opposed to Spring MVC?
> I'm all for researching alternatives to a Spring MVC if you think it will
> be easier and lighter.
>
I have several biases toward CXF:
- It's another Apache project (this is purely my preference to see Apache
projects linked against other Apache projects when possible)
- It has great OSGI and WebApp support - Last I heard Spring is removing
OSGI support from it's code [1]. You can get OSGI version from their EBR
repository but that is supposed to be shut down sometime next year.
- CXF is more stand-alone than Spring MVC. It's purely a webservice
framework while Spring MVC sucks in a bunch of UI related dependencies. CXF
does support both SOAP and RESTful webservices however it also allows you
to get distros that just support the one you need.
> What do you mean by access to various services like Twitter and Facebook?
> Are these services that we would not get access to if we switched from
> Camel?
>
Camel is an integration framework. It's job is to allow people to build
"routes" between various end points. Those endpoints can be web services
(like streams uses today) but they also support many other endpoints [2]
like Twitter and Facebook. Camel would give Streams a level of
configurability that would be hard to get with just Java. Let's say someone
needs to read messages from twitter, process them for certain information
and then write those processed messages to a file (not great use case I
know). In camel the system could just spin up a new route like this.
from("twitter://search?type=direct&keywords=streams")
.to("bean:tweetProcessor")
.file("tweetStorage")
I know there has been some talk about replacing Camel with Storm but I
don't think it has support for the same number of endpoints, at least yet.
I have seen a couple places where they were used together. I need to learn
more about Storm but I do have some questions about how the vision is to
use it for Streams if anyone can spell that out. I suspect using Storm
requires a very different approach than is being used now anyway (for
example, embedding Storm in a webapp probably isn't the plan I assume).
> Thanks again for the feedback!
> Danny
>
Chris
[1] http://www.infoq.com/news/2012/10/spring-osgi-gradle
[2] http://camel.apache.org/components.html
>
> > Date: Thu, 3 Oct 2013 17:14:33 -0700
> > Subject: Re: Using Camel for Routing
>
> > From: chris@cxtsoftware.com
> > To: dev@streams.incubator.apache.org
> >
> > On Thursday, October 3, 2013, Danny Sullivan wrote:
> >
> > > Hey all,
> > > I've been considering taking a stab at moving streams from it's current
> > > configuration with Camel to a Spring Web Service.
> >
> >
> > Spring MVC != Camel. The driver for camel was an easily composable
> solution
> > that gets you access to various services like Twitter and Facebook. If
> you
> > just want a web service I'd recommend CXF.
> >
> > Concerns over failover and load balancing would be mitigated by the
> > > Cassandra DB (running on 9160) and Storm (which depends on an instance
> of
> > > Zookeeper on 2181).
> >
> > Just my 2 cents, but adding requirements for Cassandra and Zookeeper
> raises
> > the bar on "ease of use" quite a bit. Not saying it might not be the
> right
> > choice, just and observation.
> >
> >
> >
> > > Before I start lancing any windmills I thought it'd be nice to get
> input
> > > on why the design choice was made to base the application around Camel?
> > > I think that tracking the flow of an activity through the application
> will
> > > be much easier if we use Spring MVC. I'll set up a new branch using
> the MVC
> > > architecture and I'll keep the community updated as I go along.
> >
> > What is the primary driver for introducing another major framework like
> > Spring?
> >
> > > Danny
>
>
RE: Using Camel for Routing
Posted by Danny Sullivan <ds...@hotmail.com>.
Hey Chris,
Thanks for the response. The way Streams is currently implemented, Camel provides both entry and exit endpoints to the application. These are either dynamic (when a publisher or subscriber is registered) or defined in the camel context. Camel also provides routing from component to component once publishers (ActivityConsumers) and subscribers (ActivitySubscribers) are created. Because these classes are serialized so that they can be routed in Camel, it made sense to me to keep these objects as they are and pass them from class to class via method calls (as opposed of using "from" and "to" in the camel context). The reason I chose a Spring Web Service as opposed to any other library is because Spring is already in use in the application and it is what is familiar to me. With the use of a Spring MVC all of the entry and exit points will be in one Web Controller class and the user can follow an activity all the way through the application.
I don't think adding Cassandra and Zookeeper will add too much ease of use issues to the application. To incorporate storm (and zookeeper), all that's needed is a dependency in the pom, there is no added software that is needed by the user. Cassandra on the the other hand does require the user to download the Cassandra software, so you're correct in that regard. I'd really like to embed Cassandra into the application and is one of the tasks I'd like to work on.
Is there a reason that you would recommend CFX as opposed to Spring MVC? I'm all for researching alternatives to a Spring MVC if you think it will be easier and lighter.
What do you mean by access to various services like Twitter and Facebook? Are these services that we would not get access to if we switched from Camel?
Thanks again for the feedback!
Danny
> Date: Thu, 3 Oct 2013 17:14:33 -0700
> Subject: Re: Using Camel for Routing
> From: chris@cxtsoftware.com
> To: dev@streams.incubator.apache.org
>
> On Thursday, October 3, 2013, Danny Sullivan wrote:
>
> > Hey all,
> > I've been considering taking a stab at moving streams from it's current
> > configuration with Camel to a Spring Web Service.
>
>
> Spring MVC != Camel. The driver for camel was an easily composable solution
> that gets you access to various services like Twitter and Facebook. If you
> just want a web service I'd recommend CXF.
>
> Concerns over failover and load balancing would be mitigated by the
> > Cassandra DB (running on 9160) and Storm (which depends on an instance of
> > Zookeeper on 2181).
>
> Just my 2 cents, but adding requirements for Cassandra and Zookeeper raises
> the bar on "ease of use" quite a bit. Not saying it might not be the right
> choice, just and observation.
>
>
>
> > Before I start lancing any windmills I thought it'd be nice to get input
> > on why the design choice was made to base the application around Camel?
> > I think that tracking the flow of an activity through the application will
> > be much easier if we use Spring MVC. I'll set up a new branch using the MVC
> > architecture and I'll keep the community updated as I go along.
>
> What is the primary driver for introducing another major framework like
> Spring?
>
> > Danny
Re: Using Camel for Routing
Posted by Chris Geer <ch...@cxtsoftware.com>.
On Thursday, October 3, 2013, Danny Sullivan wrote:
> Hey all,
> I've been considering taking a stab at moving streams from it's current
> configuration with Camel to a Spring Web Service.
Spring MVC != Camel. The driver for camel was an easily composable solution
that gets you access to various services like Twitter and Facebook. If you
just want a web service I'd recommend CXF.
Concerns over failover and load balancing would be mitigated by the
> Cassandra DB (running on 9160) and Storm (which depends on an instance of
> Zookeeper on 2181).
Just my 2 cents, but adding requirements for Cassandra and Zookeeper raises
the bar on "ease of use" quite a bit. Not saying it might not be the right
choice, just and observation.
> Before I start lancing any windmills I thought it'd be nice to get input
> on why the design choice was made to base the application around Camel?
> I think that tracking the flow of an activity through the application will
> be much easier if we use Spring MVC. I'll set up a new branch using the MVC
> architecture and I'll keep the community updated as I go along.
What is the primary driver for introducing another major framework like
Spring?
> Danny
Re: Using Camel for Routing
Posted by "Steve Blackmon [W2O Digital]" <sb...@w2odigital.com>.
I think a community driven approach to normalizing APIs into activities
would be hugely valuable.
Capture and normalization is the sole focus of some of the modules we¹ve
been working on. Currently they can only run within storm but there¹s no
reason they shouldn¹t be able to deploy into any container that can spawn
threads which can open socket connections.
We intend to push some of this soon - with initial out-of-the-box support
for twitter and moreover metabase. My hope is that we can convince
developers within data providers to join the project to perfect and
maintain those modules.
Steve Blackmon
Director, Data Sciences
W2O Group
3000 E Cesar Chavez St
Austin, Texas 78701
cell 512.965.0451 | work 512.402.6366
twitter @steveblackmon
On 10/16/13, 1:58 PM, "Danny Sullivan" <ds...@hotmail.com> wrote:
>I see the value in Streams as the ability of subscribers to get activity
>based on filters. To be clear, I do not want to eliminate processing, but
>do want to make the focus of processing tailored to the subscribers'
>filters. My more grand vision is that only the most relevant activity
>will be returned to the Subscriber and similar activity will be
>aggregated into single activities (this would be a little down the road).
>This processing would happen after the activity is stored in the DB.
>Do you imagine that Streams will come packaged with some processing
>components that will format other non-activitystrea.ms formatted json? I
>do think it'd be pretty cool if you could fire up Streams, input a couple
>Facebook Friends and people you follow on Twitter, and have all of that
>activity posted in one place. That might be better suited for a new
>separate module (something like streams-format?) if we decide on the
>web-application route.
>-Danny
>
>> Date: Wed, 16 Oct 2013 11:22:51 -0700
>> Subject: Re: Using Camel for Routing
>> From: chris@cxtsoftware.com
>> To: dev@streams.incubator.apache.org
>>
>> On Wed, Oct 16, 2013 at 11:12 AM, Danny Sullivan
>><ds...@hotmail.com>wrote:
>>
>> > I think what is happening here is that we have different ideas about
>>the
>> > architecture of the application.
>> > Here's what I'm thinking:
>> > *All processing of activity would happen BEFORE it's posted to
>>Streams*All
>> > activity posted to Streams is in activitystrea.ms format
>> > The Camel/EIP/OSGi way:
>> > *The user is able to post anything to Streams, doesn't matter what
>>format
>> > it's in*The user can add processing components to Streams that format
>> > everything that is input to activitystrea.ms format, these can be
>>easily
>> > plugged in, moved around, or taken out
>> > If we decide to go the second route, I think Camel would be the best
>>way
>> > to go. However, I'd like to make the case for the first approach. The
>> > application can be quickly started and tested by a developer and user
>>with
>> > the the first approach. If the user decides they want to experiment
>>with
>> > twitter/facebook data (like Chris suggested) they can write a custom
>> > application (which can be a camel/eip application or even a python,
>>ruby,
>> > etc application) that takes in the twitter/facebook data, formats it
>>to
>> > activitystrea.ms format and then POSTs it to Streams. Also note: the
>> > webapp would NOT need to be restarted to add these. I don't think
>>we'd lose
>> > a lot by going this route: in order to add the custom processing
>>components
>> > in Camel, a Java component would still need to be written before it's
>> > plugged in to the camel.xml (and I think you would be tied to Java
>>with
>> > this approach).
>> > Quite simply: We should keep formatting activity separate processing
>> > activity
>> > Perhaps I am missing some processing that could happen in between an
>> > activity's entrance to Streams and it's storage? If this is the case,
>>I
>> > would make the argument that this processing should happen on an
>>activities
>> > EXIT from Streams (there should be a way for a Subscriber to specify,
>>in
>> > its filters or by some other means, the activity that it wants to see)
>> >
>>
>> Danny, what do you see as the scope for Streams? If you eliminate data
>> input in non-activitystrea.ms format and eliminate processing data what
>> does it do, just store each message (unmodified) and output it to
>> subscribers?
>>
>>
>> > -Danny
>> >
>> > > Date: Wed, 16 Oct 2013 10:11:23 -0700
>> > > Subject: Re: Using Camel for Routing
>> > > From: chris@cxtsoftware.com
>> > > To: dev@streams.incubator.apache.org
>> > >
>> > > Jason,
>> > >
>> > > I agree with you. I guess my vision of streams was a little grander
>>in
>> > the
>> > > long run where you could configure the system to accept input from
>> > various
>> > > sources (facebook, twitter...) and process them in certain ways. So
>>it
>> > > would have components that would know how to talk with Twitter and
>> > convert
>> > > tweets to activities then send it to the right process. So
>>depending on
>> > how
>> > > you configure your system, it would create the proper routes from
>> > component
>> > > to component to allow processing dynamically which would require a
>> > > framework like Camel (don't think Storm helps us here since it's a
>> > compile
>> > > time configuration to my understanding). The nice thing about
>>something
>> > > like camel is it allows you to reconfigure all the routes at
>>runtime.
>> > This
>> > > is another reason I liked OSGI support as well as it allows
>>deploying new
>> > > components without restarting the server (webapp). Of course to
>>scale it
>> > > would require being able to deploy the configuration across multiple
>> > > machines which gets fun.
>> > >
>> > > Maybe my vision is too big for this project but it's where my head
>>has
>> > been
>> > > at.
>> > >
>> > > Chris
>> > >
>> > >
>> > > On Wed, Oct 16, 2013 at 7:54 AM, Jason Letourneau
>> > > <jl...@gmail.com>wrote:
>> > >
>> > > > My only real complaint with this approach is that interjecting new
>> > > > components (processing components) becomes less plug and play
>>without
>> > > > writing Java code in streams - with the Camel routing approach,
>> > > > streams Java is a black box and you just change the camel xml.
>> > > >
>> > > > The primary rationale in using Camel was gaining all of the
>> > > > architectural benefits one gains from enterprise integration
>>patterns
>> > > > (truth: simplicity is not one and is a tradeoff for flexibility,
>>but
>> > > > generally is considered a universal truth not one relegated to
>>Camel
>> > > > and EIP).
>> > > >
>> > > > Jason
>> > > >
>> > > > On Fri, Oct 11, 2013 at 10:16 AM, Danny Sullivan <
>> > dsullivan7@hotmail.com>
>> > > > wrote:
>> > > > > Hey All,
>> > > > > I would like to merge the webservice branch with trunk. I think
>> > making
>> > > > streams web service based allows for a better focus on the goal
>>of the
>> > > > application: process activitystrea.ms formatted activity json.
>>While
>> > > > moving to a webservice model would lose us support for many of
>>the use
>> > > > cases Chris mentioned (i.e twitter and facebook), I think the
>>downsides
>> > > > would be mitigated by allowing and encouraging users to write
>>custom
>> > > > appenders that take in twitter/facebook data, format it to
>> > > > activitystrea.ms standards, and POST to the Streams Server. These
>> > > > appenders could even be written in any language as long as the
>>output
>> > is
>> > > > activitystrea.ms json. Compared to the trunk branch, I think the
>>web
>> > > > service design employs the "black box" idiom in a much simpler
>> > fashion. I
>> > > > would be happy to hear contradicting thoughts and would encourage
>>a
>> > > > discussion on this design switch.
>> > > > > Thanks!
>> > > > > Danny
>> > > > >> From: dsullivan7@hotmail.com
>> > > > >> To: dev@streams.incubator.apache.org
>> > > > >> Subject: RE: Using Camel for Routing
>> > > > >> Date: Fri, 4 Oct 2013 16:07:06 -0400
>> > > > >>
>> > > > >> I wrote up something quickly so I could demo the functionality
>>as a
>> > web
>> > > > service as opposed to Camel. The new branch is available here:
>> > > > >>
>> >
>>http://cp.mcafee.com/d/FZsScz8A71NJ5BV4QsI6zAQsIK6XCQQmhPMWWrMUSCyMy-yCOy
>>edFEIzDxRQTDzqqb339EVuujjdQ87XoGHm0ac9RgAhBfj-ndC4WEi8ODF_bCXKE_XWr_nV55B
>>ZVZBXHTbEKe9YyVtxNUsqeumKzp5dmVEVvVkffGhBrwqrhdI6XYCej79zANOoUTsSjDdqympg
>>P3JmSNf-00V7RGQPqeVa5CO6PH7ajz-GQMx7om9_j01k-9ClLMSxVAL7VJNwnojKnEgfSNlmI
>>3z2tk94pjQ_BMlisvRmDixpIxIWNXlJIj_w09J5wQsIFCMmd96y09QB0yq81blvfd43JoCy05
>>RrDa14TsT_Py-uR5BLSC
>> > > > >> note that the new publishing and subscribing urls are:
>> > > > >>
>> > > >
>> >
>>http://cp.mcafee.com/d/1jWVIg43qbbO9EVod79EVpsdTdFEIzDxRQTxNJd5x5Z5dB4srj
>>hp7f3HFLf6QQm66jhOYYCCrEgfSNlmI0kojGx8zauDYKrc9RgAhBfj-ndTth_TQT-LOabbXPX
>>bTnKnhssjV5OX3zMUQsYJt6OaqJPhO_OEuvkzaT0QSCrodTVcsCej79zANNKVI06vmSDAWQCA
>>ZtW-kD7ZlEn05CHboISgfBgDYgnV6l9_406vmSDAWQCAZtW-kD7ZlEn05CHbphIxiQp4fy2_8
>>OFfUCQfcBU_dKc2X2tOZ21-SaGRwsojGx8zauDYK2Gjz-GQWkbdAdDmfqJJyvY01dEI6zBBcS
>>2NF8Qg1eAE4jh09qHVVEwtH4Qg0KHsVg8CXC_C3cpxp4WcOz
>> > > > >> I used a Spring Web Service only because I was more familiar
>>with
>> > it, I
>> > > > would like to research other alternatives (CXF) if it's clear
>>that a
>> > web
>> > > > service is a realistic alternative. The main changes to the
>> > application are
>> > > > the web controller located here:
>> > > > >>
>> > > >
>> >
>>http://cp.mcafee.com/d/2DRPoO83gAd6QmnAjhOMqejhOOUrKrjhp7f3HFL3zqqb2bWara
>>8USCyOeu7njuudFEIccCzBVVdcTgwvJyGJo0EMDl2h6kZfVsSojGx8zauDYKrKWz_LFLZvAkm
>>nTDSnKLsKyUUDObBS77xNEVVqWdAkRrCzB_BgY-F6lK1FJASMrLOoVcsCej79zztPpesRG9pB
>>3ceRrr4_U03AvmHjdEXAEmr8reIsFefWHj24txoDZc05jUCpm_ajz-GQbw2OAroHIEo8WpT00
>>eqJJyvVisvRmCaCrpqI9NGuvYs69N_lq0bV0iH2sqDD_7x7jeCQfcBU_dKc2X2tOZ21-SaGRw
>>sojGx8zauDYK2Gjz-GQWkbdAdDmfqJJyvY01dEI6zBBcS2NF8Qg1eAE4jh09qHVVEwtH4Qg0K
>>HsVg8CXCTVkAs4DJ
>> > > > >> Which handles the input and output of the application, which
>>were
>> > > > handled by the camel context before. The other main change is the
>>use
>> > of
>> > > > the streams-components module located here:
>> > > > >>
>> > > >
>> >
>>http://cp.mcafee.com/d/2DRPoArhpuhd7b1EVd7bbxKVJd5AsYeKCYedFEI8LEFIEzzqqb
>>8VUttdVUSCyMMOqenDAQPt21-SaGRw2z2tk94pjQ_BPpxeG4ycFWvOVKXGf--C_R-hhpvuvpu
>>WZOWbzyv8Knosu76zDBHEShjlKqen-l3PWApmU6CS3r1K_9zANOoVcsCedTdAVPmEBCkcMXlJ
>>Ij_w0ehZqJcSzKixpIxIWNOAU_GJc8hS5yvQM0lfypBrYFefWHgKKJm7MDj3q7CiYvCT61txe
>>Vux0_r5lqMec9RgAhBfj-n1l9N_lqta5CO6PH7JmSNf-00CQm3hOOCr1oQAq80Dik29Ew4JlY
>>YQgeRyq80nlKsE4jtPqkR9rHK5
>> > > > >> This module contains service classes that return the correct
>>output
>> > on
>> > > > each request sent to the controller. I reused the Publisher and
>> > Subscriber
>> > > > objects as well as the activityConsumerWarehouse and
>> > > > activitySubscriberWarehouse, I think they would be moved to the
>> > > > streams-components package if we want to go this route.
>> > > > >> Overall, I think the flow through the application is much more
>> > > > understandable. Previously, a developer would have to follow an
>> > activity
>> > > > through the camelContext.xml as well as through dynamically
>>created
>> > routes
>> > > > verses the web service which provides clear entry and exit points
>>to
>> > the
>> > > > application. I'd really like to hear feedback on this one, I
>>really
>> > don't
>> > > > want to lose any efficiencies that we could be from
>> > > > messaging/osgi/camel/activemq.
>> > > > >> Thanks,
>> > > > >> Danny
>> > > > >>
>> > > > >> > From: dsullivan7@hotmail.com
>> > > > >> > To: dev@streams.incubator.apache.org
>> > > > >> > Subject: RE: Using Camel for Routing
>> > > > >> > Date: Thu, 3 Oct 2013 20:17:13 -0400
>> > > > >> >
>> > > > >> > Thanks for the update Jason. I'll keep the Spring MVC app in
>>a
>> > > > separate branch (hopefully it won't take too long) and then
>>perhaps we
>> > can
>> > > > reassess having both component and web container deployment? I
>>think
>> > it's a
>> > > > good idea to keep both options in mind especially because this
>> > technology
>> > > > is relatively young.
>> > > > >> > Thanks!
>> > > > >> > Danny
>> > > > >> >
>> > > > >> > > Date: Thu, 3 Oct 2013 19:50:59 -0400
>> > > > >> > > Subject: Re: Using Camel for Routing
>> > > > >> > > From: jletourneau80@gmail.com
>> > > > >> > > To: dev@streams.incubator.apache.org
>> > > > >> > >
>> > > > >> > > Because of the osgi container support in addition to
>>deployment
>> > as
>> > > > a web
>> > > > >> > > app - when last we visited this topic - I think the
>>majority
>> > was in
>> > > > favor
>> > > > >> > > of keeping the component style deployment in addition to
>>web
>> > > > container
>> > > > >> > > deployment
>> > > > >> > >
>> > > > >> > > On Thursday, October 3, 2013, Danny Sullivan wrote:
>> > > > >> > >
>> > > > >> > > > Hey all,
>> > > > >> > > > I've been considering taking a stab at moving streams
>>from
>> > it's
>> > > > current
>> > > > >> > > > configuration with Camel to a Spring Web Service.
>>Concerns
>> > over
>> > > > failover
>> > > > >> > > > and load balancing would be mitigated by the Cassandra DB
>> > > > (running on 9160)
>> > > > >> > > > and Storm (which depends on an instance of Zookeeper on
>>2181).
>> > > > Before I
>> > > > >> > > > start lancing any windmills I thought it'd be nice to get
>> > input
>> > > > on why the
>> > > > >> > > > design choice was made to base the application around
>>Camel?
>> > > > >> > > > I think that tracking the flow of an activity through the
>> > > > application will
>> > > > >> > > > be much easier if we use Spring MVC. I'll set up a new
>>branch
>> > > > using the MVC
>> > > > >> > > > architecture and I'll keep the community updated as I go
>> > along.
>> > > > >> > > > Danny
>> > > > >> >
>> > > > >>
>> > > > >
>> > > >
>> >
>> >
>
CONFIDENTIALITY NOTICE: This e-mail, along with any documents, files, or attachments, may contain information that is confidential, privileged, or otherwise exempt from disclosure. If you are not the intended recipient or person responsible for delivering it to the intended recipient, you are hereby notified that any disclosure, copying, printing, distribution or use of any information contained in or attached to this e-mail is strictly prohibited. If you have received this e-mail in error, please immediately notify the sender and delete the original e-mail and its attachments without reading, printing, or saving in any manner. This e-mail message should not be interpreted to include a digital or electronic signature that can be used to authenticate an agreement, contract or other legal document, nor to reflect an intention to be bound to any legally-binding agreement or contract. Your cooperation is appreciated. Thank you.
RE: Using Camel for Routing
Posted by Danny Sullivan <ds...@hotmail.com>.
That sounds like a good idea Jason, I'll set up the vote!
> Date: Thu, 24 Oct 2013 09:55:10 -0700
> Subject: Re: Using Camel for Routing
> From: chris@cxtsoftware.com
> To: dev@streams.incubator.apache.org
>
> I agree with Jason personally, flexibility and scalability trumps simple.
>
> That being said, I honestly don't have enough time right now to spend
> developing on my vision so it's unfair of me to push too hard in that
> direction. If the active developers want to go down the single service path
> maybe that's the approach for now and we can potentially look at
> alternatives down the road.
>
> Chris
>
>
> On Thu, Oct 24, 2013 at 8:08 AM, Jason Letourneau
> <jl...@gmail.com>wrote:
>
> > I think this comes down to a fundamental question - how flexible vs
> > simple do people want the deployment and configuration of Streams to
> > be?
> >
> > Personally, as I've said, I prefer flexible over simple, but I think
> > the current architecture reaches a happy medium with the WAR
> > deployment option.
> >
> > If the majority wants simple over flexible, then perhaps the web
> > service architecture you've proposed is the way to go. I think its
> > worth putting to vote.
> >
> > Jason
> >
> > On Wed, Oct 23, 2013 at 11:29 AM, Danny Sullivan <ds...@hotmail.com>
> > wrote:
> > > Hey Guys,
> > >
> > > I'm trying to do a little more research in the efficiency between both
> > the
> > > web-service and the camel implementations of streams. I set up Apache
> > JMeter
> > > (http://jmeter.apache.org/) to do some load testing on both setups. I
> > set up
> > > JMeter to make 1000 POST requests sending valid activitystrea.msformatted
> > > json to a single publisher url over the course of 5 seconds. I've
> > attached
> > > .csv files that give the average response the results, but I found that
> > the
> > > web service was .4 seconds faster on average. My concern is that these
> > > results may not be consistent for the bigger use cases, and Camel may be
> > > faster when deployed to multiple servers. It seems though that for early
> > use
> > > cases, the web service would be faster.
> > >
> > > I do want to make Streams as fast and as scalable as possible and I
> > > appreciate the patience you guys are giving me through through this big
> > > design change proposal. I would be happy to try a different load test
> > setup
> > > and, as always, would be happy to hear thoughts on the two
> > implementations.
> > >
> > > Thanks
> > >
> > > -Danny
> > >
> > >> Date: Wed, 16 Oct 2013 18:49:02 -0400
> > >
> > >> Subject: Re: Using Camel for Routing
> > >> From: jletourneau80@gmail.com
> > >> To: dev@streams.incubator.apache.org
> > >>
> > >> I think it depends on how you define 'no problem' - deployment
> > >> infrastructure is a big part of that - messaging provides more
> > flexibility
> > >> in deployment and capability additions at different layers of the
> > >> application vs a single web app and a shotgun approach to evolving it
> > >>
> > >> I agree it should still be on the table as we define what is most
> > >> important
> > >> to this community
> > >>
> > >> On Wednesday, October 16, 2013, Danny Sullivan wrote:
> > >>
> > >> > I think if we sacrifice scalability in the long run by switching from
> > >> > Camel to a web service then we should absolutely continue to use
> > >> > messaging
> > >> > and Camel. I'm confident, however, that applications exist that have a
> > >> > web
> > >> > service based model that service millions of requests and have no
> > >> > problem
> > >> > scaling without the help of Camel or any other messaging framework.
> > >> > I'm going to keep the web-service branch separate for now, but I'd
> > still
> > >> > like to keep the option on the table until we explore building
> > scalable
> > >> > applications without the use of Camel. Though I can see the benefits
> > of
> > >> > using Camel, I think the complexity it adds makes it worth taking a
> > look
> > >> > at
> > >> > other options.
> > >> > -Danny
> > >> >
> > >> > > Date: Wed, 16 Oct 2013 16:26:43 -0400
> > >> > > Subject: Re: Using Camel for Routing
> > >> > > From: jletourneau80@gmail.com
> > >> > > To: dev@streams.incubator.apache.org
> > >> > >
> > >> > > Yeah the more I think through this in terms of implications of a
> > sole
> > >> > > web app - I am skepticle that it scales to interesting use cases. I
> > >> > > am not interested in streams to monitor "a few" facebook friends, I
> > am
> > >> > > interested in streams to monitor millions of events and activities
> > >> > > generated from web site activity (clicks or tweets whatever) or
> > >> > > enterprise assets across the globe.
> > >> > >
> > >> > > On Wed, Oct 16, 2013 at 4:15 PM, Chris Geer <ch...@cxtsoftware.com>
> > >> > wrote:
> > >> > > > On Wed, Oct 16, 2013 at 11:58 AM, Danny Sullivan <
> > >> > dsullivan7@hotmail.com>wrote:
> > >> > > >
> > >> > > >> I see the value in Streams as the ability of subscribers to get
> > >> > activity
> > >> > > >> based on filters.
> > >> > > >
> > >> > > >
> > >> > > > I agree
> > >> > > >
> > >> > > >
> > >> > > >> To be clear, I do not want to eliminate processing, but do want
> > to
> > >> > make
> > >> > > >> the focus of processing tailored to the subscribers' filters. My
> > >> > > >> more
> > >> > grand
> > >> > > >> vision is that only the most relevant activity will be returned
> > to
> > >> > > >> the
> > >> > > >> Subscriber and similar activity will be aggregated into single
> > >> > activities
> > >> > > >> (this would be a little down the road). This processing would
> > >> > > >> happen
> > >> > after
> > >> > > >> the activity is stored in the DB.
> > >> > > >>
> > >> > > >
> > >> > > > This brings me back to the concern about having it all within one
> > >> > > > web
> > >> > app.
> > >> > > > If the webapp accepts inputs, aggregates/filters it and manages
> > it's
> > >> > own
> > >> > > > subscribers, how you do scale it, or provide redundancy? The
> > beauty
> > >> > > > of
> > >> > > > using a message based system (i.e. Camel...along with Storm
> > >> > > > possibly)
> > >> > is
> > >> > > > that it allows you to partition the application however you want.
> > So
> > >> > if you
> > >> > > > want to have your ingest web service running on one machine and
> > your
> > >> > > > subscriber service running somewhere else and a third machine for
> > >> > > > your
> > >> > > > really resource intensive aggregation component, you can. With the
> > >> > > > web
> > >> > app
> > >> > > > you could do vertical scaling on one machine, or load balancing of
> > >> > > > all
> > >> > > > components across multiple machines but it doesn't give you the
> > save
> > >> > > > flexibility.
> > >> > > >
> > >> > > >
> > >> > > >> Do you imagine that Streams will come packaged with some
> > processing
> > >> > > >> components that will format other non-activitystrea.ms formatted
> > >> > json? I
> > >> > > >> do think it'd be pretty cool if you could fire up Streams, input
> > a
> > >> > couple
> > >> > > >> Facebook Friends and people you follow on Twitter, and have all
> > of
> > >> > that
> > >> > > >> activity posted in one place. That might be better suited for a
> > new
> > >> > > >> separate module (something like streams-format?) if we decide on
> > >> > > >> the
> > >> > > >> web-application route.
> > >> > > >>
> > >> > > >
> > >> > > > I think it has to come with something to entice more users. If
> > >> > everyone has
> > >> > > > to build their own translation to even run "Hello World" it will
> > be
> > >> > > > a
> > >> > > > larger barrier to entry.
> > >> > > >
> > >> > > >
> > >> > > >> -Danny
> > >> > > >>
> > >> > > >> > Date: Wed, 16 Oct 2013 11:22:51 -0700
> > >> > > >> > Subject: Re: Using Camel for Routing
> > >> > > >> > From: chris@cxtsoftware.com
> > >> > > >> > To: dev@streams.incubator.apache.org
> > >> > > >> >
> > >> > > >> > On Wed, Oct 16, 2013 at 11:12 AM, Danny Sullivan <
> > >> > dsullivan7@hotmail.com
> > >> > > >> >wrote:
> > >> > > >> >
> > >> > > >> > > I think what is happening here is that we have different
> > ideas
> > >> > about
> > >> > > >> the
> > >> > > >> > > architecture of the application.
> > >> > > >> > > Here's what I'm thinking:
> > >> > > >> > > *All processing of activity would happen BEFORE it's posted
> > to
> > >> > > >> Streams*All
> > >> > > >> > > activity posted to Streams is in activitystrea.ms format
> > >> > > >> > > The Camel/EIP/OSGi way:
> > >> > > >>
> >
Re: Using Camel for Routing
Posted by Chris Geer <ch...@cxtsoftware.com>.
I agree with Jason personally, flexibility and scalability trumps simple.
That being said, I honestly don't have enough time right now to spend
developing on my vision so it's unfair of me to push too hard in that
direction. If the active developers want to go down the single service path
maybe that's the approach for now and we can potentially look at
alternatives down the road.
Chris
On Thu, Oct 24, 2013 at 8:08 AM, Jason Letourneau
<jl...@gmail.com>wrote:
> I think this comes down to a fundamental question - how flexible vs
> simple do people want the deployment and configuration of Streams to
> be?
>
> Personally, as I've said, I prefer flexible over simple, but I think
> the current architecture reaches a happy medium with the WAR
> deployment option.
>
> If the majority wants simple over flexible, then perhaps the web
> service architecture you've proposed is the way to go. I think its
> worth putting to vote.
>
> Jason
>
> On Wed, Oct 23, 2013 at 11:29 AM, Danny Sullivan <ds...@hotmail.com>
> wrote:
> > Hey Guys,
> >
> > I'm trying to do a little more research in the efficiency between both
> the
> > web-service and the camel implementations of streams. I set up Apache
> JMeter
> > (http://jmeter.apache.org/) to do some load testing on both setups. I
> set up
> > JMeter to make 1000 POST requests sending valid activitystrea.msformatted
> > json to a single publisher url over the course of 5 seconds. I've
> attached
> > .csv files that give the average response the results, but I found that
> the
> > web service was .4 seconds faster on average. My concern is that these
> > results may not be consistent for the bigger use cases, and Camel may be
> > faster when deployed to multiple servers. It seems though that for early
> use
> > cases, the web service would be faster.
> >
> > I do want to make Streams as fast and as scalable as possible and I
> > appreciate the patience you guys are giving me through through this big
> > design change proposal. I would be happy to try a different load test
> setup
> > and, as always, would be happy to hear thoughts on the two
> implementations.
> >
> > Thanks
> >
> > -Danny
> >
> >> Date: Wed, 16 Oct 2013 18:49:02 -0400
> >
> >> Subject: Re: Using Camel for Routing
> >> From: jletourneau80@gmail.com
> >> To: dev@streams.incubator.apache.org
> >>
> >> I think it depends on how you define 'no problem' - deployment
> >> infrastructure is a big part of that - messaging provides more
> flexibility
> >> in deployment and capability additions at different layers of the
> >> application vs a single web app and a shotgun approach to evolving it
> >>
> >> I agree it should still be on the table as we define what is most
> >> important
> >> to this community
> >>
> >> On Wednesday, October 16, 2013, Danny Sullivan wrote:
> >>
> >> > I think if we sacrifice scalability in the long run by switching from
> >> > Camel to a web service then we should absolutely continue to use
> >> > messaging
> >> > and Camel. I'm confident, however, that applications exist that have a
> >> > web
> >> > service based model that service millions of requests and have no
> >> > problem
> >> > scaling without the help of Camel or any other messaging framework.
> >> > I'm going to keep the web-service branch separate for now, but I'd
> still
> >> > like to keep the option on the table until we explore building
> scalable
> >> > applications without the use of Camel. Though I can see the benefits
> of
> >> > using Camel, I think the complexity it adds makes it worth taking a
> look
> >> > at
> >> > other options.
> >> > -Danny
> >> >
> >> > > Date: Wed, 16 Oct 2013 16:26:43 -0400
> >> > > Subject: Re: Using Camel for Routing
> >> > > From: jletourneau80@gmail.com
> >> > > To: dev@streams.incubator.apache.org
> >> > >
> >> > > Yeah the more I think through this in terms of implications of a
> sole
> >> > > web app - I am skepticle that it scales to interesting use cases. I
> >> > > am not interested in streams to monitor "a few" facebook friends, I
> am
> >> > > interested in streams to monitor millions of events and activities
> >> > > generated from web site activity (clicks or tweets whatever) or
> >> > > enterprise assets across the globe.
> >> > >
> >> > > On Wed, Oct 16, 2013 at 4:15 PM, Chris Geer <ch...@cxtsoftware.com>
> >> > wrote:
> >> > > > On Wed, Oct 16, 2013 at 11:58 AM, Danny Sullivan <
> >> > dsullivan7@hotmail.com>wrote:
> >> > > >
> >> > > >> I see the value in Streams as the ability of subscribers to get
> >> > activity
> >> > > >> based on filters.
> >> > > >
> >> > > >
> >> > > > I agree
> >> > > >
> >> > > >
> >> > > >> To be clear, I do not want to eliminate processing, but do want
> to
> >> > make
> >> > > >> the focus of processing tailored to the subscribers' filters. My
> >> > > >> more
> >> > grand
> >> > > >> vision is that only the most relevant activity will be returned
> to
> >> > > >> the
> >> > > >> Subscriber and similar activity will be aggregated into single
> >> > activities
> >> > > >> (this would be a little down the road). This processing would
> >> > > >> happen
> >> > after
> >> > > >> the activity is stored in the DB.
> >> > > >>
> >> > > >
> >> > > > This brings me back to the concern about having it all within one
> >> > > > web
> >> > app.
> >> > > > If the webapp accepts inputs, aggregates/filters it and manages
> it's
> >> > own
> >> > > > subscribers, how you do scale it, or provide redundancy? The
> beauty
> >> > > > of
> >> > > > using a message based system (i.e. Camel...along with Storm
> >> > > > possibly)
> >> > is
> >> > > > that it allows you to partition the application however you want.
> So
> >> > if you
> >> > > > want to have your ingest web service running on one machine and
> your
> >> > > > subscriber service running somewhere else and a third machine for
> >> > > > your
> >> > > > really resource intensive aggregation component, you can. With the
> >> > > > web
> >> > app
> >> > > > you could do vertical scaling on one machine, or load balancing of
> >> > > > all
> >> > > > components across multiple machines but it doesn't give you the
> save
> >> > > > flexibility.
> >> > > >
> >> > > >
> >> > > >> Do you imagine that Streams will come packaged with some
> processing
> >> > > >> components that will format other non-activitystrea.ms formatted
> >> > json? I
> >> > > >> do think it'd be pretty cool if you could fire up Streams, input
> a
> >> > couple
> >> > > >> Facebook Friends and people you follow on Twitter, and have all
> of
> >> > that
> >> > > >> activity posted in one place. That might be better suited for a
> new
> >> > > >> separate module (something like streams-format?) if we decide on
> >> > > >> the
> >> > > >> web-application route.
> >> > > >>
> >> > > >
> >> > > > I think it has to come with something to entice more users. If
> >> > everyone has
> >> > > > to build their own translation to even run "Hello World" it will
> be
> >> > > > a
> >> > > > larger barrier to entry.
> >> > > >
> >> > > >
> >> > > >> -Danny
> >> > > >>
> >> > > >> > Date: Wed, 16 Oct 2013 11:22:51 -0700
> >> > > >> > Subject: Re: Using Camel for Routing
> >> > > >> > From: chris@cxtsoftware.com
> >> > > >> > To: dev@streams.incubator.apache.org
> >> > > >> >
> >> > > >> > On Wed, Oct 16, 2013 at 11:12 AM, Danny Sullivan <
> >> > dsullivan7@hotmail.com
> >> > > >> >wrote:
> >> > > >> >
> >> > > >> > > I think what is happening here is that we have different
> ideas
> >> > about
> >> > > >> the
> >> > > >> > > architecture of the application.
> >> > > >> > > Here's what I'm thinking:
> >> > > >> > > *All processing of activity would happen BEFORE it's posted
> to
> >> > > >> Streams*All
> >> > > >> > > activity posted to Streams is in activitystrea.ms format
> >> > > >> > > The Camel/EIP/OSGi way:
> >> > > >>
>
Re: Using Camel for Routing
Posted by Jason Letourneau <jl...@gmail.com>.
I think this comes down to a fundamental question - how flexible vs
simple do people want the deployment and configuration of Streams to
be?
Personally, as I've said, I prefer flexible over simple, but I think
the current architecture reaches a happy medium with the WAR
deployment option.
If the majority wants simple over flexible, then perhaps the web
service architecture you've proposed is the way to go. I think its
worth putting to vote.
Jason
On Wed, Oct 23, 2013 at 11:29 AM, Danny Sullivan <ds...@hotmail.com> wrote:
> Hey Guys,
>
> I'm trying to do a little more research in the efficiency between both the
> web-service and the camel implementations of streams. I set up Apache JMeter
> (http://jmeter.apache.org/) to do some load testing on both setups. I set up
> JMeter to make 1000 POST requests sending valid activitystrea.ms formatted
> json to a single publisher url over the course of 5 seconds. I've attached
> .csv files that give the average response the results, but I found that the
> web service was .4 seconds faster on average. My concern is that these
> results may not be consistent for the bigger use cases, and Camel may be
> faster when deployed to multiple servers. It seems though that for early use
> cases, the web service would be faster.
>
> I do want to make Streams as fast and as scalable as possible and I
> appreciate the patience you guys are giving me through through this big
> design change proposal. I would be happy to try a different load test setup
> and, as always, would be happy to hear thoughts on the two implementations.
>
> Thanks
>
> -Danny
>
>> Date: Wed, 16 Oct 2013 18:49:02 -0400
>
>> Subject: Re: Using Camel for Routing
>> From: jletourneau80@gmail.com
>> To: dev@streams.incubator.apache.org
>>
>> I think it depends on how you define 'no problem' - deployment
>> infrastructure is a big part of that - messaging provides more flexibility
>> in deployment and capability additions at different layers of the
>> application vs a single web app and a shotgun approach to evolving it
>>
>> I agree it should still be on the table as we define what is most
>> important
>> to this community
>>
>> On Wednesday, October 16, 2013, Danny Sullivan wrote:
>>
>> > I think if we sacrifice scalability in the long run by switching from
>> > Camel to a web service then we should absolutely continue to use
>> > messaging
>> > and Camel. I'm confident, however, that applications exist that have a
>> > web
>> > service based model that service millions of requests and have no
>> > problem
>> > scaling without the help of Camel or any other messaging framework.
>> > I'm going to keep the web-service branch separate for now, but I'd still
>> > like to keep the option on the table until we explore building scalable
>> > applications without the use of Camel. Though I can see the benefits of
>> > using Camel, I think the complexity it adds makes it worth taking a look
>> > at
>> > other options.
>> > -Danny
>> >
>> > > Date: Wed, 16 Oct 2013 16:26:43 -0400
>> > > Subject: Re: Using Camel for Routing
>> > > From: jletourneau80@gmail.com
>> > > To: dev@streams.incubator.apache.org
>> > >
>> > > Yeah the more I think through this in terms of implications of a sole
>> > > web app - I am skepticle that it scales to interesting use cases. I
>> > > am not interested in streams to monitor "a few" facebook friends, I am
>> > > interested in streams to monitor millions of events and activities
>> > > generated from web site activity (clicks or tweets whatever) or
>> > > enterprise assets across the globe.
>> > >
>> > > On Wed, Oct 16, 2013 at 4:15 PM, Chris Geer <ch...@cxtsoftware.com>
>> > wrote:
>> > > > On Wed, Oct 16, 2013 at 11:58 AM, Danny Sullivan <
>> > dsullivan7@hotmail.com>wrote:
>> > > >
>> > > >> I see the value in Streams as the ability of subscribers to get
>> > activity
>> > > >> based on filters.
>> > > >
>> > > >
>> > > > I agree
>> > > >
>> > > >
>> > > >> To be clear, I do not want to eliminate processing, but do want to
>> > make
>> > > >> the focus of processing tailored to the subscribers' filters. My
>> > > >> more
>> > grand
>> > > >> vision is that only the most relevant activity will be returned to
>> > > >> the
>> > > >> Subscriber and similar activity will be aggregated into single
>> > activities
>> > > >> (this would be a little down the road). This processing would
>> > > >> happen
>> > after
>> > > >> the activity is stored in the DB.
>> > > >>
>> > > >
>> > > > This brings me back to the concern about having it all within one
>> > > > web
>> > app.
>> > > > If the webapp accepts inputs, aggregates/filters it and manages it's
>> > own
>> > > > subscribers, how you do scale it, or provide redundancy? The beauty
>> > > > of
>> > > > using a message based system (i.e. Camel...along with Storm
>> > > > possibly)
>> > is
>> > > > that it allows you to partition the application however you want. So
>> > if you
>> > > > want to have your ingest web service running on one machine and your
>> > > > subscriber service running somewhere else and a third machine for
>> > > > your
>> > > > really resource intensive aggregation component, you can. With the
>> > > > web
>> > app
>> > > > you could do vertical scaling on one machine, or load balancing of
>> > > > all
>> > > > components across multiple machines but it doesn't give you the save
>> > > > flexibility.
>> > > >
>> > > >
>> > > >> Do you imagine that Streams will come packaged with some processing
>> > > >> components that will format other non-activitystrea.ms formatted
>> > json? I
>> > > >> do think it'd be pretty cool if you could fire up Streams, input a
>> > couple
>> > > >> Facebook Friends and people you follow on Twitter, and have all of
>> > that
>> > > >> activity posted in one place. That might be better suited for a new
>> > > >> separate module (something like streams-format?) if we decide on
>> > > >> the
>> > > >> web-application route.
>> > > >>
>> > > >
>> > > > I think it has to come with something to entice more users. If
>> > everyone has
>> > > > to build their own translation to even run "Hello World" it will be
>> > > > a
>> > > > larger barrier to entry.
>> > > >
>> > > >
>> > > >> -Danny
>> > > >>
>> > > >> > Date: Wed, 16 Oct 2013 11:22:51 -0700
>> > > >> > Subject: Re: Using Camel for Routing
>> > > >> > From: chris@cxtsoftware.com
>> > > >> > To: dev@streams.incubator.apache.org
>> > > >> >
>> > > >> > On Wed, Oct 16, 2013 at 11:12 AM, Danny Sullivan <
>> > dsullivan7@hotmail.com
>> > > >> >wrote:
>> > > >> >
>> > > >> > > I think what is happening here is that we have different ideas
>> > about
>> > > >> the
>> > > >> > > architecture of the application.
>> > > >> > > Here's what I'm thinking:
>> > > >> > > *All processing of activity would happen BEFORE it's posted to
>> > > >> Streams*All
>> > > >> > > activity posted to Streams is in activitystrea.ms format
>> > > >> > > The Camel/EIP/OSGi way:
>> > > >>
RE: Using Camel for Routing
Posted by Danny Sullivan <ds...@hotmail.com>.
Doesn't look like the attachment picked up. Here it is again.
-Danny
From: dsullivan7@hotmail.com
To: dev@streams.incubator.apache.org
Subject: RE: Using Camel for Routing
Date: Wed, 23 Oct 2013 11:29:53 -0400
Hey Guys,
I'm trying to do a little more research in the efficiency between both the web-service and the camel implementations of streams. I set up Apache JMeter (http://jmeter.apache.org/) to do some load testing on both setups. I set up JMeter to make 1000 POST requests sending valid activitystrea.ms formatted json to a single publisher url over the course of 5 seconds. I've attached .csv files that give the average response the results, but I found that the web service was .4 seconds faster on average. My concern is that these results may not be consistent for the bigger use cases, and Camel may be faster when deployed to multiple servers. It seems though that for early use cases, the web service would be faster.
I do want to make Streams as fast and as scalable as possible and I appreciate the patience you guys are giving me through through this big design change proposal. I would be happy to try a different load test setup and, as always, would be happy to hear thoughts on the two implementations.
Thanks
-Danny
> Date: Wed, 16 Oct 2013 18:49:02 -0400
> Subject: Re: Using Camel for Routing
> From: jletourneau80@gmail.com
> To: dev@streams.incubator.apache.org
>
> I think it depends on how you define 'no problem' - deployment
> infrastructure is a big part of that - messaging provides more flexibility
> in deployment and capability additions at different layers of the
> application vs a single web app and a shotgun approach to evolving it
>
> I agree it should still be on the table as we define what is most important
> to this community
>
> On Wednesday, October 16, 2013, Danny Sullivan wrote:
>
> > I think if we sacrifice scalability in the long run by switching from
> > Camel to a web service then we should absolutely continue to use messaging
> > and Camel. I'm confident, however, that applications exist that have a web
> > service based model that service millions of requests and have no problem
> > scaling without the help of Camel or any other messaging framework.
> > I'm going to keep the web-service branch separate for now, but I'd still
> > like to keep the option on the table until we explore building scalable
> > applications without the use of Camel. Though I can see the benefits of
> > using Camel, I think the complexity it adds makes it worth taking a look at
> > other options.
> > -Danny
> >
> > > Date: Wed, 16 Oct 2013 16:26:43 -0400
> > > Subject: Re: Using Camel for Routing
> > > From: jletourneau80@gmail.com
> > > To: dev@streams.incubator.apache.org
> > >
> > > Yeah the more I think through this in terms of implications of a sole
> > > web app - I am skepticle that it scales to interesting use cases. I
> > > am not interested in streams to monitor "a few" facebook friends, I am
> > > interested in streams to monitor millions of events and activities
> > > generated from web site activity (clicks or tweets whatever) or
> > > enterprise assets across the globe.
> > >
> > > On Wed, Oct 16, 2013 at 4:15 PM, Chris Geer <ch...@cxtsoftware.com>
> > wrote:
> > > > On Wed, Oct 16, 2013 at 11:58 AM, Danny Sullivan <
> > dsullivan7@hotmail.com>wrote:
> > > >
> > > >> I see the value in Streams as the ability of subscribers to get
> > activity
> > > >> based on filters.
> > > >
> > > >
> > > > I agree
> > > >
> > > >
> > > >> To be clear, I do not want to eliminate processing, but do want to
> > make
> > > >> the focus of processing tailored to the subscribers' filters. My more
> > grand
> > > >> vision is that only the most relevant activity will be returned to the
> > > >> Subscriber and similar activity will be aggregated into single
> > activities
> > > >> (this would be a little down the road). This processing would happen
> > after
> > > >> the activity is stored in the DB.
> > > >>
> > > >
> > > > This brings me back to the concern about having it all within one web
> > app.
> > > > If the webapp accepts inputs, aggregates/filters it and manages it's
> > own
> > > > subscribers, how you do scale it, or provide redundancy? The beauty of
> > > > using a message based system (i.e. Camel...along with Storm possibly)
> > is
> > > > that it allows you to partition the application however you want. So
> > if you
> > > > want to have your ingest web service running on one machine and your
> > > > subscriber service running somewhere else and a third machine for your
> > > > really resource intensive aggregation component, you can. With the web
> > app
> > > > you could do vertical scaling on one machine, or load balancing of all
> > > > components across multiple machines but it doesn't give you the save
> > > > flexibility.
> > > >
> > > >
> > > >> Do you imagine that Streams will come packaged with some processing
> > > >> components that will format other non-activitystrea.ms formatted
> > json? I
> > > >> do think it'd be pretty cool if you could fire up Streams, input a
> > couple
> > > >> Facebook Friends and people you follow on Twitter, and have all of
> > that
> > > >> activity posted in one place. That might be better suited for a new
> > > >> separate module (something like streams-format?) if we decide on the
> > > >> web-application route.
> > > >>
> > > >
> > > > I think it has to come with something to entice more users. If
> > everyone has
> > > > to build their own translation to even run "Hello World" it will be a
> > > > larger barrier to entry.
> > > >
> > > >
> > > >> -Danny
> > > >>
> > > >> > Date: Wed, 16 Oct 2013 11:22:51 -0700
> > > >> > Subject: Re: Using Camel for Routing
> > > >> > From: chris@cxtsoftware.com
> > > >> > To: dev@streams.incubator.apache.org
> > > >> >
> > > >> > On Wed, Oct 16, 2013 at 11:12 AM, Danny Sullivan <
> > dsullivan7@hotmail.com
> > > >> >wrote:
> > > >> >
> > > >> > > I think what is happening here is that we have different ideas
> > about
> > > >> the
> > > >> > > architecture of the application.
> > > >> > > Here's what I'm thinking:
> > > >> > > *All processing of activity would happen BEFORE it's posted to
> > > >> Streams*All
> > > >> > > activity posted to Streams is in activitystrea.ms format
> > > >> > > The Camel/EIP/OSGi way:
> > > >>
RE: Using Camel for Routing
Posted by Danny Sullivan <ds...@hotmail.com>.
Hey Guys,
I'm trying to do a little more research in the efficiency between both the web-service and the camel implementations of streams. I set up Apache JMeter (http://jmeter.apache.org/) to do some load testing on both setups. I set up JMeter to make 1000 POST requests sending valid activitystrea.ms formatted json to a single publisher url over the course of 5 seconds. I've attached .csv files that give the average response the results, but I found that the web service was .4 seconds faster on average. My concern is that these results may not be consistent for the bigger use cases, and Camel may be faster when deployed to multiple servers. It seems though that for early use cases, the web service would be faster.
I do want to make Streams as fast and as scalable as possible and I appreciate the patience you guys are giving me through through this big design change proposal. I would be happy to try a different load test setup and, as always, would be happy to hear thoughts on the two implementations.
Thanks
-Danny
> Date: Wed, 16 Oct 2013 18:49:02 -0400
> Subject: Re: Using Camel for Routing
> From: jletourneau80@gmail.com
> To: dev@streams.incubator.apache.org
>
> I think it depends on how you define 'no problem' - deployment
> infrastructure is a big part of that - messaging provides more flexibility
> in deployment and capability additions at different layers of the
> application vs a single web app and a shotgun approach to evolving it
>
> I agree it should still be on the table as we define what is most important
> to this community
>
> On Wednesday, October 16, 2013, Danny Sullivan wrote:
>
> > I think if we sacrifice scalability in the long run by switching from
> > Camel to a web service then we should absolutely continue to use messaging
> > and Camel. I'm confident, however, that applications exist that have a web
> > service based model that service millions of requests and have no problem
> > scaling without the help of Camel or any other messaging framework.
> > I'm going to keep the web-service branch separate for now, but I'd still
> > like to keep the option on the table until we explore building scalable
> > applications without the use of Camel. Though I can see the benefits of
> > using Camel, I think the complexity it adds makes it worth taking a look at
> > other options.
> > -Danny
> >
> > > Date: Wed, 16 Oct 2013 16:26:43 -0400
> > > Subject: Re: Using Camel for Routing
> > > From: jletourneau80@gmail.com
> > > To: dev@streams.incubator.apache.org
> > >
> > > Yeah the more I think through this in terms of implications of a sole
> > > web app - I am skepticle that it scales to interesting use cases. I
> > > am not interested in streams to monitor "a few" facebook friends, I am
> > > interested in streams to monitor millions of events and activities
> > > generated from web site activity (clicks or tweets whatever) or
> > > enterprise assets across the globe.
> > >
> > > On Wed, Oct 16, 2013 at 4:15 PM, Chris Geer <ch...@cxtsoftware.com>
> > wrote:
> > > > On Wed, Oct 16, 2013 at 11:58 AM, Danny Sullivan <
> > dsullivan7@hotmail.com>wrote:
> > > >
> > > >> I see the value in Streams as the ability of subscribers to get
> > activity
> > > >> based on filters.
> > > >
> > > >
> > > > I agree
> > > >
> > > >
> > > >> To be clear, I do not want to eliminate processing, but do want to
> > make
> > > >> the focus of processing tailored to the subscribers' filters. My more
> > grand
> > > >> vision is that only the most relevant activity will be returned to the
> > > >> Subscriber and similar activity will be aggregated into single
> > activities
> > > >> (this would be a little down the road). This processing would happen
> > after
> > > >> the activity is stored in the DB.
> > > >>
> > > >
> > > > This brings me back to the concern about having it all within one web
> > app.
> > > > If the webapp accepts inputs, aggregates/filters it and manages it's
> > own
> > > > subscribers, how you do scale it, or provide redundancy? The beauty of
> > > > using a message based system (i.e. Camel...along with Storm possibly)
> > is
> > > > that it allows you to partition the application however you want. So
> > if you
> > > > want to have your ingest web service running on one machine and your
> > > > subscriber service running somewhere else and a third machine for your
> > > > really resource intensive aggregation component, you can. With the web
> > app
> > > > you could do vertical scaling on one machine, or load balancing of all
> > > > components across multiple machines but it doesn't give you the save
> > > > flexibility.
> > > >
> > > >
> > > >> Do you imagine that Streams will come packaged with some processing
> > > >> components that will format other non-activitystrea.ms formatted
> > json? I
> > > >> do think it'd be pretty cool if you could fire up Streams, input a
> > couple
> > > >> Facebook Friends and people you follow on Twitter, and have all of
> > that
> > > >> activity posted in one place. That might be better suited for a new
> > > >> separate module (something like streams-format?) if we decide on the
> > > >> web-application route.
> > > >>
> > > >
> > > > I think it has to come with something to entice more users. If
> > everyone has
> > > > to build their own translation to even run "Hello World" it will be a
> > > > larger barrier to entry.
> > > >
> > > >
> > > >> -Danny
> > > >>
> > > >> > Date: Wed, 16 Oct 2013 11:22:51 -0700
> > > >> > Subject: Re: Using Camel for Routing
> > > >> > From: chris@cxtsoftware.com
> > > >> > To: dev@streams.incubator.apache.org
> > > >> >
> > > >> > On Wed, Oct 16, 2013 at 11:12 AM, Danny Sullivan <
> > dsullivan7@hotmail.com
> > > >> >wrote:
> > > >> >
> > > >> > > I think what is happening here is that we have different ideas
> > about
> > > >> the
> > > >> > > architecture of the application.
> > > >> > > Here's what I'm thinking:
> > > >> > > *All processing of activity would happen BEFORE it's posted to
> > > >> Streams*All
> > > >> > > activity posted to Streams is in activitystrea.ms format
> > > >> > > The Camel/EIP/OSGi way:
> > > >>
Re: Using Camel for Routing
Posted by Jason Letourneau <jl...@gmail.com>.
I think it depends on how you define 'no problem' - deployment
infrastructure is a big part of that - messaging provides more flexibility
in deployment and capability additions at different layers of the
application vs a single web app and a shotgun approach to evolving it
I agree it should still be on the table as we define what is most important
to this community
On Wednesday, October 16, 2013, Danny Sullivan wrote:
> I think if we sacrifice scalability in the long run by switching from
> Camel to a web service then we should absolutely continue to use messaging
> and Camel. I'm confident, however, that applications exist that have a web
> service based model that service millions of requests and have no problem
> scaling without the help of Camel or any other messaging framework.
> I'm going to keep the web-service branch separate for now, but I'd still
> like to keep the option on the table until we explore building scalable
> applications without the use of Camel. Though I can see the benefits of
> using Camel, I think the complexity it adds makes it worth taking a look at
> other options.
> -Danny
>
> > Date: Wed, 16 Oct 2013 16:26:43 -0400
> > Subject: Re: Using Camel for Routing
> > From: jletourneau80@gmail.com
> > To: dev@streams.incubator.apache.org
> >
> > Yeah the more I think through this in terms of implications of a sole
> > web app - I am skepticle that it scales to interesting use cases. I
> > am not interested in streams to monitor "a few" facebook friends, I am
> > interested in streams to monitor millions of events and activities
> > generated from web site activity (clicks or tweets whatever) or
> > enterprise assets across the globe.
> >
> > On Wed, Oct 16, 2013 at 4:15 PM, Chris Geer <ch...@cxtsoftware.com>
> wrote:
> > > On Wed, Oct 16, 2013 at 11:58 AM, Danny Sullivan <
> dsullivan7@hotmail.com>wrote:
> > >
> > >> I see the value in Streams as the ability of subscribers to get
> activity
> > >> based on filters.
> > >
> > >
> > > I agree
> > >
> > >
> > >> To be clear, I do not want to eliminate processing, but do want to
> make
> > >> the focus of processing tailored to the subscribers' filters. My more
> grand
> > >> vision is that only the most relevant activity will be returned to the
> > >> Subscriber and similar activity will be aggregated into single
> activities
> > >> (this would be a little down the road). This processing would happen
> after
> > >> the activity is stored in the DB.
> > >>
> > >
> > > This brings me back to the concern about having it all within one web
> app.
> > > If the webapp accepts inputs, aggregates/filters it and manages it's
> own
> > > subscribers, how you do scale it, or provide redundancy? The beauty of
> > > using a message based system (i.e. Camel...along with Storm possibly)
> is
> > > that it allows you to partition the application however you want. So
> if you
> > > want to have your ingest web service running on one machine and your
> > > subscriber service running somewhere else and a third machine for your
> > > really resource intensive aggregation component, you can. With the web
> app
> > > you could do vertical scaling on one machine, or load balancing of all
> > > components across multiple machines but it doesn't give you the save
> > > flexibility.
> > >
> > >
> > >> Do you imagine that Streams will come packaged with some processing
> > >> components that will format other non-activitystrea.ms formatted
> json? I
> > >> do think it'd be pretty cool if you could fire up Streams, input a
> couple
> > >> Facebook Friends and people you follow on Twitter, and have all of
> that
> > >> activity posted in one place. That might be better suited for a new
> > >> separate module (something like streams-format?) if we decide on the
> > >> web-application route.
> > >>
> > >
> > > I think it has to come with something to entice more users. If
> everyone has
> > > to build their own translation to even run "Hello World" it will be a
> > > larger barrier to entry.
> > >
> > >
> > >> -Danny
> > >>
> > >> > Date: Wed, 16 Oct 2013 11:22:51 -0700
> > >> > Subject: Re: Using Camel for Routing
> > >> > From: chris@cxtsoftware.com
> > >> > To: dev@streams.incubator.apache.org
> > >> >
> > >> > On Wed, Oct 16, 2013 at 11:12 AM, Danny Sullivan <
> dsullivan7@hotmail.com
> > >> >wrote:
> > >> >
> > >> > > I think what is happening here is that we have different ideas
> about
> > >> the
> > >> > > architecture of the application.
> > >> > > Here's what I'm thinking:
> > >> > > *All processing of activity would happen BEFORE it's posted to
> > >> Streams*All
> > >> > > activity posted to Streams is in activitystrea.ms format
> > >> > > The Camel/EIP/OSGi way:
> > >>
RE: Using Camel for Routing
Posted by Danny Sullivan <ds...@hotmail.com>.
I think if we sacrifice scalability in the long run by switching from Camel to a web service then we should absolutely continue to use messaging and Camel. I'm confident, however, that applications exist that have a web service based model that service millions of requests and have no problem scaling without the help of Camel or any other messaging framework.
I'm going to keep the web-service branch separate for now, but I'd still like to keep the option on the table until we explore building scalable applications without the use of Camel. Though I can see the benefits of using Camel, I think the complexity it adds makes it worth taking a look at other options.
-Danny
> Date: Wed, 16 Oct 2013 16:26:43 -0400
> Subject: Re: Using Camel for Routing
> From: jletourneau80@gmail.com
> To: dev@streams.incubator.apache.org
>
> Yeah the more I think through this in terms of implications of a sole
> web app - I am skepticle that it scales to interesting use cases. I
> am not interested in streams to monitor "a few" facebook friends, I am
> interested in streams to monitor millions of events and activities
> generated from web site activity (clicks or tweets whatever) or
> enterprise assets across the globe.
>
> On Wed, Oct 16, 2013 at 4:15 PM, Chris Geer <ch...@cxtsoftware.com> wrote:
> > On Wed, Oct 16, 2013 at 11:58 AM, Danny Sullivan <ds...@hotmail.com>wrote:
> >
> >> I see the value in Streams as the ability of subscribers to get activity
> >> based on filters.
> >
> >
> > I agree
> >
> >
> >> To be clear, I do not want to eliminate processing, but do want to make
> >> the focus of processing tailored to the subscribers' filters. My more grand
> >> vision is that only the most relevant activity will be returned to the
> >> Subscriber and similar activity will be aggregated into single activities
> >> (this would be a little down the road). This processing would happen after
> >> the activity is stored in the DB.
> >>
> >
> > This brings me back to the concern about having it all within one web app.
> > If the webapp accepts inputs, aggregates/filters it and manages it's own
> > subscribers, how you do scale it, or provide redundancy? The beauty of
> > using a message based system (i.e. Camel...along with Storm possibly) is
> > that it allows you to partition the application however you want. So if you
> > want to have your ingest web service running on one machine and your
> > subscriber service running somewhere else and a third machine for your
> > really resource intensive aggregation component, you can. With the web app
> > you could do vertical scaling on one machine, or load balancing of all
> > components across multiple machines but it doesn't give you the save
> > flexibility.
> >
> >
> >> Do you imagine that Streams will come packaged with some processing
> >> components that will format other non-activitystrea.ms formatted json? I
> >> do think it'd be pretty cool if you could fire up Streams, input a couple
> >> Facebook Friends and people you follow on Twitter, and have all of that
> >> activity posted in one place. That might be better suited for a new
> >> separate module (something like streams-format?) if we decide on the
> >> web-application route.
> >>
> >
> > I think it has to come with something to entice more users. If everyone has
> > to build their own translation to even run "Hello World" it will be a
> > larger barrier to entry.
> >
> >
> >> -Danny
> >>
> >> > Date: Wed, 16 Oct 2013 11:22:51 -0700
> >> > Subject: Re: Using Camel for Routing
> >> > From: chris@cxtsoftware.com
> >> > To: dev@streams.incubator.apache.org
> >> >
> >> > On Wed, Oct 16, 2013 at 11:12 AM, Danny Sullivan <dsullivan7@hotmail.com
> >> >wrote:
> >> >
> >> > > I think what is happening here is that we have different ideas about
> >> the
> >> > > architecture of the application.
> >> > > Here's what I'm thinking:
> >> > > *All processing of activity would happen BEFORE it's posted to
> >> Streams*All
> >> > > activity posted to Streams is in activitystrea.ms format
> >> > > The Camel/EIP/OSGi way:
> >> > > *The user is able to post anything to Streams, doesn't matter what
> >> format
> >> > > it's in*The user can add processing components to Streams that format
> >> > > everything that is input to activitystrea.ms format, these can be
> >> easily
> >> > > plugged in, moved around, or taken out
> >> > > If we decide to go the second route, I think Camel would be the best
> >> way
> >> > > to go. However, I'd like to make the case for the first approach. The
> >> > > application can be quickly started and tested by a developer and user
> >> with
> >> > > the the first approach. If the user decides they want to experiment
> >> with
> >> > > twitter/facebook data (like Chris suggested) they can write a custom
> >> > > application (which can be a camel/eip application or even a python,
> >> ruby,
> >> > > etc application) that takes in the twitter/facebook data, formats it to
> >> > > activitystrea.ms format and then POSTs it to Streams. Also note: the
> >> > > webapp would NOT need to be restarted to add these. I don't think we'd
> >> lose
> >> > > a lot by going this route: in order to add the custom processing
> >> components
> >> > > in Camel, a Java component would still need to be written before it's
> >> > > plugged in to the camel.xml (and I think you would be tied to Java with
> >> > > this approach).
> >> > > Quite simply: We should keep formatting activity separate processing
> >> > > activity
> >> > > Perhaps I am missing some processing that could happen in between an
> >> > > activity's entrance to Streams and it's storage? If this is the case, I
> >> > > would make the argument that this processing should happen on an
> >> activities
> >> > > EXIT from Streams (there should be a way for a Subscriber to specify,
> >> in
> >> > > its filters or by some other means, the activity that it wants to see)
> >> > >
> >> >
> >> > Danny, what do you see as the scope for Streams? If you eliminate data
> >> > input in non-activitystrea.ms format and eliminate processing data what
> >> > does it do, just store each message (unmodified) and output it to
> >> > subscribers?
> >> >
> >> >
> >> > > -Danny
> >> > >
> >> > > > Date: Wed, 16 Oct 2013 10:11:23 -0700
> >> > > > Subject: Re: Using Camel for Routing
> >> > > > From: chris@cxtsoftware.com
> >> > > > To: dev@streams.incubator.apache.org
> >> > > >
> >> > > > Jason,
> >> > > >
> >> > > > I agree with you. I guess my vision of streams was a little grander
> >> in
> >> > > the
> >> > > > long run where you could configure the system to accept input from
> >> > > various
> >> > > > sources (facebook, twitter...) and process them in certain ways. So
> >> it
> >> > > > would have components that would know how to talk with Twitter and
> >> > > convert
> >> > > > tweets to activities then send it to the right process. So depending
> >> on
> >> > > how
> >> > > > you configure your system, it would create the proper routes from
> >> > > component
> >> > > > to component to allow processing dynamically which would require a
> >> > > > framework like Camel (don't think Storm helps us here since it's a
> >> > > compile
> >> > > > time configuration to my understanding). The nice thing about
> >> something
> >> > > > like camel is it allows you to reconfigure all the routes at runtime.
> >> > > This
> >> > > > is another reason I liked OSGI support as well as it allows
> >> deploying new
> >> > > > components without restarting the server (webapp). Of course to
> >> scale it
> >> > > > would require being able to deploy the configuration across multiple
> >> > > > machines which gets fun.
> >> > > >
> >> > > > Maybe my vision is too big for this project but it's where my head
> >> has
> >> > > been
> >> > > > at.
> >> > > >
> >> > > > Chris
> >> > > >
> >> > > >
> >> > > > On Wed, Oct 16, 2013 at 7:54 AM, Jason Letourneau
> >> > > > <jl...@gmail.com>wrote:
> >> > > >
> >> > > > > My only real complaint with this approach is that interjecting new
> >> > > > > components (processing components) becomes less plug and play
> >> without
> >> > > > > writing Java code in streams - with the Camel routing approach,
> >> > > > > streams Java is a black box and you just change the camel xml.
> >> > > > >
> >> > > > > The primary rationale in using Camel was gaining all of the
> >> > > > > architectural benefits one gains from enterprise integration
> >> patterns
> >> > > > > (truth: simplicity is not one and is a tradeoff for flexibility,
> >> but
> >> > > > > generally is considered a universal truth not one relegated to
> >> Camel
> >> > > > > and EIP).
> >> > > > >
> >> > > > > Jason
> >> > > > >
> >> > > > > On Fri, Oct 11, 2013 at 10:16 AM, Danny Sullivan <
> >> > > dsullivan7@hotmail.com>
> >> > > > > wrote:
> >> > > > > > Hey All,
> >> > > > > > I would like to merge the webservice branch with trunk. I think
> >> > > making
> >> > > > > streams web service based allows for a better focus on the goal of
> >> the
> >> > > > > application: process activitystrea.ms formatted activity json.
> >> While
> >> > > > > moving to a webservice model would lose us support for many of the
> >> use
> >> > > > > cases Chris mentioned (i.e twitter and facebook), I think the
> >> downsides
> >> > > > > would be mitigated by allowing and encouraging users to write
> >> custom
> >> > > > > appenders that take in twitter/facebook data, format it to
> >> > > > > activitystrea.ms standards, and POST to the Streams Server. These
> >> > > > > appenders could even be written in any language as long as the
> >> output
> >> > > is
> >> > > > > activitystrea.ms json. Compared to the trunk branch, I think the
> >> web
> >> > > > > service design employs the "black box" idiom in a much simpler
> >> > > fashion. I
> >> > > > > would be happy to hear contradicting thoughts and would encourage a
> >> > > > > discussion on this design switch.
> >> > > > > > Thanks!
> >> > > > > > Danny
> >> > > > > >> From: dsullivan7@hotmail.com
> >> > > > > >> To: dev@streams.incubator.apache.org
> >> > > > > >> Subject: RE: Using Camel for Routing
> >> > > > > >> Date: Fri, 4 Oct 2013 16:07:06 -0400
> >> > > > > >>
> >> > > > > >> I wrote up something quickly so I could demo the functionality
> >> as a
> >> > > web
> >> > > > > service as opposed to Camel. The new branch is available here:
> >> > > > > >>
> >> > >
> >> https://svn.apache.org/repos/asf/incubator/streams/branches/webservice/
> >> > > > > >> note that the new publishing and subscribing urls are:
> >> > > > > >>
> >> > > > >
> >> > >
> >> http://localhost:8080/streams-web/app/publisherRegisterhttp://localhost:8080/streams-web/app/subscriberRegister
> >> > > > > >> I used a Spring Web Service only because I was more familiar
> >> with
> >> > > it, I
> >> > > > > would like to research other alternatives (CXF) if it's clear that
> >> a
> >> > > web
> >> > > > > service is a realistic alternative. The main changes to the
> >> > > application are
> >> > > > > the web controller located here:
> >> > > > > >>
> >> > > > >
> >> > >
> >> https://svn.apache.org/repos/asf/incubator/streams/branches/webservice/streams-web/src/main/java/org/apache/streams/mvc/controller/StreamsWebController.java
> >> > > > > >> Which handles the input and output of the application, which
> >> were
> >> > > > > handled by the camel context before. The other main change is the
> >> use
> >> > > of
> >> > > > > the streams-components module located here:
> >> > > > > >>
> >> > > > >
> >> > >
> >> https://svn.apache.org/repos/asf/incubator/streams/branches/webservice/streams-components/
> >> > > > > >> This module contains service classes that return the correct
> >> output
> >> > > on
> >> > > > > each request sent to the controller. I reused the Publisher and
> >> > > Subscriber
> >> > > > > objects as well as the activityConsumerWarehouse and
> >> > > > > activitySubscriberWarehouse, I think they would be moved to the
> >> > > > > streams-components package if we want to go this route.
> >> > > > > >> Overall, I think the flow through the application is much more
> >> > > > > understandable. Previously, a developer would have to follow an
> >> > > activity
> >> > > > > through the camelContext.xml as well as through dynamically created
> >> > > routes
> >> > > > > verses the web service which provides clear entry and exit points
> >> to
> >> > > the
> >> > > > > application. I'd really like to hear feedback on this one, I really
> >> > > don't
> >> > > > > want to lose any efficiencies that we could be from
> >> > > > > messaging/osgi/camel/activemq.
> >> > > > > >> Thanks,
> >> > > > > >> Danny
> >> > > > > >>
> >> > > > > >> > From: dsullivan7@hotmail.com
> >> > > > > >> > To: dev@streams.incubator.apache.org
> >> > > > > >> > Subject: RE: Using Camel for Routing
> >> > > > > >> > Date: Thu, 3 Oct 2013 20:17:13 -0400
> >> > > > > >> >
> >> > > > > >> > Thanks for the update Jason. I'll keep the Spring MVC app in a
> >> > > > > separate branch (hopefully it won't take too long) and then
> >> perhaps we
> >> > > can
> >> > > > > reassess having both component and web container deployment? I
> >> think
> >> > > it's a
> >> > > > > good idea to keep both options in mind especially because this
> >> > > technology
> >> > > > > is relatively young.
> >> > > > > >> > Thanks!
> >> > > > > >> > Danny
> >> > > > > >> >
> >> > > > > >> > > Date: Thu, 3 Oct 2013 19:50:59 -0400
> >> > > > > >> > > Subject: Re: Using Camel for Routing
> >> > > > > >> > > From: jletourneau80@gmail.com
> >> > > > > >> > > To: dev@streams.incubator.apache.org
> >> > > > > >> > >
> >> > > > > >> > > Because of the osgi container support in addition to
> >> deployment
> >> > > as
> >> > > > > a web
> >> > > > > >> > > app - when last we visited this topic - I think the majority
> >> > > was in
> >> > > > > favor
> >> > > > > >> > > of keeping the component style deployment in addition to web
> >> > > > > container
> >> > > > > >> > > deployment
> >> > > > > >> > >
> >> > > > > >> > > On Thursday, October 3, 2013, Danny Sullivan wrote:
> >> > > > > >> > >
> >> > > > > >> > > > Hey all,
> >> > > > > >> > > > I've been considering taking a stab at moving streams from
> >> > > it's
> >> > > > > current
> >> > > > > >> > > > configuration with Camel to a Spring Web Service. Concerns
> >> > > over
> >> > > > > failover
> >> > > > > >> > > > and load balancing would be mitigated by the Cassandra DB
> >> > > > > (running on 9160)
> >> > > > > >> > > > and Storm (which depends on an instance of Zookeeper on
> >> 2181).
> >> > > > > Before I
> >> > > > > >> > > > start lancing any windmills I thought it'd be nice to get
> >> > > input
> >> > > > > on why the
> >> > > > > >> > > > design choice was made to base the application around
> >> Camel?
> >> > > > > >> > > > I think that tracking the flow of an activity through the
> >> > > > > application will
> >> > > > > >> > > > be much easier if we use Spring MVC. I'll set up a new
> >> branch
> >> > > > > using the MVC
> >> > > > > >> > > > architecture and I'll keep the community updated as I go
> >> > > along.
> >> > > > > >> > > > Danny
> >> > > > > >> >
> >> > > > > >>
> >> > > > > >
> >> > > > >
> >> > >
> >> > >
> >>
> >>
Re: Using Camel for Routing
Posted by Jason Letourneau <jl...@gmail.com>.
Yeah the more I think through this in terms of implications of a sole
web app - I am skepticle that it scales to interesting use cases. I
am not interested in streams to monitor "a few" facebook friends, I am
interested in streams to monitor millions of events and activities
generated from web site activity (clicks or tweets whatever) or
enterprise assets across the globe.
On Wed, Oct 16, 2013 at 4:15 PM, Chris Geer <ch...@cxtsoftware.com> wrote:
> On Wed, Oct 16, 2013 at 11:58 AM, Danny Sullivan <ds...@hotmail.com>wrote:
>
>> I see the value in Streams as the ability of subscribers to get activity
>> based on filters.
>
>
> I agree
>
>
>> To be clear, I do not want to eliminate processing, but do want to make
>> the focus of processing tailored to the subscribers' filters. My more grand
>> vision is that only the most relevant activity will be returned to the
>> Subscriber and similar activity will be aggregated into single activities
>> (this would be a little down the road). This processing would happen after
>> the activity is stored in the DB.
>>
>
> This brings me back to the concern about having it all within one web app.
> If the webapp accepts inputs, aggregates/filters it and manages it's own
> subscribers, how you do scale it, or provide redundancy? The beauty of
> using a message based system (i.e. Camel...along with Storm possibly) is
> that it allows you to partition the application however you want. So if you
> want to have your ingest web service running on one machine and your
> subscriber service running somewhere else and a third machine for your
> really resource intensive aggregation component, you can. With the web app
> you could do vertical scaling on one machine, or load balancing of all
> components across multiple machines but it doesn't give you the save
> flexibility.
>
>
>> Do you imagine that Streams will come packaged with some processing
>> components that will format other non-activitystrea.ms formatted json? I
>> do think it'd be pretty cool if you could fire up Streams, input a couple
>> Facebook Friends and people you follow on Twitter, and have all of that
>> activity posted in one place. That might be better suited for a new
>> separate module (something like streams-format?) if we decide on the
>> web-application route.
>>
>
> I think it has to come with something to entice more users. If everyone has
> to build their own translation to even run "Hello World" it will be a
> larger barrier to entry.
>
>
>> -Danny
>>
>> > Date: Wed, 16 Oct 2013 11:22:51 -0700
>> > Subject: Re: Using Camel for Routing
>> > From: chris@cxtsoftware.com
>> > To: dev@streams.incubator.apache.org
>> >
>> > On Wed, Oct 16, 2013 at 11:12 AM, Danny Sullivan <dsullivan7@hotmail.com
>> >wrote:
>> >
>> > > I think what is happening here is that we have different ideas about
>> the
>> > > architecture of the application.
>> > > Here's what I'm thinking:
>> > > *All processing of activity would happen BEFORE it's posted to
>> Streams*All
>> > > activity posted to Streams is in activitystrea.ms format
>> > > The Camel/EIP/OSGi way:
>> > > *The user is able to post anything to Streams, doesn't matter what
>> format
>> > > it's in*The user can add processing components to Streams that format
>> > > everything that is input to activitystrea.ms format, these can be
>> easily
>> > > plugged in, moved around, or taken out
>> > > If we decide to go the second route, I think Camel would be the best
>> way
>> > > to go. However, I'd like to make the case for the first approach. The
>> > > application can be quickly started and tested by a developer and user
>> with
>> > > the the first approach. If the user decides they want to experiment
>> with
>> > > twitter/facebook data (like Chris suggested) they can write a custom
>> > > application (which can be a camel/eip application or even a python,
>> ruby,
>> > > etc application) that takes in the twitter/facebook data, formats it to
>> > > activitystrea.ms format and then POSTs it to Streams. Also note: the
>> > > webapp would NOT need to be restarted to add these. I don't think we'd
>> lose
>> > > a lot by going this route: in order to add the custom processing
>> components
>> > > in Camel, a Java component would still need to be written before it's
>> > > plugged in to the camel.xml (and I think you would be tied to Java with
>> > > this approach).
>> > > Quite simply: We should keep formatting activity separate processing
>> > > activity
>> > > Perhaps I am missing some processing that could happen in between an
>> > > activity's entrance to Streams and it's storage? If this is the case, I
>> > > would make the argument that this processing should happen on an
>> activities
>> > > EXIT from Streams (there should be a way for a Subscriber to specify,
>> in
>> > > its filters or by some other means, the activity that it wants to see)
>> > >
>> >
>> > Danny, what do you see as the scope for Streams? If you eliminate data
>> > input in non-activitystrea.ms format and eliminate processing data what
>> > does it do, just store each message (unmodified) and output it to
>> > subscribers?
>> >
>> >
>> > > -Danny
>> > >
>> > > > Date: Wed, 16 Oct 2013 10:11:23 -0700
>> > > > Subject: Re: Using Camel for Routing
>> > > > From: chris@cxtsoftware.com
>> > > > To: dev@streams.incubator.apache.org
>> > > >
>> > > > Jason,
>> > > >
>> > > > I agree with you. I guess my vision of streams was a little grander
>> in
>> > > the
>> > > > long run where you could configure the system to accept input from
>> > > various
>> > > > sources (facebook, twitter...) and process them in certain ways. So
>> it
>> > > > would have components that would know how to talk with Twitter and
>> > > convert
>> > > > tweets to activities then send it to the right process. So depending
>> on
>> > > how
>> > > > you configure your system, it would create the proper routes from
>> > > component
>> > > > to component to allow processing dynamically which would require a
>> > > > framework like Camel (don't think Storm helps us here since it's a
>> > > compile
>> > > > time configuration to my understanding). The nice thing about
>> something
>> > > > like camel is it allows you to reconfigure all the routes at runtime.
>> > > This
>> > > > is another reason I liked OSGI support as well as it allows
>> deploying new
>> > > > components without restarting the server (webapp). Of course to
>> scale it
>> > > > would require being able to deploy the configuration across multiple
>> > > > machines which gets fun.
>> > > >
>> > > > Maybe my vision is too big for this project but it's where my head
>> has
>> > > been
>> > > > at.
>> > > >
>> > > > Chris
>> > > >
>> > > >
>> > > > On Wed, Oct 16, 2013 at 7:54 AM, Jason Letourneau
>> > > > <jl...@gmail.com>wrote:
>> > > >
>> > > > > My only real complaint with this approach is that interjecting new
>> > > > > components (processing components) becomes less plug and play
>> without
>> > > > > writing Java code in streams - with the Camel routing approach,
>> > > > > streams Java is a black box and you just change the camel xml.
>> > > > >
>> > > > > The primary rationale in using Camel was gaining all of the
>> > > > > architectural benefits one gains from enterprise integration
>> patterns
>> > > > > (truth: simplicity is not one and is a tradeoff for flexibility,
>> but
>> > > > > generally is considered a universal truth not one relegated to
>> Camel
>> > > > > and EIP).
>> > > > >
>> > > > > Jason
>> > > > >
>> > > > > On Fri, Oct 11, 2013 at 10:16 AM, Danny Sullivan <
>> > > dsullivan7@hotmail.com>
>> > > > > wrote:
>> > > > > > Hey All,
>> > > > > > I would like to merge the webservice branch with trunk. I think
>> > > making
>> > > > > streams web service based allows for a better focus on the goal of
>> the
>> > > > > application: process activitystrea.ms formatted activity json.
>> While
>> > > > > moving to a webservice model would lose us support for many of the
>> use
>> > > > > cases Chris mentioned (i.e twitter and facebook), I think the
>> downsides
>> > > > > would be mitigated by allowing and encouraging users to write
>> custom
>> > > > > appenders that take in twitter/facebook data, format it to
>> > > > > activitystrea.ms standards, and POST to the Streams Server. These
>> > > > > appenders could even be written in any language as long as the
>> output
>> > > is
>> > > > > activitystrea.ms json. Compared to the trunk branch, I think the
>> web
>> > > > > service design employs the "black box" idiom in a much simpler
>> > > fashion. I
>> > > > > would be happy to hear contradicting thoughts and would encourage a
>> > > > > discussion on this design switch.
>> > > > > > Thanks!
>> > > > > > Danny
>> > > > > >> From: dsullivan7@hotmail.com
>> > > > > >> To: dev@streams.incubator.apache.org
>> > > > > >> Subject: RE: Using Camel for Routing
>> > > > > >> Date: Fri, 4 Oct 2013 16:07:06 -0400
>> > > > > >>
>> > > > > >> I wrote up something quickly so I could demo the functionality
>> as a
>> > > web
>> > > > > service as opposed to Camel. The new branch is available here:
>> > > > > >>
>> > >
>> https://svn.apache.org/repos/asf/incubator/streams/branches/webservice/
>> > > > > >> note that the new publishing and subscribing urls are:
>> > > > > >>
>> > > > >
>> > >
>> http://localhost:8080/streams-web/app/publisherRegisterhttp://localhost:8080/streams-web/app/subscriberRegister
>> > > > > >> I used a Spring Web Service only because I was more familiar
>> with
>> > > it, I
>> > > > > would like to research other alternatives (CXF) if it's clear that
>> a
>> > > web
>> > > > > service is a realistic alternative. The main changes to the
>> > > application are
>> > > > > the web controller located here:
>> > > > > >>
>> > > > >
>> > >
>> https://svn.apache.org/repos/asf/incubator/streams/branches/webservice/streams-web/src/main/java/org/apache/streams/mvc/controller/StreamsWebController.java
>> > > > > >> Which handles the input and output of the application, which
>> were
>> > > > > handled by the camel context before. The other main change is the
>> use
>> > > of
>> > > > > the streams-components module located here:
>> > > > > >>
>> > > > >
>> > >
>> https://svn.apache.org/repos/asf/incubator/streams/branches/webservice/streams-components/
>> > > > > >> This module contains service classes that return the correct
>> output
>> > > on
>> > > > > each request sent to the controller. I reused the Publisher and
>> > > Subscriber
>> > > > > objects as well as the activityConsumerWarehouse and
>> > > > > activitySubscriberWarehouse, I think they would be moved to the
>> > > > > streams-components package if we want to go this route.
>> > > > > >> Overall, I think the flow through the application is much more
>> > > > > understandable. Previously, a developer would have to follow an
>> > > activity
>> > > > > through the camelContext.xml as well as through dynamically created
>> > > routes
>> > > > > verses the web service which provides clear entry and exit points
>> to
>> > > the
>> > > > > application. I'd really like to hear feedback on this one, I really
>> > > don't
>> > > > > want to lose any efficiencies that we could be from
>> > > > > messaging/osgi/camel/activemq.
>> > > > > >> Thanks,
>> > > > > >> Danny
>> > > > > >>
>> > > > > >> > From: dsullivan7@hotmail.com
>> > > > > >> > To: dev@streams.incubator.apache.org
>> > > > > >> > Subject: RE: Using Camel for Routing
>> > > > > >> > Date: Thu, 3 Oct 2013 20:17:13 -0400
>> > > > > >> >
>> > > > > >> > Thanks for the update Jason. I'll keep the Spring MVC app in a
>> > > > > separate branch (hopefully it won't take too long) and then
>> perhaps we
>> > > can
>> > > > > reassess having both component and web container deployment? I
>> think
>> > > it's a
>> > > > > good idea to keep both options in mind especially because this
>> > > technology
>> > > > > is relatively young.
>> > > > > >> > Thanks!
>> > > > > >> > Danny
>> > > > > >> >
>> > > > > >> > > Date: Thu, 3 Oct 2013 19:50:59 -0400
>> > > > > >> > > Subject: Re: Using Camel for Routing
>> > > > > >> > > From: jletourneau80@gmail.com
>> > > > > >> > > To: dev@streams.incubator.apache.org
>> > > > > >> > >
>> > > > > >> > > Because of the osgi container support in addition to
>> deployment
>> > > as
>> > > > > a web
>> > > > > >> > > app - when last we visited this topic - I think the majority
>> > > was in
>> > > > > favor
>> > > > > >> > > of keeping the component style deployment in addition to web
>> > > > > container
>> > > > > >> > > deployment
>> > > > > >> > >
>> > > > > >> > > On Thursday, October 3, 2013, Danny Sullivan wrote:
>> > > > > >> > >
>> > > > > >> > > > Hey all,
>> > > > > >> > > > I've been considering taking a stab at moving streams from
>> > > it's
>> > > > > current
>> > > > > >> > > > configuration with Camel to a Spring Web Service. Concerns
>> > > over
>> > > > > failover
>> > > > > >> > > > and load balancing would be mitigated by the Cassandra DB
>> > > > > (running on 9160)
>> > > > > >> > > > and Storm (which depends on an instance of Zookeeper on
>> 2181).
>> > > > > Before I
>> > > > > >> > > > start lancing any windmills I thought it'd be nice to get
>> > > input
>> > > > > on why the
>> > > > > >> > > > design choice was made to base the application around
>> Camel?
>> > > > > >> > > > I think that tracking the flow of an activity through the
>> > > > > application will
>> > > > > >> > > > be much easier if we use Spring MVC. I'll set up a new
>> branch
>> > > > > using the MVC
>> > > > > >> > > > architecture and I'll keep the community updated as I go
>> > > along.
>> > > > > >> > > > Danny
>> > > > > >> >
>> > > > > >>
>> > > > > >
>> > > > >
>> > >
>> > >
>>
>>
Re: Using Camel for Routing
Posted by Chris Geer <ch...@cxtsoftware.com>.
On Wed, Oct 16, 2013 at 11:58 AM, Danny Sullivan <ds...@hotmail.com>wrote:
> I see the value in Streams as the ability of subscribers to get activity
> based on filters.
I agree
> To be clear, I do not want to eliminate processing, but do want to make
> the focus of processing tailored to the subscribers' filters. My more grand
> vision is that only the most relevant activity will be returned to the
> Subscriber and similar activity will be aggregated into single activities
> (this would be a little down the road). This processing would happen after
> the activity is stored in the DB.
>
This brings me back to the concern about having it all within one web app.
If the webapp accepts inputs, aggregates/filters it and manages it's own
subscribers, how you do scale it, or provide redundancy? The beauty of
using a message based system (i.e. Camel...along with Storm possibly) is
that it allows you to partition the application however you want. So if you
want to have your ingest web service running on one machine and your
subscriber service running somewhere else and a third machine for your
really resource intensive aggregation component, you can. With the web app
you could do vertical scaling on one machine, or load balancing of all
components across multiple machines but it doesn't give you the save
flexibility.
> Do you imagine that Streams will come packaged with some processing
> components that will format other non-activitystrea.ms formatted json? I
> do think it'd be pretty cool if you could fire up Streams, input a couple
> Facebook Friends and people you follow on Twitter, and have all of that
> activity posted in one place. That might be better suited for a new
> separate module (something like streams-format?) if we decide on the
> web-application route.
>
I think it has to come with something to entice more users. If everyone has
to build their own translation to even run "Hello World" it will be a
larger barrier to entry.
> -Danny
>
> > Date: Wed, 16 Oct 2013 11:22:51 -0700
> > Subject: Re: Using Camel for Routing
> > From: chris@cxtsoftware.com
> > To: dev@streams.incubator.apache.org
> >
> > On Wed, Oct 16, 2013 at 11:12 AM, Danny Sullivan <dsullivan7@hotmail.com
> >wrote:
> >
> > > I think what is happening here is that we have different ideas about
> the
> > > architecture of the application.
> > > Here's what I'm thinking:
> > > *All processing of activity would happen BEFORE it's posted to
> Streams*All
> > > activity posted to Streams is in activitystrea.ms format
> > > The Camel/EIP/OSGi way:
> > > *The user is able to post anything to Streams, doesn't matter what
> format
> > > it's in*The user can add processing components to Streams that format
> > > everything that is input to activitystrea.ms format, these can be
> easily
> > > plugged in, moved around, or taken out
> > > If we decide to go the second route, I think Camel would be the best
> way
> > > to go. However, I'd like to make the case for the first approach. The
> > > application can be quickly started and tested by a developer and user
> with
> > > the the first approach. If the user decides they want to experiment
> with
> > > twitter/facebook data (like Chris suggested) they can write a custom
> > > application (which can be a camel/eip application or even a python,
> ruby,
> > > etc application) that takes in the twitter/facebook data, formats it to
> > > activitystrea.ms format and then POSTs it to Streams. Also note: the
> > > webapp would NOT need to be restarted to add these. I don't think we'd
> lose
> > > a lot by going this route: in order to add the custom processing
> components
> > > in Camel, a Java component would still need to be written before it's
> > > plugged in to the camel.xml (and I think you would be tied to Java with
> > > this approach).
> > > Quite simply: We should keep formatting activity separate processing
> > > activity
> > > Perhaps I am missing some processing that could happen in between an
> > > activity's entrance to Streams and it's storage? If this is the case, I
> > > would make the argument that this processing should happen on an
> activities
> > > EXIT from Streams (there should be a way for a Subscriber to specify,
> in
> > > its filters or by some other means, the activity that it wants to see)
> > >
> >
> > Danny, what do you see as the scope for Streams? If you eliminate data
> > input in non-activitystrea.ms format and eliminate processing data what
> > does it do, just store each message (unmodified) and output it to
> > subscribers?
> >
> >
> > > -Danny
> > >
> > > > Date: Wed, 16 Oct 2013 10:11:23 -0700
> > > > Subject: Re: Using Camel for Routing
> > > > From: chris@cxtsoftware.com
> > > > To: dev@streams.incubator.apache.org
> > > >
> > > > Jason,
> > > >
> > > > I agree with you. I guess my vision of streams was a little grander
> in
> > > the
> > > > long run where you could configure the system to accept input from
> > > various
> > > > sources (facebook, twitter...) and process them in certain ways. So
> it
> > > > would have components that would know how to talk with Twitter and
> > > convert
> > > > tweets to activities then send it to the right process. So depending
> on
> > > how
> > > > you configure your system, it would create the proper routes from
> > > component
> > > > to component to allow processing dynamically which would require a
> > > > framework like Camel (don't think Storm helps us here since it's a
> > > compile
> > > > time configuration to my understanding). The nice thing about
> something
> > > > like camel is it allows you to reconfigure all the routes at runtime.
> > > This
> > > > is another reason I liked OSGI support as well as it allows
> deploying new
> > > > components without restarting the server (webapp). Of course to
> scale it
> > > > would require being able to deploy the configuration across multiple
> > > > machines which gets fun.
> > > >
> > > > Maybe my vision is too big for this project but it's where my head
> has
> > > been
> > > > at.
> > > >
> > > > Chris
> > > >
> > > >
> > > > On Wed, Oct 16, 2013 at 7:54 AM, Jason Letourneau
> > > > <jl...@gmail.com>wrote:
> > > >
> > > > > My only real complaint with this approach is that interjecting new
> > > > > components (processing components) becomes less plug and play
> without
> > > > > writing Java code in streams - with the Camel routing approach,
> > > > > streams Java is a black box and you just change the camel xml.
> > > > >
> > > > > The primary rationale in using Camel was gaining all of the
> > > > > architectural benefits one gains from enterprise integration
> patterns
> > > > > (truth: simplicity is not one and is a tradeoff for flexibility,
> but
> > > > > generally is considered a universal truth not one relegated to
> Camel
> > > > > and EIP).
> > > > >
> > > > > Jason
> > > > >
> > > > > On Fri, Oct 11, 2013 at 10:16 AM, Danny Sullivan <
> > > dsullivan7@hotmail.com>
> > > > > wrote:
> > > > > > Hey All,
> > > > > > I would like to merge the webservice branch with trunk. I think
> > > making
> > > > > streams web service based allows for a better focus on the goal of
> the
> > > > > application: process activitystrea.ms formatted activity json.
> While
> > > > > moving to a webservice model would lose us support for many of the
> use
> > > > > cases Chris mentioned (i.e twitter and facebook), I think the
> downsides
> > > > > would be mitigated by allowing and encouraging users to write
> custom
> > > > > appenders that take in twitter/facebook data, format it to
> > > > > activitystrea.ms standards, and POST to the Streams Server. These
> > > > > appenders could even be written in any language as long as the
> output
> > > is
> > > > > activitystrea.ms json. Compared to the trunk branch, I think the
> web
> > > > > service design employs the "black box" idiom in a much simpler
> > > fashion. I
> > > > > would be happy to hear contradicting thoughts and would encourage a
> > > > > discussion on this design switch.
> > > > > > Thanks!
> > > > > > Danny
> > > > > >> From: dsullivan7@hotmail.com
> > > > > >> To: dev@streams.incubator.apache.org
> > > > > >> Subject: RE: Using Camel for Routing
> > > > > >> Date: Fri, 4 Oct 2013 16:07:06 -0400
> > > > > >>
> > > > > >> I wrote up something quickly so I could demo the functionality
> as a
> > > web
> > > > > service as opposed to Camel. The new branch is available here:
> > > > > >>
> > >
> https://svn.apache.org/repos/asf/incubator/streams/branches/webservice/
> > > > > >> note that the new publishing and subscribing urls are:
> > > > > >>
> > > > >
> > >
> http://localhost:8080/streams-web/app/publisherRegisterhttp://localhost:8080/streams-web/app/subscriberRegister
> > > > > >> I used a Spring Web Service only because I was more familiar
> with
> > > it, I
> > > > > would like to research other alternatives (CXF) if it's clear that
> a
> > > web
> > > > > service is a realistic alternative. The main changes to the
> > > application are
> > > > > the web controller located here:
> > > > > >>
> > > > >
> > >
> https://svn.apache.org/repos/asf/incubator/streams/branches/webservice/streams-web/src/main/java/org/apache/streams/mvc/controller/StreamsWebController.java
> > > > > >> Which handles the input and output of the application, which
> were
> > > > > handled by the camel context before. The other main change is the
> use
> > > of
> > > > > the streams-components module located here:
> > > > > >>
> > > > >
> > >
> https://svn.apache.org/repos/asf/incubator/streams/branches/webservice/streams-components/
> > > > > >> This module contains service classes that return the correct
> output
> > > on
> > > > > each request sent to the controller. I reused the Publisher and
> > > Subscriber
> > > > > objects as well as the activityConsumerWarehouse and
> > > > > activitySubscriberWarehouse, I think they would be moved to the
> > > > > streams-components package if we want to go this route.
> > > > > >> Overall, I think the flow through the application is much more
> > > > > understandable. Previously, a developer would have to follow an
> > > activity
> > > > > through the camelContext.xml as well as through dynamically created
> > > routes
> > > > > verses the web service which provides clear entry and exit points
> to
> > > the
> > > > > application. I'd really like to hear feedback on this one, I really
> > > don't
> > > > > want to lose any efficiencies that we could be from
> > > > > messaging/osgi/camel/activemq.
> > > > > >> Thanks,
> > > > > >> Danny
> > > > > >>
> > > > > >> > From: dsullivan7@hotmail.com
> > > > > >> > To: dev@streams.incubator.apache.org
> > > > > >> > Subject: RE: Using Camel for Routing
> > > > > >> > Date: Thu, 3 Oct 2013 20:17:13 -0400
> > > > > >> >
> > > > > >> > Thanks for the update Jason. I'll keep the Spring MVC app in a
> > > > > separate branch (hopefully it won't take too long) and then
> perhaps we
> > > can
> > > > > reassess having both component and web container deployment? I
> think
> > > it's a
> > > > > good idea to keep both options in mind especially because this
> > > technology
> > > > > is relatively young.
> > > > > >> > Thanks!
> > > > > >> > Danny
> > > > > >> >
> > > > > >> > > Date: Thu, 3 Oct 2013 19:50:59 -0400
> > > > > >> > > Subject: Re: Using Camel for Routing
> > > > > >> > > From: jletourneau80@gmail.com
> > > > > >> > > To: dev@streams.incubator.apache.org
> > > > > >> > >
> > > > > >> > > Because of the osgi container support in addition to
> deployment
> > > as
> > > > > a web
> > > > > >> > > app - when last we visited this topic - I think the majority
> > > was in
> > > > > favor
> > > > > >> > > of keeping the component style deployment in addition to web
> > > > > container
> > > > > >> > > deployment
> > > > > >> > >
> > > > > >> > > On Thursday, October 3, 2013, Danny Sullivan wrote:
> > > > > >> > >
> > > > > >> > > > Hey all,
> > > > > >> > > > I've been considering taking a stab at moving streams from
> > > it's
> > > > > current
> > > > > >> > > > configuration with Camel to a Spring Web Service. Concerns
> > > over
> > > > > failover
> > > > > >> > > > and load balancing would be mitigated by the Cassandra DB
> > > > > (running on 9160)
> > > > > >> > > > and Storm (which depends on an instance of Zookeeper on
> 2181).
> > > > > Before I
> > > > > >> > > > start lancing any windmills I thought it'd be nice to get
> > > input
> > > > > on why the
> > > > > >> > > > design choice was made to base the application around
> Camel?
> > > > > >> > > > I think that tracking the flow of an activity through the
> > > > > application will
> > > > > >> > > > be much easier if we use Spring MVC. I'll set up a new
> branch
> > > > > using the MVC
> > > > > >> > > > architecture and I'll keep the community updated as I go
> > > along.
> > > > > >> > > > Danny
> > > > > >> >
> > > > > >>
> > > > > >
> > > > >
> > >
> > >
>
>
RE: Using Camel for Routing
Posted by Danny Sullivan <ds...@hotmail.com>.
I see the value in Streams as the ability of subscribers to get activity based on filters. To be clear, I do not want to eliminate processing, but do want to make the focus of processing tailored to the subscribers' filters. My more grand vision is that only the most relevant activity will be returned to the Subscriber and similar activity will be aggregated into single activities (this would be a little down the road). This processing would happen after the activity is stored in the DB.
Do you imagine that Streams will come packaged with some processing components that will format other non-activitystrea.ms formatted json? I do think it'd be pretty cool if you could fire up Streams, input a couple Facebook Friends and people you follow on Twitter, and have all of that activity posted in one place. That might be better suited for a new separate module (something like streams-format?) if we decide on the web-application route.
-Danny
> Date: Wed, 16 Oct 2013 11:22:51 -0700
> Subject: Re: Using Camel for Routing
> From: chris@cxtsoftware.com
> To: dev@streams.incubator.apache.org
>
> On Wed, Oct 16, 2013 at 11:12 AM, Danny Sullivan <ds...@hotmail.com>wrote:
>
> > I think what is happening here is that we have different ideas about the
> > architecture of the application.
> > Here's what I'm thinking:
> > *All processing of activity would happen BEFORE it's posted to Streams*All
> > activity posted to Streams is in activitystrea.ms format
> > The Camel/EIP/OSGi way:
> > *The user is able to post anything to Streams, doesn't matter what format
> > it's in*The user can add processing components to Streams that format
> > everything that is input to activitystrea.ms format, these can be easily
> > plugged in, moved around, or taken out
> > If we decide to go the second route, I think Camel would be the best way
> > to go. However, I'd like to make the case for the first approach. The
> > application can be quickly started and tested by a developer and user with
> > the the first approach. If the user decides they want to experiment with
> > twitter/facebook data (like Chris suggested) they can write a custom
> > application (which can be a camel/eip application or even a python, ruby,
> > etc application) that takes in the twitter/facebook data, formats it to
> > activitystrea.ms format and then POSTs it to Streams. Also note: the
> > webapp would NOT need to be restarted to add these. I don't think we'd lose
> > a lot by going this route: in order to add the custom processing components
> > in Camel, a Java component would still need to be written before it's
> > plugged in to the camel.xml (and I think you would be tied to Java with
> > this approach).
> > Quite simply: We should keep formatting activity separate processing
> > activity
> > Perhaps I am missing some processing that could happen in between an
> > activity's entrance to Streams and it's storage? If this is the case, I
> > would make the argument that this processing should happen on an activities
> > EXIT from Streams (there should be a way for a Subscriber to specify, in
> > its filters or by some other means, the activity that it wants to see)
> >
>
> Danny, what do you see as the scope for Streams? If you eliminate data
> input in non-activitystrea.ms format and eliminate processing data what
> does it do, just store each message (unmodified) and output it to
> subscribers?
>
>
> > -Danny
> >
> > > Date: Wed, 16 Oct 2013 10:11:23 -0700
> > > Subject: Re: Using Camel for Routing
> > > From: chris@cxtsoftware.com
> > > To: dev@streams.incubator.apache.org
> > >
> > > Jason,
> > >
> > > I agree with you. I guess my vision of streams was a little grander in
> > the
> > > long run where you could configure the system to accept input from
> > various
> > > sources (facebook, twitter...) and process them in certain ways. So it
> > > would have components that would know how to talk with Twitter and
> > convert
> > > tweets to activities then send it to the right process. So depending on
> > how
> > > you configure your system, it would create the proper routes from
> > component
> > > to component to allow processing dynamically which would require a
> > > framework like Camel (don't think Storm helps us here since it's a
> > compile
> > > time configuration to my understanding). The nice thing about something
> > > like camel is it allows you to reconfigure all the routes at runtime.
> > This
> > > is another reason I liked OSGI support as well as it allows deploying new
> > > components without restarting the server (webapp). Of course to scale it
> > > would require being able to deploy the configuration across multiple
> > > machines which gets fun.
> > >
> > > Maybe my vision is too big for this project but it's where my head has
> > been
> > > at.
> > >
> > > Chris
> > >
> > >
> > > On Wed, Oct 16, 2013 at 7:54 AM, Jason Letourneau
> > > <jl...@gmail.com>wrote:
> > >
> > > > My only real complaint with this approach is that interjecting new
> > > > components (processing components) becomes less plug and play without
> > > > writing Java code in streams - with the Camel routing approach,
> > > > streams Java is a black box and you just change the camel xml.
> > > >
> > > > The primary rationale in using Camel was gaining all of the
> > > > architectural benefits one gains from enterprise integration patterns
> > > > (truth: simplicity is not one and is a tradeoff for flexibility, but
> > > > generally is considered a universal truth not one relegated to Camel
> > > > and EIP).
> > > >
> > > > Jason
> > > >
> > > > On Fri, Oct 11, 2013 at 10:16 AM, Danny Sullivan <
> > dsullivan7@hotmail.com>
> > > > wrote:
> > > > > Hey All,
> > > > > I would like to merge the webservice branch with trunk. I think
> > making
> > > > streams web service based allows for a better focus on the goal of the
> > > > application: process activitystrea.ms formatted activity json. While
> > > > moving to a webservice model would lose us support for many of the use
> > > > cases Chris mentioned (i.e twitter and facebook), I think the downsides
> > > > would be mitigated by allowing and encouraging users to write custom
> > > > appenders that take in twitter/facebook data, format it to
> > > > activitystrea.ms standards, and POST to the Streams Server. These
> > > > appenders could even be written in any language as long as the output
> > is
> > > > activitystrea.ms json. Compared to the trunk branch, I think the web
> > > > service design employs the "black box" idiom in a much simpler
> > fashion. I
> > > > would be happy to hear contradicting thoughts and would encourage a
> > > > discussion on this design switch.
> > > > > Thanks!
> > > > > Danny
> > > > >> From: dsullivan7@hotmail.com
> > > > >> To: dev@streams.incubator.apache.org
> > > > >> Subject: RE: Using Camel for Routing
> > > > >> Date: Fri, 4 Oct 2013 16:07:06 -0400
> > > > >>
> > > > >> I wrote up something quickly so I could demo the functionality as a
> > web
> > > > service as opposed to Camel. The new branch is available here:
> > > > >>
> > https://svn.apache.org/repos/asf/incubator/streams/branches/webservice/
> > > > >> note that the new publishing and subscribing urls are:
> > > > >>
> > > >
> > http://localhost:8080/streams-web/app/publisherRegisterhttp://localhost:8080/streams-web/app/subscriberRegister
> > > > >> I used a Spring Web Service only because I was more familiar with
> > it, I
> > > > would like to research other alternatives (CXF) if it's clear that a
> > web
> > > > service is a realistic alternative. The main changes to the
> > application are
> > > > the web controller located here:
> > > > >>
> > > >
> > https://svn.apache.org/repos/asf/incubator/streams/branches/webservice/streams-web/src/main/java/org/apache/streams/mvc/controller/StreamsWebController.java
> > > > >> Which handles the input and output of the application, which were
> > > > handled by the camel context before. The other main change is the use
> > of
> > > > the streams-components module located here:
> > > > >>
> > > >
> > https://svn.apache.org/repos/asf/incubator/streams/branches/webservice/streams-components/
> > > > >> This module contains service classes that return the correct output
> > on
> > > > each request sent to the controller. I reused the Publisher and
> > Subscriber
> > > > objects as well as the activityConsumerWarehouse and
> > > > activitySubscriberWarehouse, I think they would be moved to the
> > > > streams-components package if we want to go this route.
> > > > >> Overall, I think the flow through the application is much more
> > > > understandable. Previously, a developer would have to follow an
> > activity
> > > > through the camelContext.xml as well as through dynamically created
> > routes
> > > > verses the web service which provides clear entry and exit points to
> > the
> > > > application. I'd really like to hear feedback on this one, I really
> > don't
> > > > want to lose any efficiencies that we could be from
> > > > messaging/osgi/camel/activemq.
> > > > >> Thanks,
> > > > >> Danny
> > > > >>
> > > > >> > From: dsullivan7@hotmail.com
> > > > >> > To: dev@streams.incubator.apache.org
> > > > >> > Subject: RE: Using Camel for Routing
> > > > >> > Date: Thu, 3 Oct 2013 20:17:13 -0400
> > > > >> >
> > > > >> > Thanks for the update Jason. I'll keep the Spring MVC app in a
> > > > separate branch (hopefully it won't take too long) and then perhaps we
> > can
> > > > reassess having both component and web container deployment? I think
> > it's a
> > > > good idea to keep both options in mind especially because this
> > technology
> > > > is relatively young.
> > > > >> > Thanks!
> > > > >> > Danny
> > > > >> >
> > > > >> > > Date: Thu, 3 Oct 2013 19:50:59 -0400
> > > > >> > > Subject: Re: Using Camel for Routing
> > > > >> > > From: jletourneau80@gmail.com
> > > > >> > > To: dev@streams.incubator.apache.org
> > > > >> > >
> > > > >> > > Because of the osgi container support in addition to deployment
> > as
> > > > a web
> > > > >> > > app - when last we visited this topic - I think the majority
> > was in
> > > > favor
> > > > >> > > of keeping the component style deployment in addition to web
> > > > container
> > > > >> > > deployment
> > > > >> > >
> > > > >> > > On Thursday, October 3, 2013, Danny Sullivan wrote:
> > > > >> > >
> > > > >> > > > Hey all,
> > > > >> > > > I've been considering taking a stab at moving streams from
> > it's
> > > > current
> > > > >> > > > configuration with Camel to a Spring Web Service. Concerns
> > over
> > > > failover
> > > > >> > > > and load balancing would be mitigated by the Cassandra DB
> > > > (running on 9160)
> > > > >> > > > and Storm (which depends on an instance of Zookeeper on 2181).
> > > > Before I
> > > > >> > > > start lancing any windmills I thought it'd be nice to get
> > input
> > > > on why the
> > > > >> > > > design choice was made to base the application around Camel?
> > > > >> > > > I think that tracking the flow of an activity through the
> > > > application will
> > > > >> > > > be much easier if we use Spring MVC. I'll set up a new branch
> > > > using the MVC
> > > > >> > > > architecture and I'll keep the community updated as I go
> > along.
> > > > >> > > > Danny
> > > > >> >
> > > > >>
> > > > >
> > > >
> >
> >
Re: Using Camel for Routing
Posted by Chris Geer <ch...@cxtsoftware.com>.
On Wed, Oct 16, 2013 at 11:12 AM, Danny Sullivan <ds...@hotmail.com>wrote:
> I think what is happening here is that we have different ideas about the
> architecture of the application.
> Here's what I'm thinking:
> *All processing of activity would happen BEFORE it's posted to Streams*All
> activity posted to Streams is in activitystrea.ms format
> The Camel/EIP/OSGi way:
> *The user is able to post anything to Streams, doesn't matter what format
> it's in*The user can add processing components to Streams that format
> everything that is input to activitystrea.ms format, these can be easily
> plugged in, moved around, or taken out
> If we decide to go the second route, I think Camel would be the best way
> to go. However, I'd like to make the case for the first approach. The
> application can be quickly started and tested by a developer and user with
> the the first approach. If the user decides they want to experiment with
> twitter/facebook data (like Chris suggested) they can write a custom
> application (which can be a camel/eip application or even a python, ruby,
> etc application) that takes in the twitter/facebook data, formats it to
> activitystrea.ms format and then POSTs it to Streams. Also note: the
> webapp would NOT need to be restarted to add these. I don't think we'd lose
> a lot by going this route: in order to add the custom processing components
> in Camel, a Java component would still need to be written before it's
> plugged in to the camel.xml (and I think you would be tied to Java with
> this approach).
> Quite simply: We should keep formatting activity separate processing
> activity
> Perhaps I am missing some processing that could happen in between an
> activity's entrance to Streams and it's storage? If this is the case, I
> would make the argument that this processing should happen on an activities
> EXIT from Streams (there should be a way for a Subscriber to specify, in
> its filters or by some other means, the activity that it wants to see)
>
Danny, what do you see as the scope for Streams? If you eliminate data
input in non-activitystrea.ms format and eliminate processing data what
does it do, just store each message (unmodified) and output it to
subscribers?
> -Danny
>
> > Date: Wed, 16 Oct 2013 10:11:23 -0700
> > Subject: Re: Using Camel for Routing
> > From: chris@cxtsoftware.com
> > To: dev@streams.incubator.apache.org
> >
> > Jason,
> >
> > I agree with you. I guess my vision of streams was a little grander in
> the
> > long run where you could configure the system to accept input from
> various
> > sources (facebook, twitter...) and process them in certain ways. So it
> > would have components that would know how to talk with Twitter and
> convert
> > tweets to activities then send it to the right process. So depending on
> how
> > you configure your system, it would create the proper routes from
> component
> > to component to allow processing dynamically which would require a
> > framework like Camel (don't think Storm helps us here since it's a
> compile
> > time configuration to my understanding). The nice thing about something
> > like camel is it allows you to reconfigure all the routes at runtime.
> This
> > is another reason I liked OSGI support as well as it allows deploying new
> > components without restarting the server (webapp). Of course to scale it
> > would require being able to deploy the configuration across multiple
> > machines which gets fun.
> >
> > Maybe my vision is too big for this project but it's where my head has
> been
> > at.
> >
> > Chris
> >
> >
> > On Wed, Oct 16, 2013 at 7:54 AM, Jason Letourneau
> > <jl...@gmail.com>wrote:
> >
> > > My only real complaint with this approach is that interjecting new
> > > components (processing components) becomes less plug and play without
> > > writing Java code in streams - with the Camel routing approach,
> > > streams Java is a black box and you just change the camel xml.
> > >
> > > The primary rationale in using Camel was gaining all of the
> > > architectural benefits one gains from enterprise integration patterns
> > > (truth: simplicity is not one and is a tradeoff for flexibility, but
> > > generally is considered a universal truth not one relegated to Camel
> > > and EIP).
> > >
> > > Jason
> > >
> > > On Fri, Oct 11, 2013 at 10:16 AM, Danny Sullivan <
> dsullivan7@hotmail.com>
> > > wrote:
> > > > Hey All,
> > > > I would like to merge the webservice branch with trunk. I think
> making
> > > streams web service based allows for a better focus on the goal of the
> > > application: process activitystrea.ms formatted activity json. While
> > > moving to a webservice model would lose us support for many of the use
> > > cases Chris mentioned (i.e twitter and facebook), I think the downsides
> > > would be mitigated by allowing and encouraging users to write custom
> > > appenders that take in twitter/facebook data, format it to
> > > activitystrea.ms standards, and POST to the Streams Server. These
> > > appenders could even be written in any language as long as the output
> is
> > > activitystrea.ms json. Compared to the trunk branch, I think the web
> > > service design employs the "black box" idiom in a much simpler
> fashion. I
> > > would be happy to hear contradicting thoughts and would encourage a
> > > discussion on this design switch.
> > > > Thanks!
> > > > Danny
> > > >> From: dsullivan7@hotmail.com
> > > >> To: dev@streams.incubator.apache.org
> > > >> Subject: RE: Using Camel for Routing
> > > >> Date: Fri, 4 Oct 2013 16:07:06 -0400
> > > >>
> > > >> I wrote up something quickly so I could demo the functionality as a
> web
> > > service as opposed to Camel. The new branch is available here:
> > > >>
> https://svn.apache.org/repos/asf/incubator/streams/branches/webservice/
> > > >> note that the new publishing and subscribing urls are:
> > > >>
> > >
> http://localhost:8080/streams-web/app/publisherRegisterhttp://localhost:8080/streams-web/app/subscriberRegister
> > > >> I used a Spring Web Service only because I was more familiar with
> it, I
> > > would like to research other alternatives (CXF) if it's clear that a
> web
> > > service is a realistic alternative. The main changes to the
> application are
> > > the web controller located here:
> > > >>
> > >
> https://svn.apache.org/repos/asf/incubator/streams/branches/webservice/streams-web/src/main/java/org/apache/streams/mvc/controller/StreamsWebController.java
> > > >> Which handles the input and output of the application, which were
> > > handled by the camel context before. The other main change is the use
> of
> > > the streams-components module located here:
> > > >>
> > >
> https://svn.apache.org/repos/asf/incubator/streams/branches/webservice/streams-components/
> > > >> This module contains service classes that return the correct output
> on
> > > each request sent to the controller. I reused the Publisher and
> Subscriber
> > > objects as well as the activityConsumerWarehouse and
> > > activitySubscriberWarehouse, I think they would be moved to the
> > > streams-components package if we want to go this route.
> > > >> Overall, I think the flow through the application is much more
> > > understandable. Previously, a developer would have to follow an
> activity
> > > through the camelContext.xml as well as through dynamically created
> routes
> > > verses the web service which provides clear entry and exit points to
> the
> > > application. I'd really like to hear feedback on this one, I really
> don't
> > > want to lose any efficiencies that we could be from
> > > messaging/osgi/camel/activemq.
> > > >> Thanks,
> > > >> Danny
> > > >>
> > > >> > From: dsullivan7@hotmail.com
> > > >> > To: dev@streams.incubator.apache.org
> > > >> > Subject: RE: Using Camel for Routing
> > > >> > Date: Thu, 3 Oct 2013 20:17:13 -0400
> > > >> >
> > > >> > Thanks for the update Jason. I'll keep the Spring MVC app in a
> > > separate branch (hopefully it won't take too long) and then perhaps we
> can
> > > reassess having both component and web container deployment? I think
> it's a
> > > good idea to keep both options in mind especially because this
> technology
> > > is relatively young.
> > > >> > Thanks!
> > > >> > Danny
> > > >> >
> > > >> > > Date: Thu, 3 Oct 2013 19:50:59 -0400
> > > >> > > Subject: Re: Using Camel for Routing
> > > >> > > From: jletourneau80@gmail.com
> > > >> > > To: dev@streams.incubator.apache.org
> > > >> > >
> > > >> > > Because of the osgi container support in addition to deployment
> as
> > > a web
> > > >> > > app - when last we visited this topic - I think the majority
> was in
> > > favor
> > > >> > > of keeping the component style deployment in addition to web
> > > container
> > > >> > > deployment
> > > >> > >
> > > >> > > On Thursday, October 3, 2013, Danny Sullivan wrote:
> > > >> > >
> > > >> > > > Hey all,
> > > >> > > > I've been considering taking a stab at moving streams from
> it's
> > > current
> > > >> > > > configuration with Camel to a Spring Web Service. Concerns
> over
> > > failover
> > > >> > > > and load balancing would be mitigated by the Cassandra DB
> > > (running on 9160)
> > > >> > > > and Storm (which depends on an instance of Zookeeper on 2181).
> > > Before I
> > > >> > > > start lancing any windmills I thought it'd be nice to get
> input
> > > on why the
> > > >> > > > design choice was made to base the application around Camel?
> > > >> > > > I think that tracking the flow of an activity through the
> > > application will
> > > >> > > > be much easier if we use Spring MVC. I'll set up a new branch
> > > using the MVC
> > > >> > > > architecture and I'll keep the community updated as I go
> along.
> > > >> > > > Danny
> > > >> >
> > > >>
> > > >
> > >
>
>
RE: Using Camel for Routing
Posted by Danny Sullivan <ds...@hotmail.com>.
I think what is happening here is that we have different ideas about the architecture of the application.
Here's what I'm thinking:
*All processing of activity would happen BEFORE it's posted to Streams*All activity posted to Streams is in activitystrea.ms format
The Camel/EIP/OSGi way:
*The user is able to post anything to Streams, doesn't matter what format it's in*The user can add processing components to Streams that format everything that is input to activitystrea.ms format, these can be easily plugged in, moved around, or taken out
If we decide to go the second route, I think Camel would be the best way to go. However, I'd like to make the case for the first approach. The application can be quickly started and tested by a developer and user with the the first approach. If the user decides they want to experiment with twitter/facebook data (like Chris suggested) they can write a custom application (which can be a camel/eip application or even a python, ruby, etc application) that takes in the twitter/facebook data, formats it to activitystrea.ms format and then POSTs it to Streams. Also note: the webapp would NOT need to be restarted to add these. I don't think we'd lose a lot by going this route: in order to add the custom processing components in Camel, a Java component would still need to be written before it's plugged in to the camel.xml (and I think you would be tied to Java with this approach).
Quite simply: We should keep formatting activity separate processing activity
Perhaps I am missing some processing that could happen in between an activity's entrance to Streams and it's storage? If this is the case, I would make the argument that this processing should happen on an activities EXIT from Streams (there should be a way for a Subscriber to specify, in its filters or by some other means, the activity that it wants to see)
-Danny
> Date: Wed, 16 Oct 2013 10:11:23 -0700
> Subject: Re: Using Camel for Routing
> From: chris@cxtsoftware.com
> To: dev@streams.incubator.apache.org
>
> Jason,
>
> I agree with you. I guess my vision of streams was a little grander in the
> long run where you could configure the system to accept input from various
> sources (facebook, twitter...) and process them in certain ways. So it
> would have components that would know how to talk with Twitter and convert
> tweets to activities then send it to the right process. So depending on how
> you configure your system, it would create the proper routes from component
> to component to allow processing dynamically which would require a
> framework like Camel (don't think Storm helps us here since it's a compile
> time configuration to my understanding). The nice thing about something
> like camel is it allows you to reconfigure all the routes at runtime. This
> is another reason I liked OSGI support as well as it allows deploying new
> components without restarting the server (webapp). Of course to scale it
> would require being able to deploy the configuration across multiple
> machines which gets fun.
>
> Maybe my vision is too big for this project but it's where my head has been
> at.
>
> Chris
>
>
> On Wed, Oct 16, 2013 at 7:54 AM, Jason Letourneau
> <jl...@gmail.com>wrote:
>
> > My only real complaint with this approach is that interjecting new
> > components (processing components) becomes less plug and play without
> > writing Java code in streams - with the Camel routing approach,
> > streams Java is a black box and you just change the camel xml.
> >
> > The primary rationale in using Camel was gaining all of the
> > architectural benefits one gains from enterprise integration patterns
> > (truth: simplicity is not one and is a tradeoff for flexibility, but
> > generally is considered a universal truth not one relegated to Camel
> > and EIP).
> >
> > Jason
> >
> > On Fri, Oct 11, 2013 at 10:16 AM, Danny Sullivan <ds...@hotmail.com>
> > wrote:
> > > Hey All,
> > > I would like to merge the webservice branch with trunk. I think making
> > streams web service based allows for a better focus on the goal of the
> > application: process activitystrea.ms formatted activity json. While
> > moving to a webservice model would lose us support for many of the use
> > cases Chris mentioned (i.e twitter and facebook), I think the downsides
> > would be mitigated by allowing and encouraging users to write custom
> > appenders that take in twitter/facebook data, format it to
> > activitystrea.ms standards, and POST to the Streams Server. These
> > appenders could even be written in any language as long as the output is
> > activitystrea.ms json. Compared to the trunk branch, I think the web
> > service design employs the "black box" idiom in a much simpler fashion. I
> > would be happy to hear contradicting thoughts and would encourage a
> > discussion on this design switch.
> > > Thanks!
> > > Danny
> > >> From: dsullivan7@hotmail.com
> > >> To: dev@streams.incubator.apache.org
> > >> Subject: RE: Using Camel for Routing
> > >> Date: Fri, 4 Oct 2013 16:07:06 -0400
> > >>
> > >> I wrote up something quickly so I could demo the functionality as a web
> > service as opposed to Camel. The new branch is available here:
> > >> https://svn.apache.org/repos/asf/incubator/streams/branches/webservice/
> > >> note that the new publishing and subscribing urls are:
> > >>
> > http://localhost:8080/streams-web/app/publisherRegisterhttp://localhost:8080/streams-web/app/subscriberRegister
> > >> I used a Spring Web Service only because I was more familiar with it, I
> > would like to research other alternatives (CXF) if it's clear that a web
> > service is a realistic alternative. The main changes to the application are
> > the web controller located here:
> > >>
> > https://svn.apache.org/repos/asf/incubator/streams/branches/webservice/streams-web/src/main/java/org/apache/streams/mvc/controller/StreamsWebController.java
> > >> Which handles the input and output of the application, which were
> > handled by the camel context before. The other main change is the use of
> > the streams-components module located here:
> > >>
> > https://svn.apache.org/repos/asf/incubator/streams/branches/webservice/streams-components/
> > >> This module contains service classes that return the correct output on
> > each request sent to the controller. I reused the Publisher and Subscriber
> > objects as well as the activityConsumerWarehouse and
> > activitySubscriberWarehouse, I think they would be moved to the
> > streams-components package if we want to go this route.
> > >> Overall, I think the flow through the application is much more
> > understandable. Previously, a developer would have to follow an activity
> > through the camelContext.xml as well as through dynamically created routes
> > verses the web service which provides clear entry and exit points to the
> > application. I'd really like to hear feedback on this one, I really don't
> > want to lose any efficiencies that we could be from
> > messaging/osgi/camel/activemq.
> > >> Thanks,
> > >> Danny
> > >>
> > >> > From: dsullivan7@hotmail.com
> > >> > To: dev@streams.incubator.apache.org
> > >> > Subject: RE: Using Camel for Routing
> > >> > Date: Thu, 3 Oct 2013 20:17:13 -0400
> > >> >
> > >> > Thanks for the update Jason. I'll keep the Spring MVC app in a
> > separate branch (hopefully it won't take too long) and then perhaps we can
> > reassess having both component and web container deployment? I think it's a
> > good idea to keep both options in mind especially because this technology
> > is relatively young.
> > >> > Thanks!
> > >> > Danny
> > >> >
> > >> > > Date: Thu, 3 Oct 2013 19:50:59 -0400
> > >> > > Subject: Re: Using Camel for Routing
> > >> > > From: jletourneau80@gmail.com
> > >> > > To: dev@streams.incubator.apache.org
> > >> > >
> > >> > > Because of the osgi container support in addition to deployment as
> > a web
> > >> > > app - when last we visited this topic - I think the majority was in
> > favor
> > >> > > of keeping the component style deployment in addition to web
> > container
> > >> > > deployment
> > >> > >
> > >> > > On Thursday, October 3, 2013, Danny Sullivan wrote:
> > >> > >
> > >> > > > Hey all,
> > >> > > > I've been considering taking a stab at moving streams from it's
> > current
> > >> > > > configuration with Camel to a Spring Web Service. Concerns over
> > failover
> > >> > > > and load balancing would be mitigated by the Cassandra DB
> > (running on 9160)
> > >> > > > and Storm (which depends on an instance of Zookeeper on 2181).
> > Before I
> > >> > > > start lancing any windmills I thought it'd be nice to get input
> > on why the
> > >> > > > design choice was made to base the application around Camel?
> > >> > > > I think that tracking the flow of an activity through the
> > application will
> > >> > > > be much easier if we use Spring MVC. I'll set up a new branch
> > using the MVC
> > >> > > > architecture and I'll keep the community updated as I go along.
> > >> > > > Danny
> > >> >
> > >>
> > >
> >
Re: Using Camel for Routing
Posted by Chris Geer <ch...@cxtsoftware.com>.
Jason,
I agree with you. I guess my vision of streams was a little grander in the
long run where you could configure the system to accept input from various
sources (facebook, twitter...) and process them in certain ways. So it
would have components that would know how to talk with Twitter and convert
tweets to activities then send it to the right process. So depending on how
you configure your system, it would create the proper routes from component
to component to allow processing dynamically which would require a
framework like Camel (don't think Storm helps us here since it's a compile
time configuration to my understanding). The nice thing about something
like camel is it allows you to reconfigure all the routes at runtime. This
is another reason I liked OSGI support as well as it allows deploying new
components without restarting the server (webapp). Of course to scale it
would require being able to deploy the configuration across multiple
machines which gets fun.
Maybe my vision is too big for this project but it's where my head has been
at.
Chris
On Wed, Oct 16, 2013 at 7:54 AM, Jason Letourneau
<jl...@gmail.com>wrote:
> My only real complaint with this approach is that interjecting new
> components (processing components) becomes less plug and play without
> writing Java code in streams - with the Camel routing approach,
> streams Java is a black box and you just change the camel xml.
>
> The primary rationale in using Camel was gaining all of the
> architectural benefits one gains from enterprise integration patterns
> (truth: simplicity is not one and is a tradeoff for flexibility, but
> generally is considered a universal truth not one relegated to Camel
> and EIP).
>
> Jason
>
> On Fri, Oct 11, 2013 at 10:16 AM, Danny Sullivan <ds...@hotmail.com>
> wrote:
> > Hey All,
> > I would like to merge the webservice branch with trunk. I think making
> streams web service based allows for a better focus on the goal of the
> application: process activitystrea.ms formatted activity json. While
> moving to a webservice model would lose us support for many of the use
> cases Chris mentioned (i.e twitter and facebook), I think the downsides
> would be mitigated by allowing and encouraging users to write custom
> appenders that take in twitter/facebook data, format it to
> activitystrea.ms standards, and POST to the Streams Server. These
> appenders could even be written in any language as long as the output is
> activitystrea.ms json. Compared to the trunk branch, I think the web
> service design employs the "black box" idiom in a much simpler fashion. I
> would be happy to hear contradicting thoughts and would encourage a
> discussion on this design switch.
> > Thanks!
> > Danny
> >> From: dsullivan7@hotmail.com
> >> To: dev@streams.incubator.apache.org
> >> Subject: RE: Using Camel for Routing
> >> Date: Fri, 4 Oct 2013 16:07:06 -0400
> >>
> >> I wrote up something quickly so I could demo the functionality as a web
> service as opposed to Camel. The new branch is available here:
> >> https://svn.apache.org/repos/asf/incubator/streams/branches/webservice/
> >> note that the new publishing and subscribing urls are:
> >>
> http://localhost:8080/streams-web/app/publisherRegisterhttp://localhost:8080/streams-web/app/subscriberRegister
> >> I used a Spring Web Service only because I was more familiar with it, I
> would like to research other alternatives (CXF) if it's clear that a web
> service is a realistic alternative. The main changes to the application are
> the web controller located here:
> >>
> https://svn.apache.org/repos/asf/incubator/streams/branches/webservice/streams-web/src/main/java/org/apache/streams/mvc/controller/StreamsWebController.java
> >> Which handles the input and output of the application, which were
> handled by the camel context before. The other main change is the use of
> the streams-components module located here:
> >>
> https://svn.apache.org/repos/asf/incubator/streams/branches/webservice/streams-components/
> >> This module contains service classes that return the correct output on
> each request sent to the controller. I reused the Publisher and Subscriber
> objects as well as the activityConsumerWarehouse and
> activitySubscriberWarehouse, I think they would be moved to the
> streams-components package if we want to go this route.
> >> Overall, I think the flow through the application is much more
> understandable. Previously, a developer would have to follow an activity
> through the camelContext.xml as well as through dynamically created routes
> verses the web service which provides clear entry and exit points to the
> application. I'd really like to hear feedback on this one, I really don't
> want to lose any efficiencies that we could be from
> messaging/osgi/camel/activemq.
> >> Thanks,
> >> Danny
> >>
> >> > From: dsullivan7@hotmail.com
> >> > To: dev@streams.incubator.apache.org
> >> > Subject: RE: Using Camel for Routing
> >> > Date: Thu, 3 Oct 2013 20:17:13 -0400
> >> >
> >> > Thanks for the update Jason. I'll keep the Spring MVC app in a
> separate branch (hopefully it won't take too long) and then perhaps we can
> reassess having both component and web container deployment? I think it's a
> good idea to keep both options in mind especially because this technology
> is relatively young.
> >> > Thanks!
> >> > Danny
> >> >
> >> > > Date: Thu, 3 Oct 2013 19:50:59 -0400
> >> > > Subject: Re: Using Camel for Routing
> >> > > From: jletourneau80@gmail.com
> >> > > To: dev@streams.incubator.apache.org
> >> > >
> >> > > Because of the osgi container support in addition to deployment as
> a web
> >> > > app - when last we visited this topic - I think the majority was in
> favor
> >> > > of keeping the component style deployment in addition to web
> container
> >> > > deployment
> >> > >
> >> > > On Thursday, October 3, 2013, Danny Sullivan wrote:
> >> > >
> >> > > > Hey all,
> >> > > > I've been considering taking a stab at moving streams from it's
> current
> >> > > > configuration with Camel to a Spring Web Service. Concerns over
> failover
> >> > > > and load balancing would be mitigated by the Cassandra DB
> (running on 9160)
> >> > > > and Storm (which depends on an instance of Zookeeper on 2181).
> Before I
> >> > > > start lancing any windmills I thought it'd be nice to get input
> on why the
> >> > > > design choice was made to base the application around Camel?
> >> > > > I think that tracking the flow of an activity through the
> application will
> >> > > > be much easier if we use Spring MVC. I'll set up a new branch
> using the MVC
> >> > > > architecture and I'll keep the community updated as I go along.
> >> > > > Danny
> >> >
> >>
> >
>
Re: Using Camel for Routing
Posted by Jason Letourneau <jl...@gmail.com>.
My only real complaint with this approach is that interjecting new
components (processing components) becomes less plug and play without
writing Java code in streams - with the Camel routing approach,
streams Java is a black box and you just change the camel xml.
The primary rationale in using Camel was gaining all of the
architectural benefits one gains from enterprise integration patterns
(truth: simplicity is not one and is a tradeoff for flexibility, but
generally is considered a universal truth not one relegated to Camel
and EIP).
Jason
On Fri, Oct 11, 2013 at 10:16 AM, Danny Sullivan <ds...@hotmail.com> wrote:
> Hey All,
> I would like to merge the webservice branch with trunk. I think making streams web service based allows for a better focus on the goal of the application: process activitystrea.ms formatted activity json. While moving to a webservice model would lose us support for many of the use cases Chris mentioned (i.e twitter and facebook), I think the downsides would be mitigated by allowing and encouraging users to write custom appenders that take in twitter/facebook data, format it to activitystrea.ms standards, and POST to the Streams Server. These appenders could even be written in any language as long as the output is activitystrea.ms json. Compared to the trunk branch, I think the web service design employs the "black box" idiom in a much simpler fashion. I would be happy to hear contradicting thoughts and would encourage a discussion on this design switch.
> Thanks!
> Danny
>> From: dsullivan7@hotmail.com
>> To: dev@streams.incubator.apache.org
>> Subject: RE: Using Camel for Routing
>> Date: Fri, 4 Oct 2013 16:07:06 -0400
>>
>> I wrote up something quickly so I could demo the functionality as a web service as opposed to Camel. The new branch is available here:
>> https://svn.apache.org/repos/asf/incubator/streams/branches/webservice/
>> note that the new publishing and subscribing urls are:
>> http://localhost:8080/streams-web/app/publisherRegisterhttp://localhost:8080/streams-web/app/subscriberRegister
>> I used a Spring Web Service only because I was more familiar with it, I would like to research other alternatives (CXF) if it's clear that a web service is a realistic alternative. The main changes to the application are the web controller located here:
>> https://svn.apache.org/repos/asf/incubator/streams/branches/webservice/streams-web/src/main/java/org/apache/streams/mvc/controller/StreamsWebController.java
>> Which handles the input and output of the application, which were handled by the camel context before. The other main change is the use of the streams-components module located here:
>> https://svn.apache.org/repos/asf/incubator/streams/branches/webservice/streams-components/
>> This module contains service classes that return the correct output on each request sent to the controller. I reused the Publisher and Subscriber objects as well as the activityConsumerWarehouse and activitySubscriberWarehouse, I think they would be moved to the streams-components package if we want to go this route.
>> Overall, I think the flow through the application is much more understandable. Previously, a developer would have to follow an activity through the camelContext.xml as well as through dynamically created routes verses the web service which provides clear entry and exit points to the application. I'd really like to hear feedback on this one, I really don't want to lose any efficiencies that we could be from messaging/osgi/camel/activemq.
>> Thanks,
>> Danny
>>
>> > From: dsullivan7@hotmail.com
>> > To: dev@streams.incubator.apache.org
>> > Subject: RE: Using Camel for Routing
>> > Date: Thu, 3 Oct 2013 20:17:13 -0400
>> >
>> > Thanks for the update Jason. I'll keep the Spring MVC app in a separate branch (hopefully it won't take too long) and then perhaps we can reassess having both component and web container deployment? I think it's a good idea to keep both options in mind especially because this technology is relatively young.
>> > Thanks!
>> > Danny
>> >
>> > > Date: Thu, 3 Oct 2013 19:50:59 -0400
>> > > Subject: Re: Using Camel for Routing
>> > > From: jletourneau80@gmail.com
>> > > To: dev@streams.incubator.apache.org
>> > >
>> > > Because of the osgi container support in addition to deployment as a web
>> > > app - when last we visited this topic - I think the majority was in favor
>> > > of keeping the component style deployment in addition to web container
>> > > deployment
>> > >
>> > > On Thursday, October 3, 2013, Danny Sullivan wrote:
>> > >
>> > > > Hey all,
>> > > > I've been considering taking a stab at moving streams from it's current
>> > > > configuration with Camel to a Spring Web Service. Concerns over failover
>> > > > and load balancing would be mitigated by the Cassandra DB (running on 9160)
>> > > > and Storm (which depends on an instance of Zookeeper on 2181). Before I
>> > > > start lancing any windmills I thought it'd be nice to get input on why the
>> > > > design choice was made to base the application around Camel?
>> > > > I think that tracking the flow of an activity through the application will
>> > > > be much easier if we use Spring MVC. I'll set up a new branch using the MVC
>> > > > architecture and I'll keep the community updated as I go along.
>> > > > Danny
>> >
>>
>
RE: Using Camel for Routing
Posted by Danny Sullivan <ds...@hotmail.com>.
Hey All,
I would like to merge the webservice branch with trunk. I think making streams web service based allows for a better focus on the goal of the application: process activitystrea.ms formatted activity json. While moving to a webservice model would lose us support for many of the use cases Chris mentioned (i.e twitter and facebook), I think the downsides would be mitigated by allowing and encouraging users to write custom appenders that take in twitter/facebook data, format it to activitystrea.ms standards, and POST to the Streams Server. These appenders could even be written in any language as long as the output is activitystrea.ms json. Compared to the trunk branch, I think the web service design employs the "black box" idiom in a much simpler fashion. I would be happy to hear contradicting thoughts and would encourage a discussion on this design switch.
Thanks!
Danny
> From: dsullivan7@hotmail.com
> To: dev@streams.incubator.apache.org
> Subject: RE: Using Camel for Routing
> Date: Fri, 4 Oct 2013 16:07:06 -0400
>
> I wrote up something quickly so I could demo the functionality as a web service as opposed to Camel. The new branch is available here:
> https://svn.apache.org/repos/asf/incubator/streams/branches/webservice/
> note that the new publishing and subscribing urls are:
> http://localhost:8080/streams-web/app/publisherRegisterhttp://localhost:8080/streams-web/app/subscriberRegister
> I used a Spring Web Service only because I was more familiar with it, I would like to research other alternatives (CXF) if it's clear that a web service is a realistic alternative. The main changes to the application are the web controller located here:
> https://svn.apache.org/repos/asf/incubator/streams/branches/webservice/streams-web/src/main/java/org/apache/streams/mvc/controller/StreamsWebController.java
> Which handles the input and output of the application, which were handled by the camel context before. The other main change is the use of the streams-components module located here:
> https://svn.apache.org/repos/asf/incubator/streams/branches/webservice/streams-components/
> This module contains service classes that return the correct output on each request sent to the controller. I reused the Publisher and Subscriber objects as well as the activityConsumerWarehouse and activitySubscriberWarehouse, I think they would be moved to the streams-components package if we want to go this route.
> Overall, I think the flow through the application is much more understandable. Previously, a developer would have to follow an activity through the camelContext.xml as well as through dynamically created routes verses the web service which provides clear entry and exit points to the application. I'd really like to hear feedback on this one, I really don't want to lose any efficiencies that we could be from messaging/osgi/camel/activemq.
> Thanks,
> Danny
>
> > From: dsullivan7@hotmail.com
> > To: dev@streams.incubator.apache.org
> > Subject: RE: Using Camel for Routing
> > Date: Thu, 3 Oct 2013 20:17:13 -0400
> >
> > Thanks for the update Jason. I'll keep the Spring MVC app in a separate branch (hopefully it won't take too long) and then perhaps we can reassess having both component and web container deployment? I think it's a good idea to keep both options in mind especially because this technology is relatively young.
> > Thanks!
> > Danny
> >
> > > Date: Thu, 3 Oct 2013 19:50:59 -0400
> > > Subject: Re: Using Camel for Routing
> > > From: jletourneau80@gmail.com
> > > To: dev@streams.incubator.apache.org
> > >
> > > Because of the osgi container support in addition to deployment as a web
> > > app - when last we visited this topic - I think the majority was in favor
> > > of keeping the component style deployment in addition to web container
> > > deployment
> > >
> > > On Thursday, October 3, 2013, Danny Sullivan wrote:
> > >
> > > > Hey all,
> > > > I've been considering taking a stab at moving streams from it's current
> > > > configuration with Camel to a Spring Web Service. Concerns over failover
> > > > and load balancing would be mitigated by the Cassandra DB (running on 9160)
> > > > and Storm (which depends on an instance of Zookeeper on 2181). Before I
> > > > start lancing any windmills I thought it'd be nice to get input on why the
> > > > design choice was made to base the application around Camel?
> > > > I think that tracking the flow of an activity through the application will
> > > > be much easier if we use Spring MVC. I'll set up a new branch using the MVC
> > > > architecture and I'll keep the community updated as I go along.
> > > > Danny
> >
>
RE: Using Camel for Routing
Posted by Danny Sullivan <ds...@hotmail.com>.
I wrote up something quickly so I could demo the functionality as a web service as opposed to Camel. The new branch is available here:
https://svn.apache.org/repos/asf/incubator/streams/branches/webservice/
note that the new publishing and subscribing urls are:
http://localhost:8080/streams-web/app/publisherRegisterhttp://localhost:8080/streams-web/app/subscriberRegister
I used a Spring Web Service only because I was more familiar with it, I would like to research other alternatives (CXF) if it's clear that a web service is a realistic alternative. The main changes to the application are the web controller located here:
https://svn.apache.org/repos/asf/incubator/streams/branches/webservice/streams-web/src/main/java/org/apache/streams/mvc/controller/StreamsWebController.java
Which handles the input and output of the application, which were handled by the camel context before. The other main change is the use of the streams-components module located here:
https://svn.apache.org/repos/asf/incubator/streams/branches/webservice/streams-components/
This module contains service classes that return the correct output on each request sent to the controller. I reused the Publisher and Subscriber objects as well as the activityConsumerWarehouse and activitySubscriberWarehouse, I think they would be moved to the streams-components package if we want to go this route.
Overall, I think the flow through the application is much more understandable. Previously, a developer would have to follow an activity through the camelContext.xml as well as through dynamically created routes verses the web service which provides clear entry and exit points to the application. I'd really like to hear feedback on this one, I really don't want to lose any efficiencies that we could be from messaging/osgi/camel/activemq.
Thanks,
Danny
> From: dsullivan7@hotmail.com
> To: dev@streams.incubator.apache.org
> Subject: RE: Using Camel for Routing
> Date: Thu, 3 Oct 2013 20:17:13 -0400
>
> Thanks for the update Jason. I'll keep the Spring MVC app in a separate branch (hopefully it won't take too long) and then perhaps we can reassess having both component and web container deployment? I think it's a good idea to keep both options in mind especially because this technology is relatively young.
> Thanks!
> Danny
>
> > Date: Thu, 3 Oct 2013 19:50:59 -0400
> > Subject: Re: Using Camel for Routing
> > From: jletourneau80@gmail.com
> > To: dev@streams.incubator.apache.org
> >
> > Because of the osgi container support in addition to deployment as a web
> > app - when last we visited this topic - I think the majority was in favor
> > of keeping the component style deployment in addition to web container
> > deployment
> >
> > On Thursday, October 3, 2013, Danny Sullivan wrote:
> >
> > > Hey all,
> > > I've been considering taking a stab at moving streams from it's current
> > > configuration with Camel to a Spring Web Service. Concerns over failover
> > > and load balancing would be mitigated by the Cassandra DB (running on 9160)
> > > and Storm (which depends on an instance of Zookeeper on 2181). Before I
> > > start lancing any windmills I thought it'd be nice to get input on why the
> > > design choice was made to base the application around Camel?
> > > I think that tracking the flow of an activity through the application will
> > > be much easier if we use Spring MVC. I'll set up a new branch using the MVC
> > > architecture and I'll keep the community updated as I go along.
> > > Danny
>
RE: Using Camel for Routing
Posted by Danny Sullivan <ds...@hotmail.com>.
Thanks for the update Jason. I'll keep the Spring MVC app in a separate branch (hopefully it won't take too long) and then perhaps we can reassess having both component and web container deployment? I think it's a good idea to keep both options in mind especially because this technology is relatively young.
Thanks!
Danny
> Date: Thu, 3 Oct 2013 19:50:59 -0400
> Subject: Re: Using Camel for Routing
> From: jletourneau80@gmail.com
> To: dev@streams.incubator.apache.org
>
> Because of the osgi container support in addition to deployment as a web
> app - when last we visited this topic - I think the majority was in favor
> of keeping the component style deployment in addition to web container
> deployment
>
> On Thursday, October 3, 2013, Danny Sullivan wrote:
>
> > Hey all,
> > I've been considering taking a stab at moving streams from it's current
> > configuration with Camel to a Spring Web Service. Concerns over failover
> > and load balancing would be mitigated by the Cassandra DB (running on 9160)
> > and Storm (which depends on an instance of Zookeeper on 2181). Before I
> > start lancing any windmills I thought it'd be nice to get input on why the
> > design choice was made to base the application around Camel?
> > I think that tracking the flow of an activity through the application will
> > be much easier if we use Spring MVC. I'll set up a new branch using the MVC
> > architecture and I'll keep the community updated as I go along.
> > Danny
Re: Using Camel for Routing
Posted by Jason Letourneau <jl...@gmail.com>.
Because of the osgi container support in addition to deployment as a web
app - when last we visited this topic - I think the majority was in favor
of keeping the component style deployment in addition to web container
deployment
On Thursday, October 3, 2013, Danny Sullivan wrote:
> Hey all,
> I've been considering taking a stab at moving streams from it's current
> configuration with Camel to a Spring Web Service. Concerns over failover
> and load balancing would be mitigated by the Cassandra DB (running on 9160)
> and Storm (which depends on an instance of Zookeeper on 2181). Before I
> start lancing any windmills I thought it'd be nice to get input on why the
> design choice was made to base the application around Camel?
> I think that tracking the flow of an activity through the application will
> be much easier if we use Spring MVC. I'll set up a new branch using the MVC
> architecture and I'll keep the community updated as I go along.
> Danny