Re: Starting the project

[Renato Marroquín Mogrovejo <re...@gmail.com>]

Just to finish this thread.

@Matt
+1

@Steve
I found that Apache Giraph tests their algorithms with Mockito[1], but we
could other frameworks like Easymock , or JMock if we decide to go down
that road.


Renato M.

[1] https://code.google.com/p/mockito/


2014-05-22 22:46 GMT+02:00 Matt Franklin <m....@gmail.com>:

> On Thu, May 22, 2014 at 11:46 AM, Steve Blackmon <sblackmon@apache.org
> >wrote:
>
> > Renato,
> >
> > Yes there's no question we need to update the website - not least to
> > help newcomers locate and get started with the examples.
> >
> > I would like to see us creating more integration tests like
> > Top100FeedsTest which source and pass current-time real-world data
> > through provider modules to confirm they work.  Social APIs are
> > continuously evolving and this could help us catch any changes that we
> > weren't aware of and prepared for prior to the change.
> >
> > Persistance module testing should also be done and would probably look
> > a lot like gora's testing.  Let's follow up on ideas for aligning
> > efforts along those lines.  A 10-20 minute build when integration
> > testing is enabled isn't unreasonable.
> >
> > I'm not sure you meant to suggest it explicitly, but this note
> > triggered a thought that the contrib modules perhaps could be split
> > out into another repo as well.  Like examples, a failing test within a
> > specific datasource or database integration shouldn't necessarily
> > break the whole build.  APIs aren't perfect and may have transient
> > issues, and given that our poms lean toward the bleeding edge
> > occasionally database or database driver bugs are going to cause test
> > failures.
> >
> > I like the way storm has handled this - they've use git submodules to
> > associate useful integrations that make storm easier to build on with
> > the top-level project, while essentially delegating maintenance of
> > those modules to other projects/companies.  A similar model could work
> > well for streams.
> >
>
> IMO, we should avoid git submodules for this.  I also don't think it needs
> to be another entire repo, it could just be moved to a separate top-level
> directory
>
>
> >
> > Thanks for the suggestions!
> >
> > Steve Blackmon
> > sblackmon@apache.org
> >
> > On Thu, May 22, 2014 at 7:22 AM, Renato Marroquín Mogrovejo
> > <re...@gmail.com> wrote:
> > > Maybe even something like this should be done but for the contrib
> > modules.
> > >
> > >
> >
> https://github.com/apache/incubator-streams/blob/master/streams-contrib/streams-provider-rss/src/test/java/org/apache/streams/rss/test/Top100FeedsTest.java
> > >
> > > That would be the something like examples. Wdyt?
> > >
> > >
> > > Renato M.
> > >
> > >
> > > 2014-05-22 12:05 GMT+02:00 Renato Marroquín Mogrovejo <
> > > renatoj.marroquin@gmail.com>:
> > >
> > >> Hi Steve,
> > >>
> > >> Comments inline.
> > >>
> > >> 2014-05-21 20:59 GMT+02:00 Steve Blackmon <sb...@apache.org>:
> > >>
> > >> I will happily accept pull requests to streams-examples, although I
> > >>> don't think it would need to change for an update to the provider to
> > >>> be applied (except perhaps to add that verb filtering step)
> > >>>
> > >>
> > >> This is what I meant.
> > >>
> > >>
> > >>> I am willing to donate all of the examples, but I don't think they
> > >>> belong in the main repo.  What if there were 50 examples many
> > >>> containing integration tests?  Build would take forever - plus I
> don't
> > >>> think that a single example failing a test should break the build.
> > >>> Mahout suffered from really long-builds and handled it by separating
> > >>> the examples out from core.
> > >>>
> > >>
> > >> Well, over in Apache Giraph they have over 20 graph example algorithms
> > and
> > >> the way they test them is just by doing some simple computations, I
> > don't
> > >> know how long it takes, but I think it makes sense to at least try it
> > and
> > >> see how long it takes.
> > >> Over in Gora we have a test suite with the different operations
> > supported
> > >> that is run over the different data stores, and depending on your
> > machine,
> > >> it might take from 10 - 20 mins. There was a time where the HBase
> tests
> > >> were taking around 30 mins by themselves but we fixed them and are now
> > run
> > >> in less than 5 mins.
> > >> Maybe your examples could be converted into a test suite? I mean
> reading
> > >> from twitter(or a file) and persisting into different places. But
> > anyhow,
> > >> your examples should be referenced somewhere as it is what it makes
> > sense
> > >> for a newcomer :D
> > >>
> > >> When the project was hosting in SVN, we had the ability to create and
> > >>> checkout different roots (the master pom was in a different root for
> > >>> example).  If we can do the same using Git I propose we put the
> > >>> examples in as a flat multi-module project that way.
> > >>>
> > >>
> > >> +1
> > >>
> > >>
> > >> Renato M.
> > >>
> > >> Any thoughts?
> > >>>
> > >>> Steve Blackmon
> > >>> sblackmon@apache.org
> > >>>
> > >>> On Wed, May 21, 2014 at 11:59 AM, Renato Marroquín Mogrovejo
> > >>> <re...@gmail.com> wrote:
> > >>> > I tried updating the site through ApacheCMS but I guess I don't
> have
> > the
> > >>> > rights to do it ;)
> > >>> > This is what I got:
> > >>> >
> > >>> > The operation is forbidden by the server: Commit failed (details
> > >>> follow)::
> > >>> > Changing file
> > >>> >
> > >>>
> >
> '/usr/local/cms/wc/streams/rmarroquin-NlOO6g/trunk/content/source-repository.mdtext'
> > >>> > is forbidden by the server: Access to
> > >>> >
> > >>>
> >
> '/repos/asf/!svn/txr/1596615-yz8u/incubator/streams/site/trunk/content/source-repository.mdtext'
> > >>> > forbidden at /usr/local/cms/webgui/lib/ASF/CMS/WC/Commit.pm line 64
> > >>> >
> > >>> > So what is the best/easy way to update the site?
> > >>> >
> > >>> > @Steve
> > >>> > Yeah, I finally got the time to try this out :) and looks very
> > promising
> > >>> > for sure!
> > >>> > Is anybody working on STREAMS-57? Could I pick it up? I think this
> > >>> should
> > >>> > be pretty straight forward.
> > >>> > And are there any plans to put your examples into the code base?
> It's
> > >>> quite
> > >>> > puzzling going from repo to another (plus the one from the site) to
> > >>> > accomplish this. And then if I could work on STREAMS-57, would I
> have
> > >>> to do
> > >>> > the pull request into W2OGroup/streams-examples repo? I don't think
> > so
> > >>> > right? how would this work?
> > >>> > Thanks!
> > >>> >
> > >>> >
> > >>> > Renato M.
> > >>> >
> > >>> >
> > >>> > 2014-05-21 17:42 GMT+02:00 Steve Blackmon <st...@blackmon.org>:
> > >>> >
> > >>> >> Cool I'm glad to hear you got it running. That exception occurs
> > because
> > >>> >> twitter sends a preamble listing all of your friends when you
> open a
> > >>> >> stream. The provider doesn't have logic to handle it yet, so it
> > tries
> > >>> to
> > >>> >> build an activity and throws that exception.
> > >>> >>
> > >>> >> STREAMS-57 would suppress this warning - although once non-content
> > >>> >> activities with new verbs like follow are generated we will want a
> > >>> >> filtering mechanism in place so it remains easy to build indexes
> of
> > >>> just
> > >>> >> shares and posts.
> > >>> >>
> > >>> >> On Wed, May 21, 2014 at 9:23 AM, Renato Marroquín Mogrovejo <
> > >>> >> renatoj.marroquin@gmail.com> wrote:
> > >>> >> > Hey Matt,
> > >>> >> >
> > >>> >> > So the examples worked :D at least the one retrieving the
> twitter
> > >>> user's
> > >>> >> > history. The one trying to track users activity keeps on getting
> > this
> > >>> >> > exception:
> > >>> >> >
> > >>> >> > 16:09:11.151 [pool-2-thread-2] DEBUG
> > o.a.s.t.p.TwitterTypeConverter -
> > >>> >> > ACTIVITY
> > >>> >> > org.apache.streams.exceptions.ActivitySerializerException:
> > >>> unrecognized
> > >>> >> type
> > >>> >> > at
> > >>> >> >
> > >>> >>
> > >>> >>
> > >>>
> >
> org.apache.streams.twitter.serializer.TwitterJsonActivitySerializer.deserialize(TwitterJsonActivitySerializer.java:74)
> > >>> >> > at
> > >>> >> >
> > >>> >>
> > >>> >>
> > >>>
> >
> org.apache.streams.twitter.processor.TwitterTypeConverter.convert(TwitterTypeConverter.java:65)
> > >>> >> > at
> > >>> >> >
> > >>> >>
> > >>> >>
> > >>>
> >
> org.apache.streams.twitter.processor.TwitterTypeConverter.process(TwitterTypeConverter.java:146)
> > >>> >> > at
> > >>> >> >
> > >>> >>
> > >>> >>
> > >>>
> >
> org.apache.streams.local.tasks.StreamsProcessorTask.run(StreamsProcessorTask.java:72)
> > >>> >> > at
> > >>> >>
> > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> > >>> >> > at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> > >>> >> > at
> > >>> >> >
> > >>> >>
> > >>> >>
> > >>>
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> > >>> >> > at
> > >>> >> >
> > >>> >>
> > >>> >>
> > >>>
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> > >>> >> > at java.lang.Thread.run(Thread.java:744)
> > >>> >> >
> > >>> >> > But everything seems to work in spite of that exception :) Cool
> > >>> stuff! So
> > >>> >> > can I open a JIRA issue for updating the site and putting the
> > patch
> > >>> there
> > >>> >> > then?
> > >>> >> > I will write another email with some other issues.
> > >>> >> > Thanks Matt!
> > >>> >> >
> > >>> >> >
> > >>> >> > Renato M.
> > >>> >> >
> > >>> >> >
> > >>> >> > 2014-05-21 15:59 GMT+02:00 Matt Franklin <
> > m.ben.franklin@gmail.com>:
> > >>> >> >
> > >>> >> >> On Wed, May 21, 2014 at 9:00 AM, Renato Marroquín Mogrovejo <
> > >>> >> >> renatoj.marroquin@gmail.com> wrote:
> > >>> >> >>
> > >>> >> >> > Hi Matt,
> > >>> >> >> >
> > >>> >> >> > Thank for pointing me to these examples. I will try them out
> > and
> > >>> >> report
> > >>> >> >> > back.
> > >>> >> >> > But I have more questions now. I have found:
> > >>> >> >> >
> > >>> >> >> > https://github.com/apache/incubator-streams --> referenced
> by
> > the
> > >>> >> >> examples
> > >>> >> >> > you sent me and more active
> > >>> >> >> > https://github.com/apache/streams --> referenced by the
> > website
> > >>> >> >> >
> > >>> >> >>
> > >>> >> >> Oops. Yes it is the first one. The second one was a mirror of
> > SVN.
> > >>> >> >>
> > >>> >> >>
> > >>> >> >> >
> > >>> >> >> > Which one is the official one? I guess the first one right.
> > >>> Shouldn't
> > >>> >> we
> > >>> >> >> > change this on the website? does that mean that Apache
> Streams
> > >>> have
> > >>> >> two
> > >>> >> >> git
> > >>> >> >> > repos? I can fix that on the website if you point me to the
> > right
> > >>> >> place.
> > >>> >> >> >
> > >>> >> >>
> > >>> >> >> Website is in SVN using Apache CMS.
> > >>> >> >>
> > >>> >> >>
> > >>> >> >> >
> > >>> >> >> >
> > >>> >> >> > Renato M.
> > >>> >> >> >
> > >>> >> >> >
> > >>> >> >> >
> > >>> >> >> > 2014-05-21 2:14 GMT+02:00 Matt Franklin <
> > m.ben.franklin@gmail.com
> > >>> >:
> > >>> >> >> >
> > >>> >> >> > > On Tue, May 20, 2014 at 4:28 PM, Renato Marroquín
> Mogrovejo <
> > >>> >> >> > > renatoj.marroquin@gmail.com> wrote:
> > >>> >> >> > >
> > >>> >> >> > > > Sorry I pressed the sent button before I finished writing
> > the
> > >>> >> email.
> > >>> >> >> > > > Anyways . . . Tomcat 7.0.39, ServiceMix 5.0.1 and
> Cassandra
> > >>> 2.0.4.
> > >>> >> >> > > > But I am getting this exception:
> > >>> >> >> > > >
> > >>> >> >> > >
> > >>> >> >> > > We are unfortunately in the middle of converging the new
> > model
> > >>> and
> > >>> >> the
> > >>> >> >> > old
> > >>> >> >> > > model. This runtime should be working, but it needs some
> > work to
> > >>> >> >> ensure
> > >>> >> >> > > that the new components in streams-contrib work. If you are
> > >>> looking
> > >>> >> to
> > >>> >> >> > get
> > >>> >> >> > > started with a newer runtime, streams-runtime-local is a
> good
> > >>> place.
> > >>> >> >> > >
> > >>> >> >> > > Steve posted a link to some examples he wrote for how to
> get
> > >>> >> started:
> > >>> >> >> > >
> > >>> >> >> > >
> > >>> >> >> > >
> > >>> >> >> >
> > >>> >> >>
> > >>> >>
> > >>> >>
> > >>>
> >
> https://github.com/w2ogroup/streams-examples/tree/master/twitter-history-elasticsearch
> > >>> >> >> > >
> > >>> >> >> > > We need to create some of our own examples (or ask him
> > kindly to
> > >>> >> donate
> > >>> >> >> > > some to us), but until the site is fixed, this is a good
> > start.
> > >>> I
> > >>> >> will
> > >>> >> >> > > create an issue for the Caml runtime issue you found here.
> > >>> >> >> > >
> > >>> >> >> > >
> > >>> >> >> > > >
> > >>> >> >> > > > May 20, 2014 10:15:29 PM
> > >>> org.apache.catalina.startup.ContextConfig
> > >>> >> >> > > > getServletContainerInitializer
> > >>> >> >> > > > SEVERE: The ServletContentInitializer
> > >>> >> >> > > >
> [org.springframework.web.SpringServletContainerInitializer]
> > >>> could
> > >>> >> not
> > >>> >> >> > be
> > >>> >> >> > > > created
> > >>> >> >> > > > java.lang.ClassNotFoundException:
> > >>> >> >> > > > org.springframework.web.SpringServletContainerInitializer
> > >>> >> >> > > > at
> > >>> >> >> > > >
> > >>> >> >> > > >
> > >>> >> >> > >
> > >>> >> >> >
> > >>> >> >>
> > >>> >>
> > >>> >>
> > >>>
> >
> org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1713)
> > >>> >> >> > > > at
> > >>> >> >> > > >
> > >>> >> >> > > >
> > >>> >> >> > >
> > >>> >> >> >
> > >>> >> >>
> > >>> >>
> > >>> >>
> > >>>
> >
> org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1558)
> > >>> >> >> > > > at java.lang.Class.forName0(Native Method)
> > >>> >> >> > > > at java.lang.Class.forName(Class.java:270)
> > >>> >> >> > > > at
> > >>> >> >> > > >
> > >>> >> >> > > >
> > >>> >> >> > >
> > >>> >> >> >
> > >>> >> >>
> > >>> >>
> > >>> >>
> > >>>
> >
> org.apache.catalina.startup.ContextConfig.getServletContainerInitializer(ContextConfig.java:1659)
> > >>> >> >> > > > at
> > >>> >> >> > > >
> > >>> >> >> > > >
> > >>> >> >> > >
> > >>> >> >> >
> > >>> >> >>
> > >>> >>
> > >>> >>
> > >>>
> >
> org.apache.catalina.startup.ContextConfig.processServletContainerInitializers(ContextConfig.java:1569)
> > >>> >> >> > > > at
> > >>> >> >> > > >
> > >>> >> >> > > >
> > >>> >> >> > >
> > >>> >> >> >
> > >>> >> >>
> > >>> >>
> > >>> >>
> > >>>
> >
> org.apache.catalina.startup.ContextConfig.webConfig(ContextConfig.java:1277)
> > >>> >> >> > > > at
> > >>> >> >> > > >
> > >>> >> >> > > >
> > >>> >> >> > >
> > >>> >> >> >
> > >>> >> >>
> > >>> >>
> > >>> >>
> > >>>
> >
> org.apache.catalina.startup.ContextConfig.configureStart(ContextConfig.java:878)
> > >>> >> >> > > > at
> > >>> >> >> > > >
> > >>> >> >> > > >
> > >>> >> >> > >
> > >>> >> >> >
> > >>> >> >>
> > >>> >>
> > >>> >>
> > >>>
> >
> org.apache.catalina.startup.ContextConfig.lifecycleEvent(ContextConfig.java:369)
> > >>> >> >> > > > at
> > >>> >> >> > > >
> > >>> >> >> > > >
> > >>> >> >> > >
> > >>> >> >> >
> > >>> >> >>
> > >>> >>
> > >>> >>
> > >>>
> >
> org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:119)
> > >>> >> >> > > > at
> > >>> >> >> > > >
> > >>> >> >> > > >
> > >>> >> >> > >
> > >>> >> >> >
> > >>> >> >>
> > >>> >>
> > >>> >>
> > >>>
> >
> org.apache.catalina.util.LifecycleBase.fireLifecycleEvent(LifecycleBase.java:90)
> > >>> >> >> > > > at
> > >>> >> >> > > >
> > >>> >> >> > > >
> > >>> >> >> > >
> > >>> >> >> >
> > >>> >> >>
> > >>> >>
> > >>> >>
> > >>>
> >
> org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5269)
> > >>> >> >> > > > at
> > >>> >> >> > > >
> > >>> >>
> org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
> > >>> >> >> > > >
> > >>> >> >> > > > That let me to [2]. Is this related to this? or is it
> just
> > me
> > >>> >> doing
> > >>> >> >> > > things
> > >>> >> >> > > > wrong here?
> > >>> >> >> > > > Thanks for any pointers!
> > >>> >> >> > > >
> > >>> >> >> > > >
> > >>> >> >> > > > Renato M.
> > >>> >> >> > > >
> > >>> >> >> > > > 2014-05-20 22:25 GMT+02:00 Renato Marroquín Mogrovejo <
> > >>> >> >> > > > renatoj.marroquin@gmail.com>:
> > >>> >> >> > > >
> > >>> >> >> > > > > Hey guys,
> > >>> >> >> > > > >
> > >>> >> >> > > > > This is a long overdue email but with little steps I
> will
> > >>> start
> > >>> >> >> going
> > >>> >> >> > > :)
> > >>> >> >> > > > > So I have been trying to run the whole project based
> > on[1].
> > >>> As
> > >>> >> >> there
> > >>> >> >> > is
> > >>> >> >> > > > no
> > >>> >> >> > > > > version for any of the pieces involved in running this
> > (if
> > >>> there
> > >>> >> is
> > >>> >> >> > > > please
> > >>> >> >> > > > > point me to them, if not, I am happy to put them on the
> > site
> > >>> >> >> > > somewhere).
> > >>> >> >> > > > I
> > >>> >> >> > > > > chose
> > >>> >> >> > > > >
> > >>> >> >> > > > >
> > >>> >> >> > > > >
> > >>> >> >> > > > > [1]
> > http://streams.incubator.apache.org/streams-usage.html
> > >>> >> >> > > > > [2]
> > >>> >> >> > > > >
> > >>> >> >> > > >
> > >>> >> >> > >
> > >>> >> >> >
> > >>> >> >>
> > >>> >>
> > >>> >>
> > >>>
> >
> http://stackoverflow.com/questions/7692497/tomcat-wont-load-my-meta-inf-services-javax-servlet-servletcontainerinitializ
> > >>> >> >> > > > >
> > >>> >> >> > > >
> > >>> >> >> > >
> > >>> >> >> >
> > >>> >> >>
> > >>> >>
> > >>>
> > >>
> > >>
> >
>