You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@stanbol.apache.org by Antonio David Perez Morales <ap...@zaizi.com> on 2014/06/02 19:00:57 UTC
Re: Camel integration (was : Re: Community bonding period started)
Hi stanbolers
As part of the issue [1] , I have created a maven archetype useful to
generate Camel routes in Java DSL.
The archetype generates a Java project with all the dependencies and one
Java class with a method which has to be filled. In this method, Camel Java
DSL syntax is used to create the route.
By default and as a first approach, the class will use the route name given
during the project creation to enable a Camel direct endpoint with such
name.
The code of the first archetype version can be found at [2].
The next task will be providing a Felix custom artifact to be able to
deploy XML-based routes in Stanbol, placing a custom file in the Stanbol
datafiles directory.
After that, it will be time to think and redesign the architecture to
integrate Camel workflows inside Stanbol in a better way, more configurable
and extendable.
Comments and suggestions are more than welcome
Regards
[1] https://issues.apache.org/jira/browse/STANBOL-1348
[2] https://github.com/adperezmorales/stanbol-camel-workflow/
On Fri, May 30, 2014 at 8:03 PM, Antonio David Perez Morales <
aperez@zaizi.com> wrote:
> Hi all
>
> After a hard fight this week, I managed to get it work the Florent's proof
> of concept code in the Stanbol 1.0 branch [1]
> The code is uploaded in my github account [3]. As I said in a previous
> mail, I prefer to do it separately and after the project, uploading the
> developed code into a Stanbol branch.
>
> The 1.0.0 version has some changes in how the Jersey endpoints are
> registered and also new classes and packages, so it was not a trivial task
> to make work the current proof of concept. Moreover I don't like to simply
> copy and paste code and make the needed changes. I always want to
> understand how the things work and how they are developed in order to be
> able to change/modify them or develop new code around them.
>
> The steps done to achieve it have been the following:
> - Updated pom files to the Stanbol 1.0.0-SNAPSHOT version
> - Updated bundle levels in bundlelist package to fit the Stanbol 1.0
> version levels
> - Adapted cameljobmanager package code to Stanbol 1.0.0-SNAPSHOT classes
> and using Java OSGI annotations instead of SCR annotations in Javadoc
> - Updated flow web package to Stanbol 1.0.0-SNAPSHOT classes and modified
> needed resources
> - Added Java OSGI annotations to the route (WeightedChain) instead of SCR
> annotations in javadoc
> - Updated launcher to use the 1.0.0-SNAPSHOT packages and needed bundles
>
> So now, the http://localhost:8080/flow endpoint will use the only Camel
> route (defined by WeightedChain) to call all the registered Enhancement
> Engines (ordered by EnhancementEngine order property).
> For testing purposes, the /flow/{flowName} has been removed, because all
> this code needs to be re-designed and re-implemented so I only wanted to
> make it work to have a first (simple) integration in Stanbol 1.0. This
> functionality will be added again to trigger custom routes once the next
> step (defined below) is developed.
>
> The next step [2] will be support to write and configure routes in XML
> format, putting the file in datafiles in order to be loaded by a Felix
> custom artifact (as Rupert pointed out in a previous mail) and create a
> Maven archetype to create bundles defining routes which will be loaded
> using the Felix bundle tab. If necessary, as we talked in previous
> messages, a REST endpoint receiving routes in XML can be developed as an
> alternative to the first approach. This is my objective for the midterm.
>
> After the midterm, the new Stanbol components for Apache Camel will be
> developed and also the new architecture for Camel in Stanbol.
>
> Comments on this and for use cases for Stanbol Camel components are more
> than welcome.
>
> Regards
>
> [1] https://issues.apache.org/jira/browse/STANBOL-1347
> [2] https://issues.apache.org/jira/browse/STANBOL-1348
> [3] https://github.com/adperezmorales/stanbol-camel-workflow/
>
>
> On Tue, May 27, 2014 at 6:18 PM, Antonio David Perez Morales <
> aperez@zaizi.com> wrote:
>
>> Hi people
>>
>> I have already started to work on [1] to integrate current Florent's code
>> into Stanbol 1.0.
>> As a first approach, only changing the dependency versions to new Stanbol
>> 1.0, many issues have arisen:
>> - Deprecated use of classes
>> - Classes which have changed from package
>> - Some classes not necessary now
>> - Classes not used which were causing conflicts
>> - ...
>>
>> So now I'm trying to resolve all these problems to replicate the same
>> behavior from 0.9 into 1.0. I will upload the code to a Github repository
>> in my account (which will be pushed later into a Stanbol branch after the
>> project) in order to track the advances.
>> Once I can resolve all these problems, I will take a look to the Felix
>> Custom Artifacts poiinted out by Rupert in a previous message to find out
>> the best way to deploy (and manage) route configurations (felix artifacts,
>> watchservice java, rest endpoint to receive xml routes, etc).
>>
>> Comments on this and future tasks are more than welcome.
>>
>> Regards
>>
>> [1] https://issues.apache.org/jira/browse/STANBOL-1347
>>
>>
>>
>> On Tue, May 27, 2014 at 9:53 AM, Rafa Haro <rh...@apache.org> wrote:
>>
>>> Hi Rupert, Florent and Antonio
>>>
>>> El 27/05/14 08:51, Rupert Westenthaler escribió:
>>>
>>> As the result of Enhancement Routes is content + metadata I can not
>>>> see what you want to "store" in the Entityhub that is about managing
>>>> Entities.
>>>>
>>>> > - entityhub: To query/update the entityhub component
>>>>>
>>>> Maybe. If you can come up with a good use case ^^
>>>>
>>>> > - contenthub: To develop a new content-hub using chain/engine
>>>>> components
>>>>> >and solr/elasticsearch/whatever component (solr and elasticsearch
>>>>> component
>>>>> >already exist in Camel)
>>>>>
>>>> IMO implementing a new Contenthub like component is outside the scope
>>>> of this GSoC project. However If there is already Solr/Elasticsearch
>>>> component it would be a really useful thing
>>>>
>>>>
>>> Regarding this, in my opinion, the use case of an eventual integration
>>> with a Content hub is probably one of the most clear for this project. I'm
>>> not sure if that is what Antonio was trying to explain but, with a single
>>> route using as last endpoint Solr or any other backend system, we would be
>>> almost cloning the same functionality than the previous ContentHub
>>> implementation (Stanbol 0.12). Entities could be dereferenced using the
>>> EntityHub before storing the content along with the metadata, which is the
>>> point of integration of the EntityHub in such use case. And even most
>>> interesting, now with the integration of Marmotta contributed by Rupert, it
>>> would be possible to use a whole graph for dereferencing, so "simply"
>>> routing components like Enhancer->Marmotta->Solr sounds to me like an
>>> interesting use case.
>>>
>>> wdyt?
>>>
>>> Cheers,
>>> Rafa
>>>
>>
>>
>
--
------------------------------
This message should be regarded as confidential. If you have received this
email in error please notify the sender and destroy it immediately.
Statements of intent shall only become binding when confirmed in hard copy
by an authorised signatory.
Zaizi Ltd is registered in England and Wales with the registration number
6440931. The Registered Office is Brook House, 229 Shepherds Bush Road,
London W6 7AN.
Re: Camel integration (was : Re: Community bonding period started)
Posted by Antonio David Perez Morales <ap...@zaizi.com>.
Hi all stanbolers
Continuing with the GSoC work, I have implemented a new component to index
content in Siren. This component extends the functionality of the
previously developed Stanbol Solr component (which in turn leverages the
functionality of Camel Solr component), to index the content along with its
enhancements in Siren, using the format expected by Siren. With such format
and unlike the Stanbol Solr component (which flattened the enhancement
properties, losing the relation between property value and entity which
belongs to) we can keep the structure of the content and all its
enhancements as children, improving the results of subsequent queries to
the index.
I also implemented the web part for workflow endpoint. It allows (like the
enhancer web part) set a text to be sent to a specific route and also
upload routes using a form. There are many things to be improved here, but
as first step I think it is good to have a easier way to upload and test
routes (we have also other mechanisms to add routes, like install bundles
generated by the route archetype or placing a route file in the stanbol
fileinstall directory). Take into account that unlike the enhancer, we can
have routes where the output is sent to another place (like Solr) or can be
triggered by some event (like a message sent to an activeMQ queue, etc), so
only routes using direct endpoints as first step in the route can be tested
through the web interface.
As for here, I will spend the remaining time of the GSoC improving things
and writting documentation. But if you guys think that some further
development is needed for some of your use cases (like a new component or
whatever), please let me know in order to re-schedule the rest of the time.
Regards
On Fri, Jul 25, 2014 at 12:23 PM, Antonio David Perez Morales <
aperez@zaizi.com> wrote:
> Hi people
>
> Regarding the last mail and continuing the work about the semantic search
> use case (where a stanbol solr component was already implemented in order
> to have some similar functionality like the old contenthub component), I
> have decided to give a try and implement a Siren [1] component for Stanbol
> workflow component. Siren is an extension of Solr that allows to store
> semi-structured components , fitting perfectly with the idea of store
> documents along with their related entities in order to allow subsequent
> semantic searches.
>
> The problem of the old content hub component (and also the problem of the
> new stanbol solr component) is that all the semantic information per
> document is stored in a plain form in the same Solr document (useful for
> some kind of searches) making impossible to relate the extracted attributes
> (properties) with their respective entities, losing the "parent-child"
> (document-entities) structure.
>
> I think it can be a great component for leveraging all the information
> extracted by Stanbol in searches.
>
> Please, feel free to comment or add whatever information you think useful
> for this.
>
> Regards
>
> [1] http://sirendb.com/
>
>
> On Mon, Jul 21, 2014 at 3:41 PM, Antonio David Perez Morales <
> aperez@zaizi.com> wrote:
>
>> Hi all
>>
>> As anticipated in the previous mail, I have develop a first version of
>> the Stanbol Solr component. This component (by default managing the
>> stanbol-solr camel protocol) extends the Camel Solr component, so all the
>> properties used to configure it ca be used in this component as well.
>>
>> The component is responsible of extracting fields and values from the
>> entities in the Content-Item and creates a Solr Document with the content
>> and metadata to be indexed in Solr. In this first version, no filtering is
>> being applied to the entities (for example, get the field-values only from
>> the entity with higher confidence value).
>>
>> The first version of the component allows three conf parameters in a
>> route:
>> - ldpath : LDPath program to be used to extract the values of the
>> fields. As mentioned in the previous mail, if a different ldpath in th
>> dereference engine is used then the properties to be extracted may not
>> exist.
>> - fields : A comma-separated list of values containing the fields to be
>> extracted from the entities and indexed in Solr.
>> - useDereferenceLdpath: If no ldpath program is defined, then this
>> boolean flag allows to use the same ldpath program used by the dereference
>> engine (getting it from the information contained in the content-item and
>> passed in the HTTP request to the enhancer or configured in the
>> chain/engine component). Default value is true.
>>
>> A sample route using this component could be the following:
>> <routes xmlns="http://camel.apache.org/schema/spring">
>> <route id="stanbolsolr">
>> <from uri="direct://stanbolsolr" />
>> <to
>> uri="chain://default?enhancer.engines.dereference.ldpath=%40prefix%20test%20%3A%20%3Chttp%3A%2F%2Ftest.org%2F%3E%3B%test%3Aname%3Drdfs%3Alabel%20%3A%3A%20xsd%3Astring%3B"
>> />
>> <to uri="stanbol-solr://localhost:8983/solr" />
>> </route>
>> </routes>
>>
>> As a future extensions of this component, a new property specifying a
>> configured dereference engine to use for the ldpath and filtering the
>> entities to get only the one with the higher confidence value will be
>> developed.
>> With this component, we can have a some features similar to the old
>> Stanbol content hub. So, i think improving this component we could achieve
>> to have to content-hub back to Stanbol (but using an external Solr
>> instance, which I think is good to not overloading the Stanbol application)
>>
>> Moreover, as part of the "use cases" project part and as discussed in the
>> Stanbol IRC Channel, I'm also evaluating Siren [1], an extension of Solr
>> bringing new and improved capabilities to it. It's very useful for
>> structured document search.
>> So my idea is to try to create a Siren component for Camel integrated in
>> Stanbol, to bring the possibility to store (in an easy way) the content
>> along with the extracted metadata in a structured way, instead of simply
>> creating new fields for a document.
>>
>> Stay tuned for new advances.
>> As always, comments are more than welcome.
>>
>> Regards
>>
>> [1] http://sirendb.com/
>>
>>
>> On Wed, Jul 16, 2014 at 12:53 PM, Antonio David Perez Morales <
>> aperez@zaizi.com> wrote:
>>
>>> Hi people
>>>
>>> Continuing with the project work , I have implemented some improvements
>>> to chain and engine components to allow defining enhancer properties (like
>>> enhancer.engines.dereference.ldpath) in the route component definition.
>>> Example :
>>> from(direct://test).to(engine://dereference-engine?enhancer.engines.dereference.ldpath=EXPRESSION).
>>> As said in previous mails, the engines and chains have to be configured
>>> through Felix console.
>>>
>>> Regarding the last discussion about a new kind of ContentHub back to
>>> Stanbol as an use case for the workflow integration, I have successfully
>>> created a custom Camel processor to create the document with the content
>>> and enhancement metadata in order to be sent to Solr. It takes the LDPath
>>> expression (configured in the dereference engine component via
>>> enhancer.engines.dereference.ldpath query parameter or camel component
>>> parameter) to extract the metadata to be indexed. So using a route like
>>> from().to(chain://Default).process(ContentItemProcessor).to(solr://localhost:8983/solr),
>>> we can have new indexed documents in Solr containing the text and the
>>> extracted enhancement metadata in order to be use in semantic searchs in
>>> the external Solr. Of course, the Solr schema needs to be created in the
>>> remote Solr beforehand. It is only a brief proof of concept of such
>>> functionality.
>>>
>>> My idea is to use an external Solr to store the content and semantic
>>> metadata for semantic search purposes, as opposite of the old ContentHub
>>> which was using an internal SolrYard, creating the schema from the
>>> configured LDPath expression.
>>>
>>> The next step in this task will be create a custom StanbolSolr
>>> component, able to perform the functionality of the previous processor and
>>> Solr, but allowing configuring the LDPath, fields and properties to be
>>> extracted and put as metadata in the new Solr document. These properties
>>> will be applied to the ContentItem metadata, so if an entity dereference
>>> engine is configured with a different LDPath expression or fields, maybe
>>> the properties to be extracted will not exist.
>>> As future improvement of this component, we could add a new conf
>>> parameter specifying a configured dereference engine to be used before
>>> applying the configuration.
>>>
>>> Stay tuned for further advances.
>>>
>>> As always if you have any questions or comments, please drop some lines
>>> here.
>>>
>>> Regards
>>>
>>> PS: The example routes used are very simple and lineals, but for some
>>> scenarios, parallel executions of engines, multicast, aggregator, etc
>>> (supported by camel) could be used to speed up the enhancement process.
>>>
>>>
>>>
>>>
>>>
>>>
>>> On Tue, Jul 8, 2014 at 9:46 AM, Antonio David Perez Morales <
>>> aperez@zaizi.com> wrote:
>>>
>>>> Hi Rafa and all
>>>>
>>>> In my opinion, the Content Hub back in Stanbol for Semantic Search
>>>> capabilities is a great use case to be implemented.
>>>> Waiting for Florent's opinion, I could start first only with Solr
>>>> (whose component already exists in Camel but it needs to be adapted like
>>>> the ActiveMQ component) and creating a custom transformer bean for Camel to
>>>> have the original Content Hub. After that, we could think to create the
>>>> SIREn component and the new transformer for it, giving the users the
>>>> possibility of use one of them.
>>>>
>>>> What do you think? Is It an interesting use case for the Camel
>>>> integration application?
>>>>
>>>> Regards
>>>>
>>>>
>>>> On Mon, Jul 7, 2014 at 4:27 PM, Rafa Haro <rh...@apache.org> wrote:
>>>>
>>>>> Hi guys,
>>>>>
>>>>> El 01/07/14 10:20, Antonio David Perez Morales escribió:
>>>>>
>>>>> Hi all
>>>>>>
>>>>>> Continuing with the project, I have managed successfully the
>>>>>> integration of
>>>>>> activemq camel component (and also jms) into the Stanbol Camel
>>>>>> integration.
>>>>>> This has been a hard task due to the dependencies needed by the
>>>>>> component
>>>>>> and also due to the fact that we had to provide an activemq component
>>>>>> configurable through Felix web console.
>>>>>>
>>>>>> With this addition, we are in the position to integrate business
>>>>>> logic into
>>>>>> Stanbol routes through a message service provided by activemq (jms).
>>>>>>
>>>>> Nice Antonio, let's see is someone has an interesting use case to
>>>>> implement in this context.
>>>>>
>>>>>
>>>>>> As a first test, I have deployed a route which consumes messages
>>>>>> (content)
>>>>>> from an activemq queue, enhance them using the default chain and then
>>>>>> write
>>>>>> the result into a file. It's a simple test but it works quite well.
>>>>>> In this
>>>>>> case, Stanbol is working in a standalone mode, that is to say, we
>>>>>> don't
>>>>>> have to explicitly call Stanbol to enhance content but Stanbol is
>>>>>> triggered
>>>>>> based on some external events (a new queue message)
>>>>>>
>>>>>> As indicated in the previous mail, I still have some pending things
>>>>>> to be
>>>>>> done (because I couldn't do them last week) but in order to go
>>>>>> forward with
>>>>>> the project I ask you for some interesting use cases where to apply
>>>>>> the new
>>>>>> workflow component in order to give added value to it and also in
>>>>>> order to
>>>>>> develop and provide more workflow (camel) components useful for those
>>>>>> and
>>>>>> other use cases.
>>>>>>
>>>>> Awaiting for the community feedback and also for Florent's opinion
>>>>> regarding the rest of the project, as I have expressed in recent emails,
>>>>> I'm eager to see the Content Hub back in Stanbol. And this is because of,
>>>>> from the point of view of the use of Stanbol in the enterprise, Semantic
>>>>> Search is one of the most common use cases. So, to have an enterprise
>>>>> search backend as the last component of a processing route in any
>>>>> architecture where stanbol could be plugged sounds key for me. In recent
>>>>> discussions at the Stanbol IRC channel, we have been analysing Siren (
>>>>> https://github.com/rdelbru/SIREn), a Lucene/Solr extension which
>>>>> major advantage is the possibility to index tree structures, allowing then
>>>>> to index structured data without losing full text search capabilities. To
>>>>> refactor old ContentHub component to use Siren is out of scope of this
>>>>> project but, in my opinion, an interesting use case could be to develop a
>>>>> Siren Camel Component and a transformer from ContentItem to Siren Object or
>>>>> whatever and integrate both in Stanbol.
>>>>>
>>>>> What do you guys think?
>>>>>
>>>>> Cheers,
>>>>> Rafa
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>> Regards
>>>>>>
>>>>>>
>>>>>> On Mon, Jun 23, 2014 at 6:16 PM, Antonio David Perez Morales <
>>>>>> aperez@zaizi.com> wrote:
>>>>>>
>>>>>> Hi Stanbolers
>>>>>>>
>>>>>>> The GSoC 2014 midterm is here and I want to give you a summary of
>>>>>>> the work
>>>>>>> already done so far:
>>>>>>>
>>>>>>> - Adapted previous Camel integration PoC done by Florent into
>>>>>>> Stanbol 1.0
>>>>>>> version.
>>>>>>> - Improved EngineComponent used by Camel to execute Enhancement
>>>>>>> Engines
>>>>>>> (configured through Stanbol web console as usual) using the
>>>>>>> engine:// uri
>>>>>>> scheme in routes.
>>>>>>> - Created ChainComponent used by Camel to execute Enhancement Chains
>>>>>>> using
>>>>>>> the chain:// uri scheme in routes (both Camel components are
>>>>>>> provided as
>>>>>>> OSGI components, so the uri scheme can be changed through the
>>>>>>> Stanbol web
>>>>>>> console)
>>>>>>> - Created a custom artifact for Apache Felix Fileinstall in order to
>>>>>>> be
>>>>>>> able to install routes defined in Camel Spring XML DSL placing a
>>>>>>> route file
>>>>>>> (with 'route' extension) in the stanbol/fileinstall directory
>>>>>>> - Created a custom archetype to ease the development of bundles
>>>>>>> containing
>>>>>>> route definitions in Java DSL. The archetype generates a class
>>>>>>> extending
>>>>>>> 'RouteBuilder' which creates a default Camel direct endpoint used by
>>>>>>> other
>>>>>>> Stanbol Workflow components to execute the route.
>>>>>>> - Created a first version of Workflow API, which contains different
>>>>>>> OSGI
>>>>>>> components which allow registering Camel components/routes,
>>>>>>> start/stop/execute routes, add/remove components used in routes, etc.
>>>>>>> - REST endpoint is provided to test the execution of routes using
>>>>>>> REST
>>>>>>> requests (/flow/{routeId} )
>>>>>>> - Modified the PoC full launcher to use all the new bundles to
>>>>>>> support the
>>>>>>> workflow feature.
>>>>>>> - Installed JBoss developer studio which comes with Camel support in
>>>>>>> order
>>>>>>> to create routes in a visual way with the possibility to be exported
>>>>>>> as
>>>>>>> Spring XML DSL format
>>>>>>>
>>>>>>> Some pending things I will try to do during this week:
>>>>>>> - Improve the web package to create the needed endpoints to query the
>>>>>>> registered routes, registered camel components, etc
>>>>>>> - Improve the web package to remove classes copied from Stanbol
>>>>>>> jersey
>>>>>>> module used for testing
>>>>>>> - Update README.md files in the repository with all the new
>>>>>>> information
>>>>>>> - Document the installation and configuration of JBoss developer
>>>>>>> studio
>>>>>>> for Camel routes creation
>>>>>>> - Create all the JIRA issued related to the work already done
>>>>>>>
>>>>>>>
>>>>>>> For the second part of the project, I would like to read some
>>>>>>> comments
>>>>>>> about interesting use cases in order to develop the needed Stanbol
>>>>>>> and
>>>>>>> Camel components to support them.
>>>>>>>
>>>>>>> If you have any comment, please drop some lines in order to discuss
>>>>>>> the
>>>>>>> new things to be done.
>>>>>>>
>>>>>>> Regards
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Sat, Jun 14, 2014 at 3:39 PM, Antonio David Perez Morales <
>>>>>>> aperez@zaizi.com> wrote:
>>>>>>>
>>>>>>> Hi guys
>>>>>>>>
>>>>>>>> Continuing with the project, and as part of the refactoring/new
>>>>>>>> architecture I have started to modify some workflow components in
>>>>>>>> order to
>>>>>>>> create a better API and architecture based on OSGI components. As a
>>>>>>>> first
>>>>>>>> step and in order to have the same behavior than the current one
>>>>>>>> (regarding
>>>>>>>> enhancement process), a chain component has been created to
>>>>>>>> simulate the
>>>>>>>> chain behaviour. This new component uses internally the
>>>>>>>> ChainManager and
>>>>>>>> EnhancementJobManager component to perform the business logic. This
>>>>>>>> way, a
>>>>>>>> new protocol 'chain' can be used in the routes deployed in Stanbol.
>>>>>>>> The
>>>>>>>> chains are configured in the same way, using Stanbol admin console.
>>>>>>>>
>>>>>>>> Now, we can combine single engine executions with chains executions
>>>>>>>> in
>>>>>>>> routes deployed in Stanbol using the alternatives described in
>>>>>>>> previous
>>>>>>>> mails and in the issue [1]. Both engines and chains are configured
>>>>>>>> through
>>>>>>>> Stanbol admin console. You can see the refactoring advances in [2]
>>>>>>>> (a
>>>>>>>> branch used for refactoring the current PoC of Workflow in Stanbol
>>>>>>>> 1.0). Of
>>>>>>>> course, the Camel EIP and other Camel components can be used in the
>>>>>>>> deployed routes as well.
>>>>>>>>
>>>>>>>> With the new Camel routes support, we can have a Stanbol running and
>>>>>>>> enhancing content without receiving any HTTP request to start the
>>>>>>>> enhancement process, because the routes can be triggered by
>>>>>>>> external events
>>>>>>>> ocurred in a queue, database, etc. Moreover the semantic lifting
>>>>>>>> process
>>>>>>>> can be splitted and merged with some application steps, so the
>>>>>>>> issue [3]
>>>>>>>> requesting asynchronous call support for enhancement could be
>>>>>>>> solved.
>>>>>>>>
>>>>>>>> Anyway, if some of you have any suggestions for new components to be
>>>>>>>> deployed for the second part of the project, or another kind of
>>>>>>>> suggestion,
>>>>>>>> please drop here some lines to continue with the discussion.
>>>>>>>>
>>>>>>>> Regards
>>>>>>>>
>>>>>>>> [1] https://issues.apache.org/jira/browse/STANBOL-1348
>>>>>>>> [2]
>>>>>>>> https://github.com/adperezmorales/stanbol-camel-
>>>>>>>> workflow/tree/refactoring
>>>>>>>> [3] https://issues.apache.org/jira/browse/STANBOL-263
>>>>>>>>
>>>>>>>>
>>>>>>>> On Wed, Jun 11, 2014 at 10:01 AM, Antonio David Perez Morales <
>>>>>>>> aperez@zaizi.com> wrote:
>>>>>>>>
>>>>>>>> Hi people
>>>>>>>>>
>>>>>>>>> As part of the GSoC project for the midterm and according to the
>>>>>>>>> issue
>>>>>>>>> [1], a custom Apache Felix Fileinstall artifact has been created
>>>>>>>>> in order
>>>>>>>>> to deploy Camel routes defined in XML (Spring DSL) placing a file
>>>>>>>>> with
>>>>>>>>> .route extension in a configured directory (like
>>>>>>>>> stanbol/fileinstall
>>>>>>>>> directory). Moreover since this artifact depends on Fileinstall
>>>>>>>>> bundle, the
>>>>>>>>> created launcher has been modified to have that bundle in the OSGI
>>>>>>>>> context
>>>>>>>>> by default.
>>>>>>>>>
>>>>>>>>> So, once the current Camel integration POC has been integrated in
>>>>>>>>> Stanbol 1.0 and extended to support the deployment of routes
>>>>>>>>> defined by
>>>>>>>>> Java DSL (through bundles) and XML (route files), the next step
>>>>>>>>> will be
>>>>>>>>> thinking and redesigning the current architecture trying to avoid
>>>>>>>>> the
>>>>>>>>> duplicated code and providing a more extendable and easy to use
>>>>>>>>> Workflow
>>>>>>>>> API, because with the current integration only direct routes can be
>>>>>>>>> triggered using REST API which means that the defined routes must
>>>>>>>>> be
>>>>>>>>> configured properly using a direct endpoint consumer. Anyway,
>>>>>>>>> routes
>>>>>>>>> starting in some other way like timers are triggered directly in
>>>>>>>>> the
>>>>>>>>> deployment, so this has to be taken into account for the new API
>>>>>>>>> (and REST
>>>>>>>>> API).
>>>>>>>>>
>>>>>>>>> In parallel and for the second part, new Stanbol Camel components
>>>>>>>>> will
>>>>>>>>> be developed in order to be used in new routes. So if any of you
>>>>>>>>> have use
>>>>>>>>> cases for this involving Stanbol components, please drop some
>>>>>>>>> lines here in
>>>>>>>>> order to prioritize the Stanbol Camel components to be developed.
>>>>>>>>>
>>>>>>>>> Comments and suggestions are more than welcome
>>>>>>>>>
>>>>>>>>> Regards
>>>>>>>>>
>>>>>>>>> [1] https://issues.apache.org/jira/browse/STANBOL-1348
>>>>>>>>> [2] https://github.com/adperezmorales/stanbol-camel-workflow/
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Mon, Jun 2, 2014 at 7:00 PM, Antonio David Perez Morales <
>>>>>>>>> aperez@zaizi.com> wrote:
>>>>>>>>>
>>>>>>>>> Hi stanbolers
>>>>>>>>>>
>>>>>>>>>> As part of the issue [1] , I have created a maven archetype
>>>>>>>>>> useful to
>>>>>>>>>> generate Camel routes in Java DSL.
>>>>>>>>>> The archetype generates a Java project with all the dependencies
>>>>>>>>>> and
>>>>>>>>>> one Java class with a method which has to be filled. In this
>>>>>>>>>> method, Camel
>>>>>>>>>> Java DSL syntax is used to create the route.
>>>>>>>>>> By default and as a first approach, the class will use the route
>>>>>>>>>> name
>>>>>>>>>> given during the project creation to enable a Camel direct
>>>>>>>>>> endpoint with
>>>>>>>>>> such name.
>>>>>>>>>> The code of the first archetype version can be found at [2].
>>>>>>>>>>
>>>>>>>>>> The next task will be providing a Felix custom artifact to be
>>>>>>>>>> able to
>>>>>>>>>> deploy XML-based routes in Stanbol, placing a custom file in the
>>>>>>>>>> Stanbol
>>>>>>>>>> datafiles directory.
>>>>>>>>>> After that, it will be time to think and redesign the
>>>>>>>>>> architecture to
>>>>>>>>>> integrate Camel workflows inside Stanbol in a better way, more
>>>>>>>>>> configurable
>>>>>>>>>> and extendable.
>>>>>>>>>>
>>>>>>>>>> Comments and suggestions are more than welcome
>>>>>>>>>>
>>>>>>>>>> Regards
>>>>>>>>>>
>>>>>>>>>> [1] https://issues.apache.org/jira/browse/STANBOL-1348
>>>>>>>>>> [2] https://github.com/adperezmorales/stanbol-camel-workflow/
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Fri, May 30, 2014 at 8:03 PM, Antonio David Perez Morales <
>>>>>>>>>> aperez@zaizi.com> wrote:
>>>>>>>>>>
>>>>>>>>>> Hi all
>>>>>>>>>>>
>>>>>>>>>>> After a hard fight this week, I managed to get it work the
>>>>>>>>>>> Florent's
>>>>>>>>>>> proof of concept code in the Stanbol 1.0 branch [1]
>>>>>>>>>>> The code is uploaded in my github account [3]. As I said in a
>>>>>>>>>>> previous
>>>>>>>>>>> mail, I prefer to do it separately and after the project,
>>>>>>>>>>> uploading the
>>>>>>>>>>> developed code into a Stanbol branch.
>>>>>>>>>>>
>>>>>>>>>>> The 1.0.0 version has some changes in how the Jersey endpoints
>>>>>>>>>>> are
>>>>>>>>>>> registered and also new classes and packages, so it was not a
>>>>>>>>>>> trivial task
>>>>>>>>>>> to make work the current proof of concept. Moreover I don't like
>>>>>>>>>>> to simply
>>>>>>>>>>> copy and paste code and make the needed changes. I always want to
>>>>>>>>>>> understand how the things work and how they are developed in
>>>>>>>>>>> order to be
>>>>>>>>>>> able to change/modify them or develop new code around them.
>>>>>>>>>>>
>>>>>>>>>>> The steps done to achieve it have been the following:
>>>>>>>>>>> - Updated pom files to the Stanbol 1.0.0-SNAPSHOT version
>>>>>>>>>>> - Updated bundle levels in bundlelist package to fit the Stanbol
>>>>>>>>>>> 1.0
>>>>>>>>>>> version levels
>>>>>>>>>>> - Adapted cameljobmanager package code to Stanbol 1.0.0-SNAPSHOT
>>>>>>>>>>> classes and using Java OSGI annotations instead of SCR
>>>>>>>>>>> annotations in
>>>>>>>>>>> Javadoc
>>>>>>>>>>> - Updated flow web package to Stanbol 1.0.0-SNAPSHOT classes and
>>>>>>>>>>> modified needed resources
>>>>>>>>>>> - Added Java OSGI annotations to the route (WeightedChain)
>>>>>>>>>>> instead of
>>>>>>>>>>> SCR annotations in javadoc
>>>>>>>>>>> - Updated launcher to use the 1.0.0-SNAPSHOT packages and needed
>>>>>>>>>>> bundles
>>>>>>>>>>>
>>>>>>>>>>> So now, the http://localhost:8080/flow endpoint will use the
>>>>>>>>>>> only
>>>>>>>>>>> Camel route (defined by WeightedChain) to call all the registered
>>>>>>>>>>> Enhancement Engines (ordered by EnhancementEngine order
>>>>>>>>>>> property).
>>>>>>>>>>> For testing purposes, the /flow/{flowName} has been removed,
>>>>>>>>>>> because
>>>>>>>>>>> all this code needs to be re-designed and re-implemented so I
>>>>>>>>>>> only wanted
>>>>>>>>>>> to make it work to have a first (simple) integration in Stanbol
>>>>>>>>>>> 1.0. This
>>>>>>>>>>> functionality will be added again to trigger custom routes once
>>>>>>>>>>> the next
>>>>>>>>>>> step (defined below) is developed.
>>>>>>>>>>>
>>>>>>>>>>> The next step [2] will be support to write and configure routes
>>>>>>>>>>> in XML
>>>>>>>>>>> format, putting the file in datafiles in order to be loaded by a
>>>>>>>>>>> Felix
>>>>>>>>>>> custom artifact (as Rupert pointed out in a previous mail) and
>>>>>>>>>>> create a
>>>>>>>>>>> Maven archetype to create bundles defining routes which will be
>>>>>>>>>>> loaded
>>>>>>>>>>> using the Felix bundle tab. If necessary, as we talked in
>>>>>>>>>>> previous
>>>>>>>>>>> messages, a REST endpoint receiving routes in XML can be
>>>>>>>>>>> developed as an
>>>>>>>>>>> alternative to the first approach. This is my objective for the
>>>>>>>>>>> midterm.
>>>>>>>>>>>
>>>>>>>>>>> After the midterm, the new Stanbol components for Apache Camel
>>>>>>>>>>> will be
>>>>>>>>>>> developed and also the new architecture for Camel in Stanbol.
>>>>>>>>>>>
>>>>>>>>>>> Comments on this and for use cases for Stanbol Camel components
>>>>>>>>>>> are
>>>>>>>>>>> more than welcome.
>>>>>>>>>>>
>>>>>>>>>>> Regards
>>>>>>>>>>>
>>>>>>>>>>> [1] https://issues.apache.org/jira/browse/STANBOL-1347
>>>>>>>>>>> [2] https://issues.apache.org/jira/browse/STANBOL-1348
>>>>>>>>>>> [3] https://github.com/adperezmorales/stanbol-camel-workflow/
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Tue, May 27, 2014 at 6:18 PM, Antonio David Perez Morales <
>>>>>>>>>>> aperez@zaizi.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>> Hi people
>>>>>>>>>>>>
>>>>>>>>>>>> I have already started to work on [1] to integrate current
>>>>>>>>>>>> Florent's
>>>>>>>>>>>> code into Stanbol 1.0.
>>>>>>>>>>>> As a first approach, only changing the dependency versions to
>>>>>>>>>>>> new
>>>>>>>>>>>> Stanbol 1.0, many issues have arisen:
>>>>>>>>>>>> - Deprecated use of classes
>>>>>>>>>>>> - Classes which have changed from package
>>>>>>>>>>>> - Some classes not necessary now
>>>>>>>>>>>> - Classes not used which were causing conflicts
>>>>>>>>>>>> - ...
>>>>>>>>>>>>
>>>>>>>>>>>> So now I'm trying to resolve all these problems to replicate
>>>>>>>>>>>> the same
>>>>>>>>>>>> behavior from 0.9 into 1.0. I will upload the code to a Github
>>>>>>>>>>>> repository
>>>>>>>>>>>> in my account (which will be pushed later into a Stanbol branch
>>>>>>>>>>>> after the
>>>>>>>>>>>> project) in order to track the advances.
>>>>>>>>>>>> Once I can resolve all these problems, I will take a look to the
>>>>>>>>>>>> Felix Custom Artifacts poiinted out by Rupert in a previous
>>>>>>>>>>>> message to find
>>>>>>>>>>>> out the best way to deploy (and manage) route configurations
>>>>>>>>>>>> (felix
>>>>>>>>>>>> artifacts, watchservice java, rest endpoint to receive xml
>>>>>>>>>>>> routes, etc).
>>>>>>>>>>>>
>>>>>>>>>>>> Comments on this and future tasks are more than welcome.
>>>>>>>>>>>>
>>>>>>>>>>>> Regards
>>>>>>>>>>>>
>>>>>>>>>>>> [1] https://issues.apache.org/jira/browse/STANBOL-1347
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Tue, May 27, 2014 at 9:53 AM, Rafa Haro <rh...@apache.org>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> Hi Rupert, Florent and Antonio
>>>>>>>>>>>>>
>>>>>>>>>>>>> El 27/05/14 08:51, Rupert Westenthaler escribió:
>>>>>>>>>>>>>
>>>>>>>>>>>>> As the result of Enhancement Routes is content + metadata I
>>>>>>>>>>>>> can not
>>>>>>>>>>>>>
>>>>>>>>>>>>>> see what you want to "store" in the Entityhub that is about
>>>>>>>>>>>>>> managing
>>>>>>>>>>>>>> Entities.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> > - entityhub: To query/update the entityhub component
>>>>>>>>>>>>>> Maybe. If you can come up with a good use case ^^
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> > - contenthub: To develop a new content-hub using
>>>>>>>>>>>>>> chain/engine
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> components
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> and solr/elasticsearch/whatever component (solr and
>>>>>>>>>>>>>>>> elasticsearch
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> component
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> already exist in Camel)
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> IMO implementing a new Contenthub like component is outside
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>> scope
>>>>>>>>>>>>>> of this GSoC project. However If there is already
>>>>>>>>>>>>>> Solr/Elasticsearch
>>>>>>>>>>>>>> component it would be a really useful thing
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Regarding this, in my opinion, the use case of an eventual
>>>>>>>>>>>>> integration with a Content hub is probably one of the most
>>>>>>>>>>>>> clear for this
>>>>>>>>>>>>> project. I'm not sure if that is what Antonio was trying to
>>>>>>>>>>>>> explain but,
>>>>>>>>>>>>> with a single route using as last endpoint Solr or any other
>>>>>>>>>>>>> backend
>>>>>>>>>>>>> system, we would be almost cloning the same functionality than
>>>>>>>>>>>>> the previous
>>>>>>>>>>>>> ContentHub implementation (Stanbol 0.12). Entities could be
>>>>>>>>>>>>> dereferenced
>>>>>>>>>>>>> using the EntityHub before storing the content along with the
>>>>>>>>>>>>> metadata,
>>>>>>>>>>>>> which is the point of integration of the EntityHub in such use
>>>>>>>>>>>>> case. And
>>>>>>>>>>>>> even most interesting, now with the integration of Marmotta
>>>>>>>>>>>>> contributed by
>>>>>>>>>>>>> Rupert, it would be possible to use a whole graph for
>>>>>>>>>>>>> dereferencing, so
>>>>>>>>>>>>> "simply" routing components like Enhancer->Marmotta->Solr
>>>>>>>>>>>>> sounds to me like
>>>>>>>>>>>>> an interesting use case.
>>>>>>>>>>>>>
>>>>>>>>>>>>> wdyt?
>>>>>>>>>>>>>
>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>> Rafa
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>
>>>>
>>>
>>
>
--
------------------------------
This message should be regarded as confidential. If you have received this
email in error please notify the sender and destroy it immediately.
Statements of intent shall only become binding when confirmed in hard copy
by an authorised signatory.
Zaizi Ltd is registered in England and Wales with the registration number
6440931. The Registered Office is Brook House, 229 Shepherds Bush Road,
London W6 7AN.
Re: Camel integration (was : Re: Community bonding period started)
Posted by Antonio David Perez Morales <ap...@zaizi.com>.
Hi people
Regarding the last mail and continuing the work about the semantic search
use case (where a stanbol solr component was already implemented in order
to have some similar functionality like the old contenthub component), I
have decided to give a try and implement a Siren [1] component for Stanbol
workflow component. Siren is an extension of Solr that allows to store
semi-structured components , fitting perfectly with the idea of store
documents along with their related entities in order to allow subsequent
semantic searches.
The problem of the old content hub component (and also the problem of the
new stanbol solr component) is that all the semantic information per
document is stored in a plain form in the same Solr document (useful for
some kind of searches) making impossible to relate the extracted attributes
(properties) with their respective entities, losing the "parent-child"
(document-entities) structure.
I think it can be a great component for leveraging all the information
extracted by Stanbol in searches.
Please, feel free to comment or add whatever information you think useful
for this.
Regards
[1] http://sirendb.com/
On Mon, Jul 21, 2014 at 3:41 PM, Antonio David Perez Morales <
aperez@zaizi.com> wrote:
> Hi all
>
> As anticipated in the previous mail, I have develop a first version of the
> Stanbol Solr component. This component (by default managing the
> stanbol-solr camel protocol) extends the Camel Solr component, so all the
> properties used to configure it ca be used in this component as well.
>
> The component is responsible of extracting fields and values from the
> entities in the Content-Item and creates a Solr Document with the content
> and metadata to be indexed in Solr. In this first version, no filtering is
> being applied to the entities (for example, get the field-values only from
> the entity with higher confidence value).
>
> The first version of the component allows three conf parameters in a route:
> - ldpath : LDPath program to be used to extract the values of the fields.
> As mentioned in the previous mail, if a different ldpath in th dereference
> engine is used then the properties to be extracted may not exist.
> - fields : A comma-separated list of values containing the fields to be
> extracted from the entities and indexed in Solr.
> - useDereferenceLdpath: If no ldpath program is defined, then this
> boolean flag allows to use the same ldpath program used by the dereference
> engine (getting it from the information contained in the content-item and
> passed in the HTTP request to the enhancer or configured in the
> chain/engine component). Default value is true.
>
> A sample route using this component could be the following:
> <routes xmlns="http://camel.apache.org/schema/spring">
> <route id="stanbolsolr">
> <from uri="direct://stanbolsolr" />
> <to
> uri="chain://default?enhancer.engines.dereference.ldpath=%40prefix%20test%20%3A%20%3Chttp%3A%2F%2Ftest.org%2F%3E%3B%test%3Aname%3Drdfs%3Alabel%20%3A%3A%20xsd%3Astring%3B"
> />
> <to uri="stanbol-solr://localhost:8983/solr" />
> </route>
> </routes>
>
> As a future extensions of this component, a new property specifying a
> configured dereference engine to use for the ldpath and filtering the
> entities to get only the one with the higher confidence value will be
> developed.
> With this component, we can have a some features similar to the old
> Stanbol content hub. So, i think improving this component we could achieve
> to have to content-hub back to Stanbol (but using an external Solr
> instance, which I think is good to not overloading the Stanbol application)
>
> Moreover, as part of the "use cases" project part and as discussed in the
> Stanbol IRC Channel, I'm also evaluating Siren [1], an extension of Solr
> bringing new and improved capabilities to it. It's very useful for
> structured document search.
> So my idea is to try to create a Siren component for Camel integrated in
> Stanbol, to bring the possibility to store (in an easy way) the content
> along with the extracted metadata in a structured way, instead of simply
> creating new fields for a document.
>
> Stay tuned for new advances.
> As always, comments are more than welcome.
>
> Regards
>
> [1] http://sirendb.com/
>
>
> On Wed, Jul 16, 2014 at 12:53 PM, Antonio David Perez Morales <
> aperez@zaizi.com> wrote:
>
>> Hi people
>>
>> Continuing with the project work , I have implemented some improvements
>> to chain and engine components to allow defining enhancer properties (like
>> enhancer.engines.dereference.ldpath) in the route component definition.
>> Example :
>> from(direct://test).to(engine://dereference-engine?enhancer.engines.dereference.ldpath=EXPRESSION).
>> As said in previous mails, the engines and chains have to be configured
>> through Felix console.
>>
>> Regarding the last discussion about a new kind of ContentHub back to
>> Stanbol as an use case for the workflow integration, I have successfully
>> created a custom Camel processor to create the document with the content
>> and enhancement metadata in order to be sent to Solr. It takes the LDPath
>> expression (configured in the dereference engine component via
>> enhancer.engines.dereference.ldpath query parameter or camel component
>> parameter) to extract the metadata to be indexed. So using a route like
>> from().to(chain://Default).process(ContentItemProcessor).to(solr://localhost:8983/solr),
>> we can have new indexed documents in Solr containing the text and the
>> extracted enhancement metadata in order to be use in semantic searchs in
>> the external Solr. Of course, the Solr schema needs to be created in the
>> remote Solr beforehand. It is only a brief proof of concept of such
>> functionality.
>>
>> My idea is to use an external Solr to store the content and semantic
>> metadata for semantic search purposes, as opposite of the old ContentHub
>> which was using an internal SolrYard, creating the schema from the
>> configured LDPath expression.
>>
>> The next step in this task will be create a custom StanbolSolr component,
>> able to perform the functionality of the previous processor and Solr, but
>> allowing configuring the LDPath, fields and properties to be extracted and
>> put as metadata in the new Solr document. These properties will be applied
>> to the ContentItem metadata, so if an entity dereference engine is
>> configured with a different LDPath expression or fields, maybe the
>> properties to be extracted will not exist.
>> As future improvement of this component, we could add a new conf
>> parameter specifying a configured dereference engine to be used before
>> applying the configuration.
>>
>> Stay tuned for further advances.
>>
>> As always if you have any questions or comments, please drop some lines
>> here.
>>
>> Regards
>>
>> PS: The example routes used are very simple and lineals, but for some
>> scenarios, parallel executions of engines, multicast, aggregator, etc
>> (supported by camel) could be used to speed up the enhancement process.
>>
>>
>>
>>
>>
>>
>> On Tue, Jul 8, 2014 at 9:46 AM, Antonio David Perez Morales <
>> aperez@zaizi.com> wrote:
>>
>>> Hi Rafa and all
>>>
>>> In my opinion, the Content Hub back in Stanbol for Semantic Search
>>> capabilities is a great use case to be implemented.
>>> Waiting for Florent's opinion, I could start first only with Solr (whose
>>> component already exists in Camel but it needs to be adapted like the
>>> ActiveMQ component) and creating a custom transformer bean for Camel to
>>> have the original Content Hub. After that, we could think to create the
>>> SIREn component and the new transformer for it, giving the users the
>>> possibility of use one of them.
>>>
>>> What do you think? Is It an interesting use case for the Camel
>>> integration application?
>>>
>>> Regards
>>>
>>>
>>> On Mon, Jul 7, 2014 at 4:27 PM, Rafa Haro <rh...@apache.org> wrote:
>>>
>>>> Hi guys,
>>>>
>>>> El 01/07/14 10:20, Antonio David Perez Morales escribió:
>>>>
>>>> Hi all
>>>>>
>>>>> Continuing with the project, I have managed successfully the
>>>>> integration of
>>>>> activemq camel component (and also jms) into the Stanbol Camel
>>>>> integration.
>>>>> This has been a hard task due to the dependencies needed by the
>>>>> component
>>>>> and also due to the fact that we had to provide an activemq component
>>>>> configurable through Felix web console.
>>>>>
>>>>> With this addition, we are in the position to integrate business logic
>>>>> into
>>>>> Stanbol routes through a message service provided by activemq (jms).
>>>>>
>>>> Nice Antonio, let's see is someone has an interesting use case to
>>>> implement in this context.
>>>>
>>>>
>>>>> As a first test, I have deployed a route which consumes messages
>>>>> (content)
>>>>> from an activemq queue, enhance them using the default chain and then
>>>>> write
>>>>> the result into a file. It's a simple test but it works quite well. In
>>>>> this
>>>>> case, Stanbol is working in a standalone mode, that is to say, we don't
>>>>> have to explicitly call Stanbol to enhance content but Stanbol is
>>>>> triggered
>>>>> based on some external events (a new queue message)
>>>>>
>>>>> As indicated in the previous mail, I still have some pending things to
>>>>> be
>>>>> done (because I couldn't do them last week) but in order to go forward
>>>>> with
>>>>> the project I ask you for some interesting use cases where to apply
>>>>> the new
>>>>> workflow component in order to give added value to it and also in
>>>>> order to
>>>>> develop and provide more workflow (camel) components useful for those
>>>>> and
>>>>> other use cases.
>>>>>
>>>> Awaiting for the community feedback and also for Florent's opinion
>>>> regarding the rest of the project, as I have expressed in recent emails,
>>>> I'm eager to see the Content Hub back in Stanbol. And this is because of,
>>>> from the point of view of the use of Stanbol in the enterprise, Semantic
>>>> Search is one of the most common use cases. So, to have an enterprise
>>>> search backend as the last component of a processing route in any
>>>> architecture where stanbol could be plugged sounds key for me. In recent
>>>> discussions at the Stanbol IRC channel, we have been analysing Siren (
>>>> https://github.com/rdelbru/SIREn), a Lucene/Solr extension which major
>>>> advantage is the possibility to index tree structures, allowing then to
>>>> index structured data without losing full text search capabilities. To
>>>> refactor old ContentHub component to use Siren is out of scope of this
>>>> project but, in my opinion, an interesting use case could be to develop a
>>>> Siren Camel Component and a transformer from ContentItem to Siren Object or
>>>> whatever and integrate both in Stanbol.
>>>>
>>>> What do you guys think?
>>>>
>>>> Cheers,
>>>> Rafa
>>>>
>>>>
>>>>
>>>>
>>>>> Regards
>>>>>
>>>>>
>>>>> On Mon, Jun 23, 2014 at 6:16 PM, Antonio David Perez Morales <
>>>>> aperez@zaizi.com> wrote:
>>>>>
>>>>> Hi Stanbolers
>>>>>>
>>>>>> The GSoC 2014 midterm is here and I want to give you a summary of the
>>>>>> work
>>>>>> already done so far:
>>>>>>
>>>>>> - Adapted previous Camel integration PoC done by Florent into Stanbol
>>>>>> 1.0
>>>>>> version.
>>>>>> - Improved EngineComponent used by Camel to execute Enhancement
>>>>>> Engines
>>>>>> (configured through Stanbol web console as usual) using the engine://
>>>>>> uri
>>>>>> scheme in routes.
>>>>>> - Created ChainComponent used by Camel to execute Enhancement Chains
>>>>>> using
>>>>>> the chain:// uri scheme in routes (both Camel components are provided
>>>>>> as
>>>>>> OSGI components, so the uri scheme can be changed through the Stanbol
>>>>>> web
>>>>>> console)
>>>>>> - Created a custom artifact for Apache Felix Fileinstall in order to
>>>>>> be
>>>>>> able to install routes defined in Camel Spring XML DSL placing a
>>>>>> route file
>>>>>> (with 'route' extension) in the stanbol/fileinstall directory
>>>>>> - Created a custom archetype to ease the development of bundles
>>>>>> containing
>>>>>> route definitions in Java DSL. The archetype generates a class
>>>>>> extending
>>>>>> 'RouteBuilder' which creates a default Camel direct endpoint used by
>>>>>> other
>>>>>> Stanbol Workflow components to execute the route.
>>>>>> - Created a first version of Workflow API, which contains different
>>>>>> OSGI
>>>>>> components which allow registering Camel components/routes,
>>>>>> start/stop/execute routes, add/remove components used in routes, etc.
>>>>>> - REST endpoint is provided to test the execution of routes using REST
>>>>>> requests (/flow/{routeId} )
>>>>>> - Modified the PoC full launcher to use all the new bundles to
>>>>>> support the
>>>>>> workflow feature.
>>>>>> - Installed JBoss developer studio which comes with Camel support in
>>>>>> order
>>>>>> to create routes in a visual way with the possibility to be exported
>>>>>> as
>>>>>> Spring XML DSL format
>>>>>>
>>>>>> Some pending things I will try to do during this week:
>>>>>> - Improve the web package to create the needed endpoints to query the
>>>>>> registered routes, registered camel components, etc
>>>>>> - Improve the web package to remove classes copied from Stanbol jersey
>>>>>> module used for testing
>>>>>> - Update README.md files in the repository with all the new
>>>>>> information
>>>>>> - Document the installation and configuration of JBoss developer
>>>>>> studio
>>>>>> for Camel routes creation
>>>>>> - Create all the JIRA issued related to the work already done
>>>>>>
>>>>>>
>>>>>> For the second part of the project, I would like to read some comments
>>>>>> about interesting use cases in order to develop the needed Stanbol and
>>>>>> Camel components to support them.
>>>>>>
>>>>>> If you have any comment, please drop some lines in order to discuss
>>>>>> the
>>>>>> new things to be done.
>>>>>>
>>>>>> Regards
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Sat, Jun 14, 2014 at 3:39 PM, Antonio David Perez Morales <
>>>>>> aperez@zaizi.com> wrote:
>>>>>>
>>>>>> Hi guys
>>>>>>>
>>>>>>> Continuing with the project, and as part of the refactoring/new
>>>>>>> architecture I have started to modify some workflow components in
>>>>>>> order to
>>>>>>> create a better API and architecture based on OSGI components. As a
>>>>>>> first
>>>>>>> step and in order to have the same behavior than the current one
>>>>>>> (regarding
>>>>>>> enhancement process), a chain component has been created to simulate
>>>>>>> the
>>>>>>> chain behaviour. This new component uses internally the ChainManager
>>>>>>> and
>>>>>>> EnhancementJobManager component to perform the business logic. This
>>>>>>> way, a
>>>>>>> new protocol 'chain' can be used in the routes deployed in Stanbol.
>>>>>>> The
>>>>>>> chains are configured in the same way, using Stanbol admin console.
>>>>>>>
>>>>>>> Now, we can combine single engine executions with chains executions
>>>>>>> in
>>>>>>> routes deployed in Stanbol using the alternatives described in
>>>>>>> previous
>>>>>>> mails and in the issue [1]. Both engines and chains are configured
>>>>>>> through
>>>>>>> Stanbol admin console. You can see the refactoring advances in [2] (a
>>>>>>> branch used for refactoring the current PoC of Workflow in Stanbol
>>>>>>> 1.0). Of
>>>>>>> course, the Camel EIP and other Camel components can be used in the
>>>>>>> deployed routes as well.
>>>>>>>
>>>>>>> With the new Camel routes support, we can have a Stanbol running and
>>>>>>> enhancing content without receiving any HTTP request to start the
>>>>>>> enhancement process, because the routes can be triggered by external
>>>>>>> events
>>>>>>> ocurred in a queue, database, etc. Moreover the semantic lifting
>>>>>>> process
>>>>>>> can be splitted and merged with some application steps, so the issue
>>>>>>> [3]
>>>>>>> requesting asynchronous call support for enhancement could be solved.
>>>>>>>
>>>>>>> Anyway, if some of you have any suggestions for new components to be
>>>>>>> deployed for the second part of the project, or another kind of
>>>>>>> suggestion,
>>>>>>> please drop here some lines to continue with the discussion.
>>>>>>>
>>>>>>> Regards
>>>>>>>
>>>>>>> [1] https://issues.apache.org/jira/browse/STANBOL-1348
>>>>>>> [2]
>>>>>>> https://github.com/adperezmorales/stanbol-camel-
>>>>>>> workflow/tree/refactoring
>>>>>>> [3] https://issues.apache.org/jira/browse/STANBOL-263
>>>>>>>
>>>>>>>
>>>>>>> On Wed, Jun 11, 2014 at 10:01 AM, Antonio David Perez Morales <
>>>>>>> aperez@zaizi.com> wrote:
>>>>>>>
>>>>>>> Hi people
>>>>>>>>
>>>>>>>> As part of the GSoC project for the midterm and according to the
>>>>>>>> issue
>>>>>>>> [1], a custom Apache Felix Fileinstall artifact has been created in
>>>>>>>> order
>>>>>>>> to deploy Camel routes defined in XML (Spring DSL) placing a file
>>>>>>>> with
>>>>>>>> .route extension in a configured directory (like stanbol/fileinstall
>>>>>>>> directory). Moreover since this artifact depends on Fileinstall
>>>>>>>> bundle, the
>>>>>>>> created launcher has been modified to have that bundle in the OSGI
>>>>>>>> context
>>>>>>>> by default.
>>>>>>>>
>>>>>>>> So, once the current Camel integration POC has been integrated in
>>>>>>>> Stanbol 1.0 and extended to support the deployment of routes
>>>>>>>> defined by
>>>>>>>> Java DSL (through bundles) and XML (route files), the next step
>>>>>>>> will be
>>>>>>>> thinking and redesigning the current architecture trying to avoid
>>>>>>>> the
>>>>>>>> duplicated code and providing a more extendable and easy to use
>>>>>>>> Workflow
>>>>>>>> API, because with the current integration only direct routes can be
>>>>>>>> triggered using REST API which means that the defined routes must be
>>>>>>>> configured properly using a direct endpoint consumer. Anyway, routes
>>>>>>>> starting in some other way like timers are triggered directly in the
>>>>>>>> deployment, so this has to be taken into account for the new API
>>>>>>>> (and REST
>>>>>>>> API).
>>>>>>>>
>>>>>>>> In parallel and for the second part, new Stanbol Camel components
>>>>>>>> will
>>>>>>>> be developed in order to be used in new routes. So if any of you
>>>>>>>> have use
>>>>>>>> cases for this involving Stanbol components, please drop some lines
>>>>>>>> here in
>>>>>>>> order to prioritize the Stanbol Camel components to be developed.
>>>>>>>>
>>>>>>>> Comments and suggestions are more than welcome
>>>>>>>>
>>>>>>>> Regards
>>>>>>>>
>>>>>>>> [1] https://issues.apache.org/jira/browse/STANBOL-1348
>>>>>>>> [2] https://github.com/adperezmorales/stanbol-camel-workflow/
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Mon, Jun 2, 2014 at 7:00 PM, Antonio David Perez Morales <
>>>>>>>> aperez@zaizi.com> wrote:
>>>>>>>>
>>>>>>>> Hi stanbolers
>>>>>>>>>
>>>>>>>>> As part of the issue [1] , I have created a maven archetype useful
>>>>>>>>> to
>>>>>>>>> generate Camel routes in Java DSL.
>>>>>>>>> The archetype generates a Java project with all the dependencies
>>>>>>>>> and
>>>>>>>>> one Java class with a method which has to be filled. In this
>>>>>>>>> method, Camel
>>>>>>>>> Java DSL syntax is used to create the route.
>>>>>>>>> By default and as a first approach, the class will use the route
>>>>>>>>> name
>>>>>>>>> given during the project creation to enable a Camel direct
>>>>>>>>> endpoint with
>>>>>>>>> such name.
>>>>>>>>> The code of the first archetype version can be found at [2].
>>>>>>>>>
>>>>>>>>> The next task will be providing a Felix custom artifact to be able
>>>>>>>>> to
>>>>>>>>> deploy XML-based routes in Stanbol, placing a custom file in the
>>>>>>>>> Stanbol
>>>>>>>>> datafiles directory.
>>>>>>>>> After that, it will be time to think and redesign the architecture
>>>>>>>>> to
>>>>>>>>> integrate Camel workflows inside Stanbol in a better way, more
>>>>>>>>> configurable
>>>>>>>>> and extendable.
>>>>>>>>>
>>>>>>>>> Comments and suggestions are more than welcome
>>>>>>>>>
>>>>>>>>> Regards
>>>>>>>>>
>>>>>>>>> [1] https://issues.apache.org/jira/browse/STANBOL-1348
>>>>>>>>> [2] https://github.com/adperezmorales/stanbol-camel-workflow/
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Fri, May 30, 2014 at 8:03 PM, Antonio David Perez Morales <
>>>>>>>>> aperez@zaizi.com> wrote:
>>>>>>>>>
>>>>>>>>> Hi all
>>>>>>>>>>
>>>>>>>>>> After a hard fight this week, I managed to get it work the
>>>>>>>>>> Florent's
>>>>>>>>>> proof of concept code in the Stanbol 1.0 branch [1]
>>>>>>>>>> The code is uploaded in my github account [3]. As I said in a
>>>>>>>>>> previous
>>>>>>>>>> mail, I prefer to do it separately and after the project,
>>>>>>>>>> uploading the
>>>>>>>>>> developed code into a Stanbol branch.
>>>>>>>>>>
>>>>>>>>>> The 1.0.0 version has some changes in how the Jersey endpoints are
>>>>>>>>>> registered and also new classes and packages, so it was not a
>>>>>>>>>> trivial task
>>>>>>>>>> to make work the current proof of concept. Moreover I don't like
>>>>>>>>>> to simply
>>>>>>>>>> copy and paste code and make the needed changes. I always want to
>>>>>>>>>> understand how the things work and how they are developed in
>>>>>>>>>> order to be
>>>>>>>>>> able to change/modify them or develop new code around them.
>>>>>>>>>>
>>>>>>>>>> The steps done to achieve it have been the following:
>>>>>>>>>> - Updated pom files to the Stanbol 1.0.0-SNAPSHOT version
>>>>>>>>>> - Updated bundle levels in bundlelist package to fit the Stanbol
>>>>>>>>>> 1.0
>>>>>>>>>> version levels
>>>>>>>>>> - Adapted cameljobmanager package code to Stanbol 1.0.0-SNAPSHOT
>>>>>>>>>> classes and using Java OSGI annotations instead of SCR
>>>>>>>>>> annotations in
>>>>>>>>>> Javadoc
>>>>>>>>>> - Updated flow web package to Stanbol 1.0.0-SNAPSHOT classes and
>>>>>>>>>> modified needed resources
>>>>>>>>>> - Added Java OSGI annotations to the route (WeightedChain)
>>>>>>>>>> instead of
>>>>>>>>>> SCR annotations in javadoc
>>>>>>>>>> - Updated launcher to use the 1.0.0-SNAPSHOT packages and needed
>>>>>>>>>> bundles
>>>>>>>>>>
>>>>>>>>>> So now, the http://localhost:8080/flow endpoint will use the only
>>>>>>>>>> Camel route (defined by WeightedChain) to call all the registered
>>>>>>>>>> Enhancement Engines (ordered by EnhancementEngine order property).
>>>>>>>>>> For testing purposes, the /flow/{flowName} has been removed,
>>>>>>>>>> because
>>>>>>>>>> all this code needs to be re-designed and re-implemented so I
>>>>>>>>>> only wanted
>>>>>>>>>> to make it work to have a first (simple) integration in Stanbol
>>>>>>>>>> 1.0. This
>>>>>>>>>> functionality will be added again to trigger custom routes once
>>>>>>>>>> the next
>>>>>>>>>> step (defined below) is developed.
>>>>>>>>>>
>>>>>>>>>> The next step [2] will be support to write and configure routes
>>>>>>>>>> in XML
>>>>>>>>>> format, putting the file in datafiles in order to be loaded by a
>>>>>>>>>> Felix
>>>>>>>>>> custom artifact (as Rupert pointed out in a previous mail) and
>>>>>>>>>> create a
>>>>>>>>>> Maven archetype to create bundles defining routes which will be
>>>>>>>>>> loaded
>>>>>>>>>> using the Felix bundle tab. If necessary, as we talked in previous
>>>>>>>>>> messages, a REST endpoint receiving routes in XML can be
>>>>>>>>>> developed as an
>>>>>>>>>> alternative to the first approach. This is my objective for the
>>>>>>>>>> midterm.
>>>>>>>>>>
>>>>>>>>>> After the midterm, the new Stanbol components for Apache Camel
>>>>>>>>>> will be
>>>>>>>>>> developed and also the new architecture for Camel in Stanbol.
>>>>>>>>>>
>>>>>>>>>> Comments on this and for use cases for Stanbol Camel components
>>>>>>>>>> are
>>>>>>>>>> more than welcome.
>>>>>>>>>>
>>>>>>>>>> Regards
>>>>>>>>>>
>>>>>>>>>> [1] https://issues.apache.org/jira/browse/STANBOL-1347
>>>>>>>>>> [2] https://issues.apache.org/jira/browse/STANBOL-1348
>>>>>>>>>> [3] https://github.com/adperezmorales/stanbol-camel-workflow/
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Tue, May 27, 2014 at 6:18 PM, Antonio David Perez Morales <
>>>>>>>>>> aperez@zaizi.com> wrote:
>>>>>>>>>>
>>>>>>>>>> Hi people
>>>>>>>>>>>
>>>>>>>>>>> I have already started to work on [1] to integrate current
>>>>>>>>>>> Florent's
>>>>>>>>>>> code into Stanbol 1.0.
>>>>>>>>>>> As a first approach, only changing the dependency versions to new
>>>>>>>>>>> Stanbol 1.0, many issues have arisen:
>>>>>>>>>>> - Deprecated use of classes
>>>>>>>>>>> - Classes which have changed from package
>>>>>>>>>>> - Some classes not necessary now
>>>>>>>>>>> - Classes not used which were causing conflicts
>>>>>>>>>>> - ...
>>>>>>>>>>>
>>>>>>>>>>> So now I'm trying to resolve all these problems to replicate the
>>>>>>>>>>> same
>>>>>>>>>>> behavior from 0.9 into 1.0. I will upload the code to a Github
>>>>>>>>>>> repository
>>>>>>>>>>> in my account (which will be pushed later into a Stanbol branch
>>>>>>>>>>> after the
>>>>>>>>>>> project) in order to track the advances.
>>>>>>>>>>> Once I can resolve all these problems, I will take a look to the
>>>>>>>>>>> Felix Custom Artifacts poiinted out by Rupert in a previous
>>>>>>>>>>> message to find
>>>>>>>>>>> out the best way to deploy (and manage) route configurations
>>>>>>>>>>> (felix
>>>>>>>>>>> artifacts, watchservice java, rest endpoint to receive xml
>>>>>>>>>>> routes, etc).
>>>>>>>>>>>
>>>>>>>>>>> Comments on this and future tasks are more than welcome.
>>>>>>>>>>>
>>>>>>>>>>> Regards
>>>>>>>>>>>
>>>>>>>>>>> [1] https://issues.apache.org/jira/browse/STANBOL-1347
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Tue, May 27, 2014 at 9:53 AM, Rafa Haro <rh...@apache.org>
>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>> Hi Rupert, Florent and Antonio
>>>>>>>>>>>>
>>>>>>>>>>>> El 27/05/14 08:51, Rupert Westenthaler escribió:
>>>>>>>>>>>>
>>>>>>>>>>>> As the result of Enhancement Routes is content + metadata I
>>>>>>>>>>>> can not
>>>>>>>>>>>>
>>>>>>>>>>>>> see what you want to "store" in the Entityhub that is about
>>>>>>>>>>>>> managing
>>>>>>>>>>>>> Entities.
>>>>>>>>>>>>>
>>>>>>>>>>>>> > - entityhub: To query/update the entityhub component
>>>>>>>>>>>>> Maybe. If you can come up with a good use case ^^
>>>>>>>>>>>>>
>>>>>>>>>>>>> > - contenthub: To develop a new content-hub using
>>>>>>>>>>>>> chain/engine
>>>>>>>>>>>>>
>>>>>>>>>>>>>> components
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> and solr/elasticsearch/whatever component (solr and
>>>>>>>>>>>>>>> elasticsearch
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>> component
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> already exist in Camel)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>> IMO implementing a new Contenthub like component is outside
>>>>>>>>>>>>> the
>>>>>>>>>>>>> scope
>>>>>>>>>>>>> of this GSoC project. However If there is already
>>>>>>>>>>>>> Solr/Elasticsearch
>>>>>>>>>>>>> component it would be a really useful thing
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Regarding this, in my opinion, the use case of an eventual
>>>>>>>>>>>> integration with a Content hub is probably one of the most
>>>>>>>>>>>> clear for this
>>>>>>>>>>>> project. I'm not sure if that is what Antonio was trying to
>>>>>>>>>>>> explain but,
>>>>>>>>>>>> with a single route using as last endpoint Solr or any other
>>>>>>>>>>>> backend
>>>>>>>>>>>> system, we would be almost cloning the same functionality than
>>>>>>>>>>>> the previous
>>>>>>>>>>>> ContentHub implementation (Stanbol 0.12). Entities could be
>>>>>>>>>>>> dereferenced
>>>>>>>>>>>> using the EntityHub before storing the content along with the
>>>>>>>>>>>> metadata,
>>>>>>>>>>>> which is the point of integration of the EntityHub in such use
>>>>>>>>>>>> case. And
>>>>>>>>>>>> even most interesting, now with the integration of Marmotta
>>>>>>>>>>>> contributed by
>>>>>>>>>>>> Rupert, it would be possible to use a whole graph for
>>>>>>>>>>>> dereferencing, so
>>>>>>>>>>>> "simply" routing components like Enhancer->Marmotta->Solr
>>>>>>>>>>>> sounds to me like
>>>>>>>>>>>> an interesting use case.
>>>>>>>>>>>>
>>>>>>>>>>>> wdyt?
>>>>>>>>>>>>
>>>>>>>>>>>> Cheers,
>>>>>>>>>>>> Rafa
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>
>>>
>>
>
--
------------------------------
This message should be regarded as confidential. If you have received this
email in error please notify the sender and destroy it immediately.
Statements of intent shall only become binding when confirmed in hard copy
by an authorised signatory.
Zaizi Ltd is registered in England and Wales with the registration number
6440931. The Registered Office is Brook House, 229 Shepherds Bush Road,
London W6 7AN.
Re: Camel integration (was : Re: Community bonding period started)
Posted by Antonio David Perez Morales <ap...@zaizi.com>.
Hi all
As anticipated in the previous mail, I have develop a first version of the
Stanbol Solr component. This component (by default managing the
stanbol-solr camel protocol) extends the Camel Solr component, so all the
properties used to configure it ca be used in this component as well.
The component is responsible of extracting fields and values from the
entities in the Content-Item and creates a Solr Document with the content
and metadata to be indexed in Solr. In this first version, no filtering is
being applied to the entities (for example, get the field-values only from
the entity with higher confidence value).
The first version of the component allows three conf parameters in a route:
- ldpath : LDPath program to be used to extract the values of the fields.
As mentioned in the previous mail, if a different ldpath in th dereference
engine is used then the properties to be extracted may not exist.
- fields : A comma-separated list of values containing the fields to be
extracted from the entities and indexed in Solr.
- useDereferenceLdpath: If no ldpath program is defined, then this boolean
flag allows to use the same ldpath program used by the dereference engine
(getting it from the information contained in the content-item and passed
in the HTTP request to the enhancer or configured in the chain/engine
component). Default value is true.
A sample route using this component could be the following:
<routes xmlns="http://camel.apache.org/schema/spring">
<route id="stanbolsolr">
<from uri="direct://stanbolsolr" />
<to
uri="chain://default?enhancer.engines.dereference.ldpath=%40prefix%20test%20%3A%20%3Chttp%3A%2F%2Ftest.org%2F%3E%3B%test%3Aname%3Drdfs%3Alabel%20%3A%3A%20xsd%3Astring%3B"
/>
<to uri="stanbol-solr://localhost:8983/solr" />
</route>
</routes>
As a future extensions of this component, a new property specifying a
configured dereference engine to use for the ldpath and filtering the
entities to get only the one with the higher confidence value will be
developed.
With this component, we can have a some features similar to the old Stanbol
content hub. So, i think improving this component we could achieve to have
to content-hub back to Stanbol (but using an external Solr instance, which
I think is good to not overloading the Stanbol application)
Moreover, as part of the "use cases" project part and as discussed in the
Stanbol IRC Channel, I'm also evaluating Siren [1], an extension of Solr
bringing new and improved capabilities to it. It's very useful for
structured document search.
So my idea is to try to create a Siren component for Camel integrated in
Stanbol, to bring the possibility to store (in an easy way) the content
along with the extracted metadata in a structured way, instead of simply
creating new fields for a document.
Stay tuned for new advances.
As always, comments are more than welcome.
Regards
[1] http://sirendb.com/
On Wed, Jul 16, 2014 at 12:53 PM, Antonio David Perez Morales <
aperez@zaizi.com> wrote:
> Hi people
>
> Continuing with the project work , I have implemented some improvements to
> chain and engine components to allow defining enhancer properties (like
> enhancer.engines.dereference.ldpath) in the route component definition.
> Example :
> from(direct://test).to(engine://dereference-engine?enhancer.engines.dereference.ldpath=EXPRESSION).
> As said in previous mails, the engines and chains have to be configured
> through Felix console.
>
> Regarding the last discussion about a new kind of ContentHub back to
> Stanbol as an use case for the workflow integration, I have successfully
> created a custom Camel processor to create the document with the content
> and enhancement metadata in order to be sent to Solr. It takes the LDPath
> expression (configured in the dereference engine component via
> enhancer.engines.dereference.ldpath query parameter or camel component
> parameter) to extract the metadata to be indexed. So using a route like
> from().to(chain://Default).process(ContentItemProcessor).to(solr://localhost:8983/solr),
> we can have new indexed documents in Solr containing the text and the
> extracted enhancement metadata in order to be use in semantic searchs in
> the external Solr. Of course, the Solr schema needs to be created in the
> remote Solr beforehand. It is only a brief proof of concept of such
> functionality.
>
> My idea is to use an external Solr to store the content and semantic
> metadata for semantic search purposes, as opposite of the old ContentHub
> which was using an internal SolrYard, creating the schema from the
> configured LDPath expression.
>
> The next step in this task will be create a custom StanbolSolr component,
> able to perform the functionality of the previous processor and Solr, but
> allowing configuring the LDPath, fields and properties to be extracted and
> put as metadata in the new Solr document. These properties will be applied
> to the ContentItem metadata, so if an entity dereference engine is
> configured with a different LDPath expression or fields, maybe the
> properties to be extracted will not exist.
> As future improvement of this component, we could add a new conf parameter
> specifying a configured dereference engine to be used before applying the
> configuration.
>
> Stay tuned for further advances.
>
> As always if you have any questions or comments, please drop some lines
> here.
>
> Regards
>
> PS: The example routes used are very simple and lineals, but for some
> scenarios, parallel executions of engines, multicast, aggregator, etc
> (supported by camel) could be used to speed up the enhancement process.
>
>
>
>
>
>
> On Tue, Jul 8, 2014 at 9:46 AM, Antonio David Perez Morales <
> aperez@zaizi.com> wrote:
>
>> Hi Rafa and all
>>
>> In my opinion, the Content Hub back in Stanbol for Semantic Search
>> capabilities is a great use case to be implemented.
>> Waiting for Florent's opinion, I could start first only with Solr (whose
>> component already exists in Camel but it needs to be adapted like the
>> ActiveMQ component) and creating a custom transformer bean for Camel to
>> have the original Content Hub. After that, we could think to create the
>> SIREn component and the new transformer for it, giving the users the
>> possibility of use one of them.
>>
>> What do you think? Is It an interesting use case for the Camel
>> integration application?
>>
>> Regards
>>
>>
>> On Mon, Jul 7, 2014 at 4:27 PM, Rafa Haro <rh...@apache.org> wrote:
>>
>>> Hi guys,
>>>
>>> El 01/07/14 10:20, Antonio David Perez Morales escribió:
>>>
>>> Hi all
>>>>
>>>> Continuing with the project, I have managed successfully the
>>>> integration of
>>>> activemq camel component (and also jms) into the Stanbol Camel
>>>> integration.
>>>> This has been a hard task due to the dependencies needed by the
>>>> component
>>>> and also due to the fact that we had to provide an activemq component
>>>> configurable through Felix web console.
>>>>
>>>> With this addition, we are in the position to integrate business logic
>>>> into
>>>> Stanbol routes through a message service provided by activemq (jms).
>>>>
>>> Nice Antonio, let's see is someone has an interesting use case to
>>> implement in this context.
>>>
>>>
>>>> As a first test, I have deployed a route which consumes messages
>>>> (content)
>>>> from an activemq queue, enhance them using the default chain and then
>>>> write
>>>> the result into a file. It's a simple test but it works quite well. In
>>>> this
>>>> case, Stanbol is working in a standalone mode, that is to say, we don't
>>>> have to explicitly call Stanbol to enhance content but Stanbol is
>>>> triggered
>>>> based on some external events (a new queue message)
>>>>
>>>> As indicated in the previous mail, I still have some pending things to
>>>> be
>>>> done (because I couldn't do them last week) but in order to go forward
>>>> with
>>>> the project I ask you for some interesting use cases where to apply the
>>>> new
>>>> workflow component in order to give added value to it and also in order
>>>> to
>>>> develop and provide more workflow (camel) components useful for those
>>>> and
>>>> other use cases.
>>>>
>>> Awaiting for the community feedback and also for Florent's opinion
>>> regarding the rest of the project, as I have expressed in recent emails,
>>> I'm eager to see the Content Hub back in Stanbol. And this is because of,
>>> from the point of view of the use of Stanbol in the enterprise, Semantic
>>> Search is one of the most common use cases. So, to have an enterprise
>>> search backend as the last component of a processing route in any
>>> architecture where stanbol could be plugged sounds key for me. In recent
>>> discussions at the Stanbol IRC channel, we have been analysing Siren (
>>> https://github.com/rdelbru/SIREn), a Lucene/Solr extension which major
>>> advantage is the possibility to index tree structures, allowing then to
>>> index structured data without losing full text search capabilities. To
>>> refactor old ContentHub component to use Siren is out of scope of this
>>> project but, in my opinion, an interesting use case could be to develop a
>>> Siren Camel Component and a transformer from ContentItem to Siren Object or
>>> whatever and integrate both in Stanbol.
>>>
>>> What do you guys think?
>>>
>>> Cheers,
>>> Rafa
>>>
>>>
>>>
>>>
>>>> Regards
>>>>
>>>>
>>>> On Mon, Jun 23, 2014 at 6:16 PM, Antonio David Perez Morales <
>>>> aperez@zaizi.com> wrote:
>>>>
>>>> Hi Stanbolers
>>>>>
>>>>> The GSoC 2014 midterm is here and I want to give you a summary of the
>>>>> work
>>>>> already done so far:
>>>>>
>>>>> - Adapted previous Camel integration PoC done by Florent into Stanbol
>>>>> 1.0
>>>>> version.
>>>>> - Improved EngineComponent used by Camel to execute Enhancement Engines
>>>>> (configured through Stanbol web console as usual) using the engine://
>>>>> uri
>>>>> scheme in routes.
>>>>> - Created ChainComponent used by Camel to execute Enhancement Chains
>>>>> using
>>>>> the chain:// uri scheme in routes (both Camel components are provided
>>>>> as
>>>>> OSGI components, so the uri scheme can be changed through the Stanbol
>>>>> web
>>>>> console)
>>>>> - Created a custom artifact for Apache Felix Fileinstall in order to be
>>>>> able to install routes defined in Camel Spring XML DSL placing a route
>>>>> file
>>>>> (with 'route' extension) in the stanbol/fileinstall directory
>>>>> - Created a custom archetype to ease the development of bundles
>>>>> containing
>>>>> route definitions in Java DSL. The archetype generates a class
>>>>> extending
>>>>> 'RouteBuilder' which creates a default Camel direct endpoint used by
>>>>> other
>>>>> Stanbol Workflow components to execute the route.
>>>>> - Created a first version of Workflow API, which contains different
>>>>> OSGI
>>>>> components which allow registering Camel components/routes,
>>>>> start/stop/execute routes, add/remove components used in routes, etc.
>>>>> - REST endpoint is provided to test the execution of routes using REST
>>>>> requests (/flow/{routeId} )
>>>>> - Modified the PoC full launcher to use all the new bundles to support
>>>>> the
>>>>> workflow feature.
>>>>> - Installed JBoss developer studio which comes with Camel support in
>>>>> order
>>>>> to create routes in a visual way with the possibility to be exported as
>>>>> Spring XML DSL format
>>>>>
>>>>> Some pending things I will try to do during this week:
>>>>> - Improve the web package to create the needed endpoints to query the
>>>>> registered routes, registered camel components, etc
>>>>> - Improve the web package to remove classes copied from Stanbol jersey
>>>>> module used for testing
>>>>> - Update README.md files in the repository with all the new information
>>>>> - Document the installation and configuration of JBoss developer studio
>>>>> for Camel routes creation
>>>>> - Create all the JIRA issued related to the work already done
>>>>>
>>>>>
>>>>> For the second part of the project, I would like to read some comments
>>>>> about interesting use cases in order to develop the needed Stanbol and
>>>>> Camel components to support them.
>>>>>
>>>>> If you have any comment, please drop some lines in order to discuss the
>>>>> new things to be done.
>>>>>
>>>>> Regards
>>>>>
>>>>>
>>>>>
>>>>> On Sat, Jun 14, 2014 at 3:39 PM, Antonio David Perez Morales <
>>>>> aperez@zaizi.com> wrote:
>>>>>
>>>>> Hi guys
>>>>>>
>>>>>> Continuing with the project, and as part of the refactoring/new
>>>>>> architecture I have started to modify some workflow components in
>>>>>> order to
>>>>>> create a better API and architecture based on OSGI components. As a
>>>>>> first
>>>>>> step and in order to have the same behavior than the current one
>>>>>> (regarding
>>>>>> enhancement process), a chain component has been created to simulate
>>>>>> the
>>>>>> chain behaviour. This new component uses internally the ChainManager
>>>>>> and
>>>>>> EnhancementJobManager component to perform the business logic. This
>>>>>> way, a
>>>>>> new protocol 'chain' can be used in the routes deployed in Stanbol.
>>>>>> The
>>>>>> chains are configured in the same way, using Stanbol admin console.
>>>>>>
>>>>>> Now, we can combine single engine executions with chains executions in
>>>>>> routes deployed in Stanbol using the alternatives described in
>>>>>> previous
>>>>>> mails and in the issue [1]. Both engines and chains are configured
>>>>>> through
>>>>>> Stanbol admin console. You can see the refactoring advances in [2] (a
>>>>>> branch used for refactoring the current PoC of Workflow in Stanbol
>>>>>> 1.0). Of
>>>>>> course, the Camel EIP and other Camel components can be used in the
>>>>>> deployed routes as well.
>>>>>>
>>>>>> With the new Camel routes support, we can have a Stanbol running and
>>>>>> enhancing content without receiving any HTTP request to start the
>>>>>> enhancement process, because the routes can be triggered by external
>>>>>> events
>>>>>> ocurred in a queue, database, etc. Moreover the semantic lifting
>>>>>> process
>>>>>> can be splitted and merged with some application steps, so the issue
>>>>>> [3]
>>>>>> requesting asynchronous call support for enhancement could be solved.
>>>>>>
>>>>>> Anyway, if some of you have any suggestions for new components to be
>>>>>> deployed for the second part of the project, or another kind of
>>>>>> suggestion,
>>>>>> please drop here some lines to continue with the discussion.
>>>>>>
>>>>>> Regards
>>>>>>
>>>>>> [1] https://issues.apache.org/jira/browse/STANBOL-1348
>>>>>> [2]
>>>>>> https://github.com/adperezmorales/stanbol-camel-
>>>>>> workflow/tree/refactoring
>>>>>> [3] https://issues.apache.org/jira/browse/STANBOL-263
>>>>>>
>>>>>>
>>>>>> On Wed, Jun 11, 2014 at 10:01 AM, Antonio David Perez Morales <
>>>>>> aperez@zaizi.com> wrote:
>>>>>>
>>>>>> Hi people
>>>>>>>
>>>>>>> As part of the GSoC project for the midterm and according to the
>>>>>>> issue
>>>>>>> [1], a custom Apache Felix Fileinstall artifact has been created in
>>>>>>> order
>>>>>>> to deploy Camel routes defined in XML (Spring DSL) placing a file
>>>>>>> with
>>>>>>> .route extension in a configured directory (like stanbol/fileinstall
>>>>>>> directory). Moreover since this artifact depends on Fileinstall
>>>>>>> bundle, the
>>>>>>> created launcher has been modified to have that bundle in the OSGI
>>>>>>> context
>>>>>>> by default.
>>>>>>>
>>>>>>> So, once the current Camel integration POC has been integrated in
>>>>>>> Stanbol 1.0 and extended to support the deployment of routes defined
>>>>>>> by
>>>>>>> Java DSL (through bundles) and XML (route files), the next step will
>>>>>>> be
>>>>>>> thinking and redesigning the current architecture trying to avoid the
>>>>>>> duplicated code and providing a more extendable and easy to use
>>>>>>> Workflow
>>>>>>> API, because with the current integration only direct routes can be
>>>>>>> triggered using REST API which means that the defined routes must be
>>>>>>> configured properly using a direct endpoint consumer. Anyway, routes
>>>>>>> starting in some other way like timers are triggered directly in the
>>>>>>> deployment, so this has to be taken into account for the new API
>>>>>>> (and REST
>>>>>>> API).
>>>>>>>
>>>>>>> In parallel and for the second part, new Stanbol Camel components
>>>>>>> will
>>>>>>> be developed in order to be used in new routes. So if any of you
>>>>>>> have use
>>>>>>> cases for this involving Stanbol components, please drop some lines
>>>>>>> here in
>>>>>>> order to prioritize the Stanbol Camel components to be developed.
>>>>>>>
>>>>>>> Comments and suggestions are more than welcome
>>>>>>>
>>>>>>> Regards
>>>>>>>
>>>>>>> [1] https://issues.apache.org/jira/browse/STANBOL-1348
>>>>>>> [2] https://github.com/adperezmorales/stanbol-camel-workflow/
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Mon, Jun 2, 2014 at 7:00 PM, Antonio David Perez Morales <
>>>>>>> aperez@zaizi.com> wrote:
>>>>>>>
>>>>>>> Hi stanbolers
>>>>>>>>
>>>>>>>> As part of the issue [1] , I have created a maven archetype useful
>>>>>>>> to
>>>>>>>> generate Camel routes in Java DSL.
>>>>>>>> The archetype generates a Java project with all the dependencies and
>>>>>>>> one Java class with a method which has to be filled. In this
>>>>>>>> method, Camel
>>>>>>>> Java DSL syntax is used to create the route.
>>>>>>>> By default and as a first approach, the class will use the route
>>>>>>>> name
>>>>>>>> given during the project creation to enable a Camel direct endpoint
>>>>>>>> with
>>>>>>>> such name.
>>>>>>>> The code of the first archetype version can be found at [2].
>>>>>>>>
>>>>>>>> The next task will be providing a Felix custom artifact to be able
>>>>>>>> to
>>>>>>>> deploy XML-based routes in Stanbol, placing a custom file in the
>>>>>>>> Stanbol
>>>>>>>> datafiles directory.
>>>>>>>> After that, it will be time to think and redesign the architecture
>>>>>>>> to
>>>>>>>> integrate Camel workflows inside Stanbol in a better way, more
>>>>>>>> configurable
>>>>>>>> and extendable.
>>>>>>>>
>>>>>>>> Comments and suggestions are more than welcome
>>>>>>>>
>>>>>>>> Regards
>>>>>>>>
>>>>>>>> [1] https://issues.apache.org/jira/browse/STANBOL-1348
>>>>>>>> [2] https://github.com/adperezmorales/stanbol-camel-workflow/
>>>>>>>>
>>>>>>>>
>>>>>>>> On Fri, May 30, 2014 at 8:03 PM, Antonio David Perez Morales <
>>>>>>>> aperez@zaizi.com> wrote:
>>>>>>>>
>>>>>>>> Hi all
>>>>>>>>>
>>>>>>>>> After a hard fight this week, I managed to get it work the
>>>>>>>>> Florent's
>>>>>>>>> proof of concept code in the Stanbol 1.0 branch [1]
>>>>>>>>> The code is uploaded in my github account [3]. As I said in a
>>>>>>>>> previous
>>>>>>>>> mail, I prefer to do it separately and after the project,
>>>>>>>>> uploading the
>>>>>>>>> developed code into a Stanbol branch.
>>>>>>>>>
>>>>>>>>> The 1.0.0 version has some changes in how the Jersey endpoints are
>>>>>>>>> registered and also new classes and packages, so it was not a
>>>>>>>>> trivial task
>>>>>>>>> to make work the current proof of concept. Moreover I don't like
>>>>>>>>> to simply
>>>>>>>>> copy and paste code and make the needed changes. I always want to
>>>>>>>>> understand how the things work and how they are developed in order
>>>>>>>>> to be
>>>>>>>>> able to change/modify them or develop new code around them.
>>>>>>>>>
>>>>>>>>> The steps done to achieve it have been the following:
>>>>>>>>> - Updated pom files to the Stanbol 1.0.0-SNAPSHOT version
>>>>>>>>> - Updated bundle levels in bundlelist package to fit the Stanbol
>>>>>>>>> 1.0
>>>>>>>>> version levels
>>>>>>>>> - Adapted cameljobmanager package code to Stanbol 1.0.0-SNAPSHOT
>>>>>>>>> classes and using Java OSGI annotations instead of SCR annotations
>>>>>>>>> in
>>>>>>>>> Javadoc
>>>>>>>>> - Updated flow web package to Stanbol 1.0.0-SNAPSHOT classes and
>>>>>>>>> modified needed resources
>>>>>>>>> - Added Java OSGI annotations to the route (WeightedChain) instead
>>>>>>>>> of
>>>>>>>>> SCR annotations in javadoc
>>>>>>>>> - Updated launcher to use the 1.0.0-SNAPSHOT packages and needed
>>>>>>>>> bundles
>>>>>>>>>
>>>>>>>>> So now, the http://localhost:8080/flow endpoint will use the only
>>>>>>>>> Camel route (defined by WeightedChain) to call all the registered
>>>>>>>>> Enhancement Engines (ordered by EnhancementEngine order property).
>>>>>>>>> For testing purposes, the /flow/{flowName} has been removed,
>>>>>>>>> because
>>>>>>>>> all this code needs to be re-designed and re-implemented so I only
>>>>>>>>> wanted
>>>>>>>>> to make it work to have a first (simple) integration in Stanbol
>>>>>>>>> 1.0. This
>>>>>>>>> functionality will be added again to trigger custom routes once
>>>>>>>>> the next
>>>>>>>>> step (defined below) is developed.
>>>>>>>>>
>>>>>>>>> The next step [2] will be support to write and configure routes in
>>>>>>>>> XML
>>>>>>>>> format, putting the file in datafiles in order to be loaded by a
>>>>>>>>> Felix
>>>>>>>>> custom artifact (as Rupert pointed out in a previous mail) and
>>>>>>>>> create a
>>>>>>>>> Maven archetype to create bundles defining routes which will be
>>>>>>>>> loaded
>>>>>>>>> using the Felix bundle tab. If necessary, as we talked in previous
>>>>>>>>> messages, a REST endpoint receiving routes in XML can be developed
>>>>>>>>> as an
>>>>>>>>> alternative to the first approach. This is my objective for the
>>>>>>>>> midterm.
>>>>>>>>>
>>>>>>>>> After the midterm, the new Stanbol components for Apache Camel
>>>>>>>>> will be
>>>>>>>>> developed and also the new architecture for Camel in Stanbol.
>>>>>>>>>
>>>>>>>>> Comments on this and for use cases for Stanbol Camel components are
>>>>>>>>> more than welcome.
>>>>>>>>>
>>>>>>>>> Regards
>>>>>>>>>
>>>>>>>>> [1] https://issues.apache.org/jira/browse/STANBOL-1347
>>>>>>>>> [2] https://issues.apache.org/jira/browse/STANBOL-1348
>>>>>>>>> [3] https://github.com/adperezmorales/stanbol-camel-workflow/
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Tue, May 27, 2014 at 6:18 PM, Antonio David Perez Morales <
>>>>>>>>> aperez@zaizi.com> wrote:
>>>>>>>>>
>>>>>>>>> Hi people
>>>>>>>>>>
>>>>>>>>>> I have already started to work on [1] to integrate current
>>>>>>>>>> Florent's
>>>>>>>>>> code into Stanbol 1.0.
>>>>>>>>>> As a first approach, only changing the dependency versions to new
>>>>>>>>>> Stanbol 1.0, many issues have arisen:
>>>>>>>>>> - Deprecated use of classes
>>>>>>>>>> - Classes which have changed from package
>>>>>>>>>> - Some classes not necessary now
>>>>>>>>>> - Classes not used which were causing conflicts
>>>>>>>>>> - ...
>>>>>>>>>>
>>>>>>>>>> So now I'm trying to resolve all these problems to replicate the
>>>>>>>>>> same
>>>>>>>>>> behavior from 0.9 into 1.0. I will upload the code to a Github
>>>>>>>>>> repository
>>>>>>>>>> in my account (which will be pushed later into a Stanbol branch
>>>>>>>>>> after the
>>>>>>>>>> project) in order to track the advances.
>>>>>>>>>> Once I can resolve all these problems, I will take a look to the
>>>>>>>>>> Felix Custom Artifacts poiinted out by Rupert in a previous
>>>>>>>>>> message to find
>>>>>>>>>> out the best way to deploy (and manage) route configurations
>>>>>>>>>> (felix
>>>>>>>>>> artifacts, watchservice java, rest endpoint to receive xml
>>>>>>>>>> routes, etc).
>>>>>>>>>>
>>>>>>>>>> Comments on this and future tasks are more than welcome.
>>>>>>>>>>
>>>>>>>>>> Regards
>>>>>>>>>>
>>>>>>>>>> [1] https://issues.apache.org/jira/browse/STANBOL-1347
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Tue, May 27, 2014 at 9:53 AM, Rafa Haro <rh...@apache.org>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>> Hi Rupert, Florent and Antonio
>>>>>>>>>>>
>>>>>>>>>>> El 27/05/14 08:51, Rupert Westenthaler escribió:
>>>>>>>>>>>
>>>>>>>>>>> As the result of Enhancement Routes is content + metadata I
>>>>>>>>>>> can not
>>>>>>>>>>>
>>>>>>>>>>>> see what you want to "store" in the Entityhub that is about
>>>>>>>>>>>> managing
>>>>>>>>>>>> Entities.
>>>>>>>>>>>>
>>>>>>>>>>>> > - entityhub: To query/update the entityhub component
>>>>>>>>>>>> Maybe. If you can come up with a good use case ^^
>>>>>>>>>>>>
>>>>>>>>>>>> > - contenthub: To develop a new content-hub using
>>>>>>>>>>>> chain/engine
>>>>>>>>>>>>
>>>>>>>>>>>>> components
>>>>>>>>>>>>>
>>>>>>>>>>>>>> and solr/elasticsearch/whatever component (solr and
>>>>>>>>>>>>>> elasticsearch
>>>>>>>>>>>>>>
>>>>>>>>>>>>> component
>>>>>>>>>>>>>
>>>>>>>>>>>>>> already exist in Camel)
>>>>>>>>>>>>>>
>>>>>>>>>>>>> IMO implementing a new Contenthub like component is outside the
>>>>>>>>>>>> scope
>>>>>>>>>>>> of this GSoC project. However If there is already
>>>>>>>>>>>> Solr/Elasticsearch
>>>>>>>>>>>> component it would be a really useful thing
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Regarding this, in my opinion, the use case of an eventual
>>>>>>>>>>> integration with a Content hub is probably one of the most clear
>>>>>>>>>>> for this
>>>>>>>>>>> project. I'm not sure if that is what Antonio was trying to
>>>>>>>>>>> explain but,
>>>>>>>>>>> with a single route using as last endpoint Solr or any other
>>>>>>>>>>> backend
>>>>>>>>>>> system, we would be almost cloning the same functionality than
>>>>>>>>>>> the previous
>>>>>>>>>>> ContentHub implementation (Stanbol 0.12). Entities could be
>>>>>>>>>>> dereferenced
>>>>>>>>>>> using the EntityHub before storing the content along with the
>>>>>>>>>>> metadata,
>>>>>>>>>>> which is the point of integration of the EntityHub in such use
>>>>>>>>>>> case. And
>>>>>>>>>>> even most interesting, now with the integration of Marmotta
>>>>>>>>>>> contributed by
>>>>>>>>>>> Rupert, it would be possible to use a whole graph for
>>>>>>>>>>> dereferencing, so
>>>>>>>>>>> "simply" routing components like Enhancer->Marmotta->Solr sounds
>>>>>>>>>>> to me like
>>>>>>>>>>> an interesting use case.
>>>>>>>>>>>
>>>>>>>>>>> wdyt?
>>>>>>>>>>>
>>>>>>>>>>> Cheers,
>>>>>>>>>>> Rafa
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>
>>
>
--
------------------------------
This message should be regarded as confidential. If you have received this
email in error please notify the sender and destroy it immediately.
Statements of intent shall only become binding when confirmed in hard copy
by an authorised signatory.
Zaizi Ltd is registered in England and Wales with the registration number
6440931. The Registered Office is Brook House, 229 Shepherds Bush Road,
London W6 7AN.
Re: Camel integration (was : Re: Community bonding period started)
Posted by Antonio David Perez Morales <ap...@zaizi.com>.
Hi people
Continuing with the project work , I have implemented some improvements to
chain and engine components to allow defining enhancer properties (like
enhancer.engines.dereference.ldpath) in the route component definition.
Example :
from(direct://test).to(engine://dereference-engine?enhancer.engines.dereference.ldpath=EXPRESSION).
As said in previous mails, the engines and chains have to be configured
through Felix console.
Regarding the last discussion about a new kind of ContentHub back to
Stanbol as an use case for the workflow integration, I have successfully
created a custom Camel processor to create the document with the content
and enhancement metadata in order to be sent to Solr. It takes the LDPath
expression (configured in the dereference engine component via
enhancer.engines.dereference.ldpath query parameter or camel component
parameter) to extract the metadata to be indexed. So using a route like
from().to(chain://Default).process(ContentItemProcessor).to(solr://localhost:8983/solr),
we can have new indexed documents in Solr containing the text and the
extracted enhancement metadata in order to be use in semantic searchs in
the external Solr. Of course, the Solr schema needs to be created in the
remote Solr beforehand. It is only a brief proof of concept of such
functionality.
My idea is to use an external Solr to store the content and semantic
metadata for semantic search purposes, as opposite of the old ContentHub
which was using an internal SolrYard, creating the schema from the
configured LDPath expression.
The next step in this task will be create a custom StanbolSolr component,
able to perform the functionality of the previous processor and Solr, but
allowing configuring the LDPath, fields and properties to be extracted and
put as metadata in the new Solr document. These properties will be applied
to the ContentItem metadata, so if an entity dereference engine is
configured with a different LDPath expression or fields, maybe the
properties to be extracted will not exist.
As future improvement of this component, we could add a new conf parameter
specifying a configured dereference engine to be used before applying the
configuration.
Stay tuned for further advances.
As always if you have any questions or comments, please drop some lines
here.
Regards
PS: The example routes used are very simple and lineals, but for some
scenarios, parallel executions of engines, multicast, aggregator, etc
(supported by camel) could be used to speed up the enhancement process.
On Tue, Jul 8, 2014 at 9:46 AM, Antonio David Perez Morales <
aperez@zaizi.com> wrote:
> Hi Rafa and all
>
> In my opinion, the Content Hub back in Stanbol for Semantic Search
> capabilities is a great use case to be implemented.
> Waiting for Florent's opinion, I could start first only with Solr (whose
> component already exists in Camel but it needs to be adapted like the
> ActiveMQ component) and creating a custom transformer bean for Camel to
> have the original Content Hub. After that, we could think to create the
> SIREn component and the new transformer for it, giving the users the
> possibility of use one of them.
>
> What do you think? Is It an interesting use case for the Camel integration
> application?
>
> Regards
>
>
> On Mon, Jul 7, 2014 at 4:27 PM, Rafa Haro <rh...@apache.org> wrote:
>
>> Hi guys,
>>
>> El 01/07/14 10:20, Antonio David Perez Morales escribió:
>>
>> Hi all
>>>
>>> Continuing with the project, I have managed successfully the integration
>>> of
>>> activemq camel component (and also jms) into the Stanbol Camel
>>> integration.
>>> This has been a hard task due to the dependencies needed by the component
>>> and also due to the fact that we had to provide an activemq component
>>> configurable through Felix web console.
>>>
>>> With this addition, we are in the position to integrate business logic
>>> into
>>> Stanbol routes through a message service provided by activemq (jms).
>>>
>> Nice Antonio, let's see is someone has an interesting use case to
>> implement in this context.
>>
>>
>>> As a first test, I have deployed a route which consumes messages
>>> (content)
>>> from an activemq queue, enhance them using the default chain and then
>>> write
>>> the result into a file. It's a simple test but it works quite well. In
>>> this
>>> case, Stanbol is working in a standalone mode, that is to say, we don't
>>> have to explicitly call Stanbol to enhance content but Stanbol is
>>> triggered
>>> based on some external events (a new queue message)
>>>
>>> As indicated in the previous mail, I still have some pending things to be
>>> done (because I couldn't do them last week) but in order to go forward
>>> with
>>> the project I ask you for some interesting use cases where to apply the
>>> new
>>> workflow component in order to give added value to it and also in order
>>> to
>>> develop and provide more workflow (camel) components useful for those and
>>> other use cases.
>>>
>> Awaiting for the community feedback and also for Florent's opinion
>> regarding the rest of the project, as I have expressed in recent emails,
>> I'm eager to see the Content Hub back in Stanbol. And this is because of,
>> from the point of view of the use of Stanbol in the enterprise, Semantic
>> Search is one of the most common use cases. So, to have an enterprise
>> search backend as the last component of a processing route in any
>> architecture where stanbol could be plugged sounds key for me. In recent
>> discussions at the Stanbol IRC channel, we have been analysing Siren (
>> https://github.com/rdelbru/SIREn), a Lucene/Solr extension which major
>> advantage is the possibility to index tree structures, allowing then to
>> index structured data without losing full text search capabilities. To
>> refactor old ContentHub component to use Siren is out of scope of this
>> project but, in my opinion, an interesting use case could be to develop a
>> Siren Camel Component and a transformer from ContentItem to Siren Object or
>> whatever and integrate both in Stanbol.
>>
>> What do you guys think?
>>
>> Cheers,
>> Rafa
>>
>>
>>
>>
>>> Regards
>>>
>>>
>>> On Mon, Jun 23, 2014 at 6:16 PM, Antonio David Perez Morales <
>>> aperez@zaizi.com> wrote:
>>>
>>> Hi Stanbolers
>>>>
>>>> The GSoC 2014 midterm is here and I want to give you a summary of the
>>>> work
>>>> already done so far:
>>>>
>>>> - Adapted previous Camel integration PoC done by Florent into Stanbol
>>>> 1.0
>>>> version.
>>>> - Improved EngineComponent used by Camel to execute Enhancement Engines
>>>> (configured through Stanbol web console as usual) using the engine://
>>>> uri
>>>> scheme in routes.
>>>> - Created ChainComponent used by Camel to execute Enhancement Chains
>>>> using
>>>> the chain:// uri scheme in routes (both Camel components are provided as
>>>> OSGI components, so the uri scheme can be changed through the Stanbol
>>>> web
>>>> console)
>>>> - Created a custom artifact for Apache Felix Fileinstall in order to be
>>>> able to install routes defined in Camel Spring XML DSL placing a route
>>>> file
>>>> (with 'route' extension) in the stanbol/fileinstall directory
>>>> - Created a custom archetype to ease the development of bundles
>>>> containing
>>>> route definitions in Java DSL. The archetype generates a class extending
>>>> 'RouteBuilder' which creates a default Camel direct endpoint used by
>>>> other
>>>> Stanbol Workflow components to execute the route.
>>>> - Created a first version of Workflow API, which contains different OSGI
>>>> components which allow registering Camel components/routes,
>>>> start/stop/execute routes, add/remove components used in routes, etc.
>>>> - REST endpoint is provided to test the execution of routes using REST
>>>> requests (/flow/{routeId} )
>>>> - Modified the PoC full launcher to use all the new bundles to support
>>>> the
>>>> workflow feature.
>>>> - Installed JBoss developer studio which comes with Camel support in
>>>> order
>>>> to create routes in a visual way with the possibility to be exported as
>>>> Spring XML DSL format
>>>>
>>>> Some pending things I will try to do during this week:
>>>> - Improve the web package to create the needed endpoints to query the
>>>> registered routes, registered camel components, etc
>>>> - Improve the web package to remove classes copied from Stanbol jersey
>>>> module used for testing
>>>> - Update README.md files in the repository with all the new information
>>>> - Document the installation and configuration of JBoss developer studio
>>>> for Camel routes creation
>>>> - Create all the JIRA issued related to the work already done
>>>>
>>>>
>>>> For the second part of the project, I would like to read some comments
>>>> about interesting use cases in order to develop the needed Stanbol and
>>>> Camel components to support them.
>>>>
>>>> If you have any comment, please drop some lines in order to discuss the
>>>> new things to be done.
>>>>
>>>> Regards
>>>>
>>>>
>>>>
>>>> On Sat, Jun 14, 2014 at 3:39 PM, Antonio David Perez Morales <
>>>> aperez@zaizi.com> wrote:
>>>>
>>>> Hi guys
>>>>>
>>>>> Continuing with the project, and as part of the refactoring/new
>>>>> architecture I have started to modify some workflow components in
>>>>> order to
>>>>> create a better API and architecture based on OSGI components. As a
>>>>> first
>>>>> step and in order to have the same behavior than the current one
>>>>> (regarding
>>>>> enhancement process), a chain component has been created to simulate
>>>>> the
>>>>> chain behaviour. This new component uses internally the ChainManager
>>>>> and
>>>>> EnhancementJobManager component to perform the business logic. This
>>>>> way, a
>>>>> new protocol 'chain' can be used in the routes deployed in Stanbol. The
>>>>> chains are configured in the same way, using Stanbol admin console.
>>>>>
>>>>> Now, we can combine single engine executions with chains executions in
>>>>> routes deployed in Stanbol using the alternatives described in previous
>>>>> mails and in the issue [1]. Both engines and chains are configured
>>>>> through
>>>>> Stanbol admin console. You can see the refactoring advances in [2] (a
>>>>> branch used for refactoring the current PoC of Workflow in Stanbol
>>>>> 1.0). Of
>>>>> course, the Camel EIP and other Camel components can be used in the
>>>>> deployed routes as well.
>>>>>
>>>>> With the new Camel routes support, we can have a Stanbol running and
>>>>> enhancing content without receiving any HTTP request to start the
>>>>> enhancement process, because the routes can be triggered by external
>>>>> events
>>>>> ocurred in a queue, database, etc. Moreover the semantic lifting
>>>>> process
>>>>> can be splitted and merged with some application steps, so the issue
>>>>> [3]
>>>>> requesting asynchronous call support for enhancement could be solved.
>>>>>
>>>>> Anyway, if some of you have any suggestions for new components to be
>>>>> deployed for the second part of the project, or another kind of
>>>>> suggestion,
>>>>> please drop here some lines to continue with the discussion.
>>>>>
>>>>> Regards
>>>>>
>>>>> [1] https://issues.apache.org/jira/browse/STANBOL-1348
>>>>> [2]
>>>>> https://github.com/adperezmorales/stanbol-camel-
>>>>> workflow/tree/refactoring
>>>>> [3] https://issues.apache.org/jira/browse/STANBOL-263
>>>>>
>>>>>
>>>>> On Wed, Jun 11, 2014 at 10:01 AM, Antonio David Perez Morales <
>>>>> aperez@zaizi.com> wrote:
>>>>>
>>>>> Hi people
>>>>>>
>>>>>> As part of the GSoC project for the midterm and according to the issue
>>>>>> [1], a custom Apache Felix Fileinstall artifact has been created in
>>>>>> order
>>>>>> to deploy Camel routes defined in XML (Spring DSL) placing a file with
>>>>>> .route extension in a configured directory (like stanbol/fileinstall
>>>>>> directory). Moreover since this artifact depends on Fileinstall
>>>>>> bundle, the
>>>>>> created launcher has been modified to have that bundle in the OSGI
>>>>>> context
>>>>>> by default.
>>>>>>
>>>>>> So, once the current Camel integration POC has been integrated in
>>>>>> Stanbol 1.0 and extended to support the deployment of routes defined
>>>>>> by
>>>>>> Java DSL (through bundles) and XML (route files), the next step will
>>>>>> be
>>>>>> thinking and redesigning the current architecture trying to avoid the
>>>>>> duplicated code and providing a more extendable and easy to use
>>>>>> Workflow
>>>>>> API, because with the current integration only direct routes can be
>>>>>> triggered using REST API which means that the defined routes must be
>>>>>> configured properly using a direct endpoint consumer. Anyway, routes
>>>>>> starting in some other way like timers are triggered directly in the
>>>>>> deployment, so this has to be taken into account for the new API (and
>>>>>> REST
>>>>>> API).
>>>>>>
>>>>>> In parallel and for the second part, new Stanbol Camel components will
>>>>>> be developed in order to be used in new routes. So if any of you have
>>>>>> use
>>>>>> cases for this involving Stanbol components, please drop some lines
>>>>>> here in
>>>>>> order to prioritize the Stanbol Camel components to be developed.
>>>>>>
>>>>>> Comments and suggestions are more than welcome
>>>>>>
>>>>>> Regards
>>>>>>
>>>>>> [1] https://issues.apache.org/jira/browse/STANBOL-1348
>>>>>> [2] https://github.com/adperezmorales/stanbol-camel-workflow/
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Mon, Jun 2, 2014 at 7:00 PM, Antonio David Perez Morales <
>>>>>> aperez@zaizi.com> wrote:
>>>>>>
>>>>>> Hi stanbolers
>>>>>>>
>>>>>>> As part of the issue [1] , I have created a maven archetype useful to
>>>>>>> generate Camel routes in Java DSL.
>>>>>>> The archetype generates a Java project with all the dependencies and
>>>>>>> one Java class with a method which has to be filled. In this method,
>>>>>>> Camel
>>>>>>> Java DSL syntax is used to create the route.
>>>>>>> By default and as a first approach, the class will use the route name
>>>>>>> given during the project creation to enable a Camel direct endpoint
>>>>>>> with
>>>>>>> such name.
>>>>>>> The code of the first archetype version can be found at [2].
>>>>>>>
>>>>>>> The next task will be providing a Felix custom artifact to be able to
>>>>>>> deploy XML-based routes in Stanbol, placing a custom file in the
>>>>>>> Stanbol
>>>>>>> datafiles directory.
>>>>>>> After that, it will be time to think and redesign the architecture to
>>>>>>> integrate Camel workflows inside Stanbol in a better way, more
>>>>>>> configurable
>>>>>>> and extendable.
>>>>>>>
>>>>>>> Comments and suggestions are more than welcome
>>>>>>>
>>>>>>> Regards
>>>>>>>
>>>>>>> [1] https://issues.apache.org/jira/browse/STANBOL-1348
>>>>>>> [2] https://github.com/adperezmorales/stanbol-camel-workflow/
>>>>>>>
>>>>>>>
>>>>>>> On Fri, May 30, 2014 at 8:03 PM, Antonio David Perez Morales <
>>>>>>> aperez@zaizi.com> wrote:
>>>>>>>
>>>>>>> Hi all
>>>>>>>>
>>>>>>>> After a hard fight this week, I managed to get it work the Florent's
>>>>>>>> proof of concept code in the Stanbol 1.0 branch [1]
>>>>>>>> The code is uploaded in my github account [3]. As I said in a
>>>>>>>> previous
>>>>>>>> mail, I prefer to do it separately and after the project, uploading
>>>>>>>> the
>>>>>>>> developed code into a Stanbol branch.
>>>>>>>>
>>>>>>>> The 1.0.0 version has some changes in how the Jersey endpoints are
>>>>>>>> registered and also new classes and packages, so it was not a
>>>>>>>> trivial task
>>>>>>>> to make work the current proof of concept. Moreover I don't like to
>>>>>>>> simply
>>>>>>>> copy and paste code and make the needed changes. I always want to
>>>>>>>> understand how the things work and how they are developed in order
>>>>>>>> to be
>>>>>>>> able to change/modify them or develop new code around them.
>>>>>>>>
>>>>>>>> The steps done to achieve it have been the following:
>>>>>>>> - Updated pom files to the Stanbol 1.0.0-SNAPSHOT version
>>>>>>>> - Updated bundle levels in bundlelist package to fit the Stanbol 1.0
>>>>>>>> version levels
>>>>>>>> - Adapted cameljobmanager package code to Stanbol 1.0.0-SNAPSHOT
>>>>>>>> classes and using Java OSGI annotations instead of SCR annotations
>>>>>>>> in
>>>>>>>> Javadoc
>>>>>>>> - Updated flow web package to Stanbol 1.0.0-SNAPSHOT classes and
>>>>>>>> modified needed resources
>>>>>>>> - Added Java OSGI annotations to the route (WeightedChain) instead
>>>>>>>> of
>>>>>>>> SCR annotations in javadoc
>>>>>>>> - Updated launcher to use the 1.0.0-SNAPSHOT packages and needed
>>>>>>>> bundles
>>>>>>>>
>>>>>>>> So now, the http://localhost:8080/flow endpoint will use the only
>>>>>>>> Camel route (defined by WeightedChain) to call all the registered
>>>>>>>> Enhancement Engines (ordered by EnhancementEngine order property).
>>>>>>>> For testing purposes, the /flow/{flowName} has been removed, because
>>>>>>>> all this code needs to be re-designed and re-implemented so I only
>>>>>>>> wanted
>>>>>>>> to make it work to have a first (simple) integration in Stanbol
>>>>>>>> 1.0. This
>>>>>>>> functionality will be added again to trigger custom routes once the
>>>>>>>> next
>>>>>>>> step (defined below) is developed.
>>>>>>>>
>>>>>>>> The next step [2] will be support to write and configure routes in
>>>>>>>> XML
>>>>>>>> format, putting the file in datafiles in order to be loaded by a
>>>>>>>> Felix
>>>>>>>> custom artifact (as Rupert pointed out in a previous mail) and
>>>>>>>> create a
>>>>>>>> Maven archetype to create bundles defining routes which will be
>>>>>>>> loaded
>>>>>>>> using the Felix bundle tab. If necessary, as we talked in previous
>>>>>>>> messages, a REST endpoint receiving routes in XML can be developed
>>>>>>>> as an
>>>>>>>> alternative to the first approach. This is my objective for the
>>>>>>>> midterm.
>>>>>>>>
>>>>>>>> After the midterm, the new Stanbol components for Apache Camel will
>>>>>>>> be
>>>>>>>> developed and also the new architecture for Camel in Stanbol.
>>>>>>>>
>>>>>>>> Comments on this and for use cases for Stanbol Camel components are
>>>>>>>> more than welcome.
>>>>>>>>
>>>>>>>> Regards
>>>>>>>>
>>>>>>>> [1] https://issues.apache.org/jira/browse/STANBOL-1347
>>>>>>>> [2] https://issues.apache.org/jira/browse/STANBOL-1348
>>>>>>>> [3] https://github.com/adperezmorales/stanbol-camel-workflow/
>>>>>>>>
>>>>>>>>
>>>>>>>> On Tue, May 27, 2014 at 6:18 PM, Antonio David Perez Morales <
>>>>>>>> aperez@zaizi.com> wrote:
>>>>>>>>
>>>>>>>> Hi people
>>>>>>>>>
>>>>>>>>> I have already started to work on [1] to integrate current
>>>>>>>>> Florent's
>>>>>>>>> code into Stanbol 1.0.
>>>>>>>>> As a first approach, only changing the dependency versions to new
>>>>>>>>> Stanbol 1.0, many issues have arisen:
>>>>>>>>> - Deprecated use of classes
>>>>>>>>> - Classes which have changed from package
>>>>>>>>> - Some classes not necessary now
>>>>>>>>> - Classes not used which were causing conflicts
>>>>>>>>> - ...
>>>>>>>>>
>>>>>>>>> So now I'm trying to resolve all these problems to replicate the
>>>>>>>>> same
>>>>>>>>> behavior from 0.9 into 1.0. I will upload the code to a Github
>>>>>>>>> repository
>>>>>>>>> in my account (which will be pushed later into a Stanbol branch
>>>>>>>>> after the
>>>>>>>>> project) in order to track the advances.
>>>>>>>>> Once I can resolve all these problems, I will take a look to the
>>>>>>>>> Felix Custom Artifacts poiinted out by Rupert in a previous
>>>>>>>>> message to find
>>>>>>>>> out the best way to deploy (and manage) route configurations (felix
>>>>>>>>> artifacts, watchservice java, rest endpoint to receive xml routes,
>>>>>>>>> etc).
>>>>>>>>>
>>>>>>>>> Comments on this and future tasks are more than welcome.
>>>>>>>>>
>>>>>>>>> Regards
>>>>>>>>>
>>>>>>>>> [1] https://issues.apache.org/jira/browse/STANBOL-1347
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Tue, May 27, 2014 at 9:53 AM, Rafa Haro <rh...@apache.org>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>> Hi Rupert, Florent and Antonio
>>>>>>>>>>
>>>>>>>>>> El 27/05/14 08:51, Rupert Westenthaler escribió:
>>>>>>>>>>
>>>>>>>>>> As the result of Enhancement Routes is content + metadata I can
>>>>>>>>>> not
>>>>>>>>>>
>>>>>>>>>>> see what you want to "store" in the Entityhub that is about
>>>>>>>>>>> managing
>>>>>>>>>>> Entities.
>>>>>>>>>>>
>>>>>>>>>>> > - entityhub: To query/update the entityhub component
>>>>>>>>>>> Maybe. If you can come up with a good use case ^^
>>>>>>>>>>>
>>>>>>>>>>> > - contenthub: To develop a new content-hub using
>>>>>>>>>>> chain/engine
>>>>>>>>>>>
>>>>>>>>>>>> components
>>>>>>>>>>>>
>>>>>>>>>>>>> and solr/elasticsearch/whatever component (solr and
>>>>>>>>>>>>> elasticsearch
>>>>>>>>>>>>>
>>>>>>>>>>>> component
>>>>>>>>>>>>
>>>>>>>>>>>>> already exist in Camel)
>>>>>>>>>>>>>
>>>>>>>>>>>> IMO implementing a new Contenthub like component is outside the
>>>>>>>>>>> scope
>>>>>>>>>>> of this GSoC project. However If there is already
>>>>>>>>>>> Solr/Elasticsearch
>>>>>>>>>>> component it would be a really useful thing
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Regarding this, in my opinion, the use case of an eventual
>>>>>>>>>> integration with a Content hub is probably one of the most clear
>>>>>>>>>> for this
>>>>>>>>>> project. I'm not sure if that is what Antonio was trying to
>>>>>>>>>> explain but,
>>>>>>>>>> with a single route using as last endpoint Solr or any other
>>>>>>>>>> backend
>>>>>>>>>> system, we would be almost cloning the same functionality than
>>>>>>>>>> the previous
>>>>>>>>>> ContentHub implementation (Stanbol 0.12). Entities could be
>>>>>>>>>> dereferenced
>>>>>>>>>> using the EntityHub before storing the content along with the
>>>>>>>>>> metadata,
>>>>>>>>>> which is the point of integration of the EntityHub in such use
>>>>>>>>>> case. And
>>>>>>>>>> even most interesting, now with the integration of Marmotta
>>>>>>>>>> contributed by
>>>>>>>>>> Rupert, it would be possible to use a whole graph for
>>>>>>>>>> dereferencing, so
>>>>>>>>>> "simply" routing components like Enhancer->Marmotta->Solr sounds
>>>>>>>>>> to me like
>>>>>>>>>> an interesting use case.
>>>>>>>>>>
>>>>>>>>>> wdyt?
>>>>>>>>>>
>>>>>>>>>> Cheers,
>>>>>>>>>> Rafa
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>
>
--
------------------------------
This message should be regarded as confidential. If you have received this
email in error please notify the sender and destroy it immediately.
Statements of intent shall only become binding when confirmed in hard copy
by an authorised signatory.
Zaizi Ltd is registered in England and Wales with the registration number
6440931. The Registered Office is Brook House, 229 Shepherds Bush Road,
London W6 7AN.
Re: Camel integration (was : Re: Community bonding period started)
Posted by Antonio David Perez Morales <ap...@zaizi.com>.
Hi Rafa and all
In my opinion, the Content Hub back in Stanbol for Semantic Search
capabilities is a great use case to be implemented.
Waiting for Florent's opinion, I could start first only with Solr (whose
component already exists in Camel but it needs to be adapted like the
ActiveMQ component) and creating a custom transformer bean for Camel to
have the original Content Hub. After that, we could think to create the
SIREn component and the new transformer for it, giving the users the
possibility of use one of them.
What do you think? Is It an interesting use case for the Camel integration
application?
Regards
On Mon, Jul 7, 2014 at 4:27 PM, Rafa Haro <rh...@apache.org> wrote:
> Hi guys,
>
> El 01/07/14 10:20, Antonio David Perez Morales escribió:
>
> Hi all
>>
>> Continuing with the project, I have managed successfully the integration
>> of
>> activemq camel component (and also jms) into the Stanbol Camel
>> integration.
>> This has been a hard task due to the dependencies needed by the component
>> and also due to the fact that we had to provide an activemq component
>> configurable through Felix web console.
>>
>> With this addition, we are in the position to integrate business logic
>> into
>> Stanbol routes through a message service provided by activemq (jms).
>>
> Nice Antonio, let's see is someone has an interesting use case to
> implement in this context.
>
>
>> As a first test, I have deployed a route which consumes messages (content)
>> from an activemq queue, enhance them using the default chain and then
>> write
>> the result into a file. It's a simple test but it works quite well. In
>> this
>> case, Stanbol is working in a standalone mode, that is to say, we don't
>> have to explicitly call Stanbol to enhance content but Stanbol is
>> triggered
>> based on some external events (a new queue message)
>>
>> As indicated in the previous mail, I still have some pending things to be
>> done (because I couldn't do them last week) but in order to go forward
>> with
>> the project I ask you for some interesting use cases where to apply the
>> new
>> workflow component in order to give added value to it and also in order to
>> develop and provide more workflow (camel) components useful for those and
>> other use cases.
>>
> Awaiting for the community feedback and also for Florent's opinion
> regarding the rest of the project, as I have expressed in recent emails,
> I'm eager to see the Content Hub back in Stanbol. And this is because of,
> from the point of view of the use of Stanbol in the enterprise, Semantic
> Search is one of the most common use cases. So, to have an enterprise
> search backend as the last component of a processing route in any
> architecture where stanbol could be plugged sounds key for me. In recent
> discussions at the Stanbol IRC channel, we have been analysing Siren (
> https://github.com/rdelbru/SIREn), a Lucene/Solr extension which major
> advantage is the possibility to index tree structures, allowing then to
> index structured data without losing full text search capabilities. To
> refactor old ContentHub component to use Siren is out of scope of this
> project but, in my opinion, an interesting use case could be to develop a
> Siren Camel Component and a transformer from ContentItem to Siren Object or
> whatever and integrate both in Stanbol.
>
> What do you guys think?
>
> Cheers,
> Rafa
>
>
>
>
>> Regards
>>
>>
>> On Mon, Jun 23, 2014 at 6:16 PM, Antonio David Perez Morales <
>> aperez@zaizi.com> wrote:
>>
>> Hi Stanbolers
>>>
>>> The GSoC 2014 midterm is here and I want to give you a summary of the
>>> work
>>> already done so far:
>>>
>>> - Adapted previous Camel integration PoC done by Florent into Stanbol 1.0
>>> version.
>>> - Improved EngineComponent used by Camel to execute Enhancement Engines
>>> (configured through Stanbol web console as usual) using the engine:// uri
>>> scheme in routes.
>>> - Created ChainComponent used by Camel to execute Enhancement Chains
>>> using
>>> the chain:// uri scheme in routes (both Camel components are provided as
>>> OSGI components, so the uri scheme can be changed through the Stanbol web
>>> console)
>>> - Created a custom artifact for Apache Felix Fileinstall in order to be
>>> able to install routes defined in Camel Spring XML DSL placing a route
>>> file
>>> (with 'route' extension) in the stanbol/fileinstall directory
>>> - Created a custom archetype to ease the development of bundles
>>> containing
>>> route definitions in Java DSL. The archetype generates a class extending
>>> 'RouteBuilder' which creates a default Camel direct endpoint used by
>>> other
>>> Stanbol Workflow components to execute the route.
>>> - Created a first version of Workflow API, which contains different OSGI
>>> components which allow registering Camel components/routes,
>>> start/stop/execute routes, add/remove components used in routes, etc.
>>> - REST endpoint is provided to test the execution of routes using REST
>>> requests (/flow/{routeId} )
>>> - Modified the PoC full launcher to use all the new bundles to support
>>> the
>>> workflow feature.
>>> - Installed JBoss developer studio which comes with Camel support in
>>> order
>>> to create routes in a visual way with the possibility to be exported as
>>> Spring XML DSL format
>>>
>>> Some pending things I will try to do during this week:
>>> - Improve the web package to create the needed endpoints to query the
>>> registered routes, registered camel components, etc
>>> - Improve the web package to remove classes copied from Stanbol jersey
>>> module used for testing
>>> - Update README.md files in the repository with all the new information
>>> - Document the installation and configuration of JBoss developer studio
>>> for Camel routes creation
>>> - Create all the JIRA issued related to the work already done
>>>
>>>
>>> For the second part of the project, I would like to read some comments
>>> about interesting use cases in order to develop the needed Stanbol and
>>> Camel components to support them.
>>>
>>> If you have any comment, please drop some lines in order to discuss the
>>> new things to be done.
>>>
>>> Regards
>>>
>>>
>>>
>>> On Sat, Jun 14, 2014 at 3:39 PM, Antonio David Perez Morales <
>>> aperez@zaizi.com> wrote:
>>>
>>> Hi guys
>>>>
>>>> Continuing with the project, and as part of the refactoring/new
>>>> architecture I have started to modify some workflow components in order
>>>> to
>>>> create a better API and architecture based on OSGI components. As a
>>>> first
>>>> step and in order to have the same behavior than the current one
>>>> (regarding
>>>> enhancement process), a chain component has been created to simulate the
>>>> chain behaviour. This new component uses internally the ChainManager and
>>>> EnhancementJobManager component to perform the business logic. This
>>>> way, a
>>>> new protocol 'chain' can be used in the routes deployed in Stanbol. The
>>>> chains are configured in the same way, using Stanbol admin console.
>>>>
>>>> Now, we can combine single engine executions with chains executions in
>>>> routes deployed in Stanbol using the alternatives described in previous
>>>> mails and in the issue [1]. Both engines and chains are configured
>>>> through
>>>> Stanbol admin console. You can see the refactoring advances in [2] (a
>>>> branch used for refactoring the current PoC of Workflow in Stanbol
>>>> 1.0). Of
>>>> course, the Camel EIP and other Camel components can be used in the
>>>> deployed routes as well.
>>>>
>>>> With the new Camel routes support, we can have a Stanbol running and
>>>> enhancing content without receiving any HTTP request to start the
>>>> enhancement process, because the routes can be triggered by external
>>>> events
>>>> ocurred in a queue, database, etc. Moreover the semantic lifting process
>>>> can be splitted and merged with some application steps, so the issue [3]
>>>> requesting asynchronous call support for enhancement could be solved.
>>>>
>>>> Anyway, if some of you have any suggestions for new components to be
>>>> deployed for the second part of the project, or another kind of
>>>> suggestion,
>>>> please drop here some lines to continue with the discussion.
>>>>
>>>> Regards
>>>>
>>>> [1] https://issues.apache.org/jira/browse/STANBOL-1348
>>>> [2]
>>>> https://github.com/adperezmorales/stanbol-camel-
>>>> workflow/tree/refactoring
>>>> [3] https://issues.apache.org/jira/browse/STANBOL-263
>>>>
>>>>
>>>> On Wed, Jun 11, 2014 at 10:01 AM, Antonio David Perez Morales <
>>>> aperez@zaizi.com> wrote:
>>>>
>>>> Hi people
>>>>>
>>>>> As part of the GSoC project for the midterm and according to the issue
>>>>> [1], a custom Apache Felix Fileinstall artifact has been created in
>>>>> order
>>>>> to deploy Camel routes defined in XML (Spring DSL) placing a file with
>>>>> .route extension in a configured directory (like stanbol/fileinstall
>>>>> directory). Moreover since this artifact depends on Fileinstall
>>>>> bundle, the
>>>>> created launcher has been modified to have that bundle in the OSGI
>>>>> context
>>>>> by default.
>>>>>
>>>>> So, once the current Camel integration POC has been integrated in
>>>>> Stanbol 1.0 and extended to support the deployment of routes defined by
>>>>> Java DSL (through bundles) and XML (route files), the next step will be
>>>>> thinking and redesigning the current architecture trying to avoid the
>>>>> duplicated code and providing a more extendable and easy to use
>>>>> Workflow
>>>>> API, because with the current integration only direct routes can be
>>>>> triggered using REST API which means that the defined routes must be
>>>>> configured properly using a direct endpoint consumer. Anyway, routes
>>>>> starting in some other way like timers are triggered directly in the
>>>>> deployment, so this has to be taken into account for the new API (and
>>>>> REST
>>>>> API).
>>>>>
>>>>> In parallel and for the second part, new Stanbol Camel components will
>>>>> be developed in order to be used in new routes. So if any of you have
>>>>> use
>>>>> cases for this involving Stanbol components, please drop some lines
>>>>> here in
>>>>> order to prioritize the Stanbol Camel components to be developed.
>>>>>
>>>>> Comments and suggestions are more than welcome
>>>>>
>>>>> Regards
>>>>>
>>>>> [1] https://issues.apache.org/jira/browse/STANBOL-1348
>>>>> [2] https://github.com/adperezmorales/stanbol-camel-workflow/
>>>>>
>>>>>
>>>>>
>>>>> On Mon, Jun 2, 2014 at 7:00 PM, Antonio David Perez Morales <
>>>>> aperez@zaizi.com> wrote:
>>>>>
>>>>> Hi stanbolers
>>>>>>
>>>>>> As part of the issue [1] , I have created a maven archetype useful to
>>>>>> generate Camel routes in Java DSL.
>>>>>> The archetype generates a Java project with all the dependencies and
>>>>>> one Java class with a method which has to be filled. In this method,
>>>>>> Camel
>>>>>> Java DSL syntax is used to create the route.
>>>>>> By default and as a first approach, the class will use the route name
>>>>>> given during the project creation to enable a Camel direct endpoint
>>>>>> with
>>>>>> such name.
>>>>>> The code of the first archetype version can be found at [2].
>>>>>>
>>>>>> The next task will be providing a Felix custom artifact to be able to
>>>>>> deploy XML-based routes in Stanbol, placing a custom file in the
>>>>>> Stanbol
>>>>>> datafiles directory.
>>>>>> After that, it will be time to think and redesign the architecture to
>>>>>> integrate Camel workflows inside Stanbol in a better way, more
>>>>>> configurable
>>>>>> and extendable.
>>>>>>
>>>>>> Comments and suggestions are more than welcome
>>>>>>
>>>>>> Regards
>>>>>>
>>>>>> [1] https://issues.apache.org/jira/browse/STANBOL-1348
>>>>>> [2] https://github.com/adperezmorales/stanbol-camel-workflow/
>>>>>>
>>>>>>
>>>>>> On Fri, May 30, 2014 at 8:03 PM, Antonio David Perez Morales <
>>>>>> aperez@zaizi.com> wrote:
>>>>>>
>>>>>> Hi all
>>>>>>>
>>>>>>> After a hard fight this week, I managed to get it work the Florent's
>>>>>>> proof of concept code in the Stanbol 1.0 branch [1]
>>>>>>> The code is uploaded in my github account [3]. As I said in a
>>>>>>> previous
>>>>>>> mail, I prefer to do it separately and after the project, uploading
>>>>>>> the
>>>>>>> developed code into a Stanbol branch.
>>>>>>>
>>>>>>> The 1.0.0 version has some changes in how the Jersey endpoints are
>>>>>>> registered and also new classes and packages, so it was not a
>>>>>>> trivial task
>>>>>>> to make work the current proof of concept. Moreover I don't like to
>>>>>>> simply
>>>>>>> copy and paste code and make the needed changes. I always want to
>>>>>>> understand how the things work and how they are developed in order
>>>>>>> to be
>>>>>>> able to change/modify them or develop new code around them.
>>>>>>>
>>>>>>> The steps done to achieve it have been the following:
>>>>>>> - Updated pom files to the Stanbol 1.0.0-SNAPSHOT version
>>>>>>> - Updated bundle levels in bundlelist package to fit the Stanbol 1.0
>>>>>>> version levels
>>>>>>> - Adapted cameljobmanager package code to Stanbol 1.0.0-SNAPSHOT
>>>>>>> classes and using Java OSGI annotations instead of SCR annotations in
>>>>>>> Javadoc
>>>>>>> - Updated flow web package to Stanbol 1.0.0-SNAPSHOT classes and
>>>>>>> modified needed resources
>>>>>>> - Added Java OSGI annotations to the route (WeightedChain) instead of
>>>>>>> SCR annotations in javadoc
>>>>>>> - Updated launcher to use the 1.0.0-SNAPSHOT packages and needed
>>>>>>> bundles
>>>>>>>
>>>>>>> So now, the http://localhost:8080/flow endpoint will use the only
>>>>>>> Camel route (defined by WeightedChain) to call all the registered
>>>>>>> Enhancement Engines (ordered by EnhancementEngine order property).
>>>>>>> For testing purposes, the /flow/{flowName} has been removed, because
>>>>>>> all this code needs to be re-designed and re-implemented so I only
>>>>>>> wanted
>>>>>>> to make it work to have a first (simple) integration in Stanbol 1.0.
>>>>>>> This
>>>>>>> functionality will be added again to trigger custom routes once the
>>>>>>> next
>>>>>>> step (defined below) is developed.
>>>>>>>
>>>>>>> The next step [2] will be support to write and configure routes in
>>>>>>> XML
>>>>>>> format, putting the file in datafiles in order to be loaded by a
>>>>>>> Felix
>>>>>>> custom artifact (as Rupert pointed out in a previous mail) and
>>>>>>> create a
>>>>>>> Maven archetype to create bundles defining routes which will be
>>>>>>> loaded
>>>>>>> using the Felix bundle tab. If necessary, as we talked in previous
>>>>>>> messages, a REST endpoint receiving routes in XML can be developed
>>>>>>> as an
>>>>>>> alternative to the first approach. This is my objective for the
>>>>>>> midterm.
>>>>>>>
>>>>>>> After the midterm, the new Stanbol components for Apache Camel will
>>>>>>> be
>>>>>>> developed and also the new architecture for Camel in Stanbol.
>>>>>>>
>>>>>>> Comments on this and for use cases for Stanbol Camel components are
>>>>>>> more than welcome.
>>>>>>>
>>>>>>> Regards
>>>>>>>
>>>>>>> [1] https://issues.apache.org/jira/browse/STANBOL-1347
>>>>>>> [2] https://issues.apache.org/jira/browse/STANBOL-1348
>>>>>>> [3] https://github.com/adperezmorales/stanbol-camel-workflow/
>>>>>>>
>>>>>>>
>>>>>>> On Tue, May 27, 2014 at 6:18 PM, Antonio David Perez Morales <
>>>>>>> aperez@zaizi.com> wrote:
>>>>>>>
>>>>>>> Hi people
>>>>>>>>
>>>>>>>> I have already started to work on [1] to integrate current Florent's
>>>>>>>> code into Stanbol 1.0.
>>>>>>>> As a first approach, only changing the dependency versions to new
>>>>>>>> Stanbol 1.0, many issues have arisen:
>>>>>>>> - Deprecated use of classes
>>>>>>>> - Classes which have changed from package
>>>>>>>> - Some classes not necessary now
>>>>>>>> - Classes not used which were causing conflicts
>>>>>>>> - ...
>>>>>>>>
>>>>>>>> So now I'm trying to resolve all these problems to replicate the
>>>>>>>> same
>>>>>>>> behavior from 0.9 into 1.0. I will upload the code to a Github
>>>>>>>> repository
>>>>>>>> in my account (which will be pushed later into a Stanbol branch
>>>>>>>> after the
>>>>>>>> project) in order to track the advances.
>>>>>>>> Once I can resolve all these problems, I will take a look to the
>>>>>>>> Felix Custom Artifacts poiinted out by Rupert in a previous message
>>>>>>>> to find
>>>>>>>> out the best way to deploy (and manage) route configurations (felix
>>>>>>>> artifacts, watchservice java, rest endpoint to receive xml routes,
>>>>>>>> etc).
>>>>>>>>
>>>>>>>> Comments on this and future tasks are more than welcome.
>>>>>>>>
>>>>>>>> Regards
>>>>>>>>
>>>>>>>> [1] https://issues.apache.org/jira/browse/STANBOL-1347
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Tue, May 27, 2014 at 9:53 AM, Rafa Haro <rh...@apache.org>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>> Hi Rupert, Florent and Antonio
>>>>>>>>>
>>>>>>>>> El 27/05/14 08:51, Rupert Westenthaler escribió:
>>>>>>>>>
>>>>>>>>> As the result of Enhancement Routes is content + metadata I can
>>>>>>>>> not
>>>>>>>>>
>>>>>>>>>> see what you want to "store" in the Entityhub that is about
>>>>>>>>>> managing
>>>>>>>>>> Entities.
>>>>>>>>>>
>>>>>>>>>> > - entityhub: To query/update the entityhub component
>>>>>>>>>> Maybe. If you can come up with a good use case ^^
>>>>>>>>>>
>>>>>>>>>> > - contenthub: To develop a new content-hub using chain/engine
>>>>>>>>>>
>>>>>>>>>>> components
>>>>>>>>>>>
>>>>>>>>>>>> and solr/elasticsearch/whatever component (solr and
>>>>>>>>>>>> elasticsearch
>>>>>>>>>>>>
>>>>>>>>>>> component
>>>>>>>>>>>
>>>>>>>>>>>> already exist in Camel)
>>>>>>>>>>>>
>>>>>>>>>>> IMO implementing a new Contenthub like component is outside the
>>>>>>>>>> scope
>>>>>>>>>> of this GSoC project. However If there is already
>>>>>>>>>> Solr/Elasticsearch
>>>>>>>>>> component it would be a really useful thing
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Regarding this, in my opinion, the use case of an eventual
>>>>>>>>> integration with a Content hub is probably one of the most clear
>>>>>>>>> for this
>>>>>>>>> project. I'm not sure if that is what Antonio was trying to
>>>>>>>>> explain but,
>>>>>>>>> with a single route using as last endpoint Solr or any other
>>>>>>>>> backend
>>>>>>>>> system, we would be almost cloning the same functionality than the
>>>>>>>>> previous
>>>>>>>>> ContentHub implementation (Stanbol 0.12). Entities could be
>>>>>>>>> dereferenced
>>>>>>>>> using the EntityHub before storing the content along with the
>>>>>>>>> metadata,
>>>>>>>>> which is the point of integration of the EntityHub in such use
>>>>>>>>> case. And
>>>>>>>>> even most interesting, now with the integration of Marmotta
>>>>>>>>> contributed by
>>>>>>>>> Rupert, it would be possible to use a whole graph for
>>>>>>>>> dereferencing, so
>>>>>>>>> "simply" routing components like Enhancer->Marmotta->Solr sounds
>>>>>>>>> to me like
>>>>>>>>> an interesting use case.
>>>>>>>>>
>>>>>>>>> wdyt?
>>>>>>>>>
>>>>>>>>> Cheers,
>>>>>>>>> Rafa
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>
--
------------------------------
This message should be regarded as confidential. If you have received this
email in error please notify the sender and destroy it immediately.
Statements of intent shall only become binding when confirmed in hard copy
by an authorised signatory.
Zaizi Ltd is registered in England and Wales with the registration number
6440931. The Registered Office is Brook House, 229 Shepherds Bush Road,
London W6 7AN.
Re: Camel integration (was : Re: Community bonding period started)
Posted by Rafa Haro <rh...@apache.org>.
Hi guys,
El 01/07/14 10:20, Antonio David Perez Morales escribió:
> Hi all
>
> Continuing with the project, I have managed successfully the integration of
> activemq camel component (and also jms) into the Stanbol Camel integration.
> This has been a hard task due to the dependencies needed by the component
> and also due to the fact that we had to provide an activemq component
> configurable through Felix web console.
>
> With this addition, we are in the position to integrate business logic into
> Stanbol routes through a message service provided by activemq (jms).
Nice Antonio, let's see is someone has an interesting use case to
implement in this context.
>
> As a first test, I have deployed a route which consumes messages (content)
> from an activemq queue, enhance them using the default chain and then write
> the result into a file. It's a simple test but it works quite well. In this
> case, Stanbol is working in a standalone mode, that is to say, we don't
> have to explicitly call Stanbol to enhance content but Stanbol is triggered
> based on some external events (a new queue message)
>
> As indicated in the previous mail, I still have some pending things to be
> done (because I couldn't do them last week) but in order to go forward with
> the project I ask you for some interesting use cases where to apply the new
> workflow component in order to give added value to it and also in order to
> develop and provide more workflow (camel) components useful for those and
> other use cases.
Awaiting for the community feedback and also for Florent's opinion
regarding the rest of the project, as I have expressed in recent emails,
I'm eager to see the Content Hub back in Stanbol. And this is because
of, from the point of view of the use of Stanbol in the enterprise,
Semantic Search is one of the most common use cases. So, to have an
enterprise search backend as the last component of a processing route in
any architecture where stanbol could be plugged sounds key for me. In
recent discussions at the Stanbol IRC channel, we have been analysing
Siren (https://github.com/rdelbru/SIREn), a Lucene/Solr extension which
major advantage is the possibility to index tree structures, allowing
then to index structured data without losing full text search
capabilities. To refactor old ContentHub component to use Siren is out
of scope of this project but, in my opinion, an interesting use case
could be to develop a Siren Camel Component and a transformer from
ContentItem to Siren Object or whatever and integrate both in Stanbol.
What do you guys think?
Cheers,
Rafa
>
> Regards
>
>
> On Mon, Jun 23, 2014 at 6:16 PM, Antonio David Perez Morales <
> aperez@zaizi.com> wrote:
>
>> Hi Stanbolers
>>
>> The GSoC 2014 midterm is here and I want to give you a summary of the work
>> already done so far:
>>
>> - Adapted previous Camel integration PoC done by Florent into Stanbol 1.0
>> version.
>> - Improved EngineComponent used by Camel to execute Enhancement Engines
>> (configured through Stanbol web console as usual) using the engine:// uri
>> scheme in routes.
>> - Created ChainComponent used by Camel to execute Enhancement Chains using
>> the chain:// uri scheme in routes (both Camel components are provided as
>> OSGI components, so the uri scheme can be changed through the Stanbol web
>> console)
>> - Created a custom artifact for Apache Felix Fileinstall in order to be
>> able to install routes defined in Camel Spring XML DSL placing a route file
>> (with 'route' extension) in the stanbol/fileinstall directory
>> - Created a custom archetype to ease the development of bundles containing
>> route definitions in Java DSL. The archetype generates a class extending
>> 'RouteBuilder' which creates a default Camel direct endpoint used by other
>> Stanbol Workflow components to execute the route.
>> - Created a first version of Workflow API, which contains different OSGI
>> components which allow registering Camel components/routes,
>> start/stop/execute routes, add/remove components used in routes, etc.
>> - REST endpoint is provided to test the execution of routes using REST
>> requests (/flow/{routeId} )
>> - Modified the PoC full launcher to use all the new bundles to support the
>> workflow feature.
>> - Installed JBoss developer studio which comes with Camel support in order
>> to create routes in a visual way with the possibility to be exported as
>> Spring XML DSL format
>>
>> Some pending things I will try to do during this week:
>> - Improve the web package to create the needed endpoints to query the
>> registered routes, registered camel components, etc
>> - Improve the web package to remove classes copied from Stanbol jersey
>> module used for testing
>> - Update README.md files in the repository with all the new information
>> - Document the installation and configuration of JBoss developer studio
>> for Camel routes creation
>> - Create all the JIRA issued related to the work already done
>>
>>
>> For the second part of the project, I would like to read some comments
>> about interesting use cases in order to develop the needed Stanbol and
>> Camel components to support them.
>>
>> If you have any comment, please drop some lines in order to discuss the
>> new things to be done.
>>
>> Regards
>>
>>
>>
>> On Sat, Jun 14, 2014 at 3:39 PM, Antonio David Perez Morales <
>> aperez@zaizi.com> wrote:
>>
>>> Hi guys
>>>
>>> Continuing with the project, and as part of the refactoring/new
>>> architecture I have started to modify some workflow components in order to
>>> create a better API and architecture based on OSGI components. As a first
>>> step and in order to have the same behavior than the current one (regarding
>>> enhancement process), a chain component has been created to simulate the
>>> chain behaviour. This new component uses internally the ChainManager and
>>> EnhancementJobManager component to perform the business logic. This way, a
>>> new protocol 'chain' can be used in the routes deployed in Stanbol. The
>>> chains are configured in the same way, using Stanbol admin console.
>>>
>>> Now, we can combine single engine executions with chains executions in
>>> routes deployed in Stanbol using the alternatives described in previous
>>> mails and in the issue [1]. Both engines and chains are configured through
>>> Stanbol admin console. You can see the refactoring advances in [2] (a
>>> branch used for refactoring the current PoC of Workflow in Stanbol 1.0). Of
>>> course, the Camel EIP and other Camel components can be used in the
>>> deployed routes as well.
>>>
>>> With the new Camel routes support, we can have a Stanbol running and
>>> enhancing content without receiving any HTTP request to start the
>>> enhancement process, because the routes can be triggered by external events
>>> ocurred in a queue, database, etc. Moreover the semantic lifting process
>>> can be splitted and merged with some application steps, so the issue [3]
>>> requesting asynchronous call support for enhancement could be solved.
>>>
>>> Anyway, if some of you have any suggestions for new components to be
>>> deployed for the second part of the project, or another kind of suggestion,
>>> please drop here some lines to continue with the discussion.
>>>
>>> Regards
>>>
>>> [1] https://issues.apache.org/jira/browse/STANBOL-1348
>>> [2]
>>> https://github.com/adperezmorales/stanbol-camel-workflow/tree/refactoring
>>> [3] https://issues.apache.org/jira/browse/STANBOL-263
>>>
>>>
>>> On Wed, Jun 11, 2014 at 10:01 AM, Antonio David Perez Morales <
>>> aperez@zaizi.com> wrote:
>>>
>>>> Hi people
>>>>
>>>> As part of the GSoC project for the midterm and according to the issue
>>>> [1], a custom Apache Felix Fileinstall artifact has been created in order
>>>> to deploy Camel routes defined in XML (Spring DSL) placing a file with
>>>> .route extension in a configured directory (like stanbol/fileinstall
>>>> directory). Moreover since this artifact depends on Fileinstall bundle, the
>>>> created launcher has been modified to have that bundle in the OSGI context
>>>> by default.
>>>>
>>>> So, once the current Camel integration POC has been integrated in
>>>> Stanbol 1.0 and extended to support the deployment of routes defined by
>>>> Java DSL (through bundles) and XML (route files), the next step will be
>>>> thinking and redesigning the current architecture trying to avoid the
>>>> duplicated code and providing a more extendable and easy to use Workflow
>>>> API, because with the current integration only direct routes can be
>>>> triggered using REST API which means that the defined routes must be
>>>> configured properly using a direct endpoint consumer. Anyway, routes
>>>> starting in some other way like timers are triggered directly in the
>>>> deployment, so this has to be taken into account for the new API (and REST
>>>> API).
>>>>
>>>> In parallel and for the second part, new Stanbol Camel components will
>>>> be developed in order to be used in new routes. So if any of you have use
>>>> cases for this involving Stanbol components, please drop some lines here in
>>>> order to prioritize the Stanbol Camel components to be developed.
>>>>
>>>> Comments and suggestions are more than welcome
>>>>
>>>> Regards
>>>>
>>>> [1] https://issues.apache.org/jira/browse/STANBOL-1348
>>>> [2] https://github.com/adperezmorales/stanbol-camel-workflow/
>>>>
>>>>
>>>>
>>>> On Mon, Jun 2, 2014 at 7:00 PM, Antonio David Perez Morales <
>>>> aperez@zaizi.com> wrote:
>>>>
>>>>> Hi stanbolers
>>>>>
>>>>> As part of the issue [1] , I have created a maven archetype useful to
>>>>> generate Camel routes in Java DSL.
>>>>> The archetype generates a Java project with all the dependencies and
>>>>> one Java class with a method which has to be filled. In this method, Camel
>>>>> Java DSL syntax is used to create the route.
>>>>> By default and as a first approach, the class will use the route name
>>>>> given during the project creation to enable a Camel direct endpoint with
>>>>> such name.
>>>>> The code of the first archetype version can be found at [2].
>>>>>
>>>>> The next task will be providing a Felix custom artifact to be able to
>>>>> deploy XML-based routes in Stanbol, placing a custom file in the Stanbol
>>>>> datafiles directory.
>>>>> After that, it will be time to think and redesign the architecture to
>>>>> integrate Camel workflows inside Stanbol in a better way, more configurable
>>>>> and extendable.
>>>>>
>>>>> Comments and suggestions are more than welcome
>>>>>
>>>>> Regards
>>>>>
>>>>> [1] https://issues.apache.org/jira/browse/STANBOL-1348
>>>>> [2] https://github.com/adperezmorales/stanbol-camel-workflow/
>>>>>
>>>>>
>>>>> On Fri, May 30, 2014 at 8:03 PM, Antonio David Perez Morales <
>>>>> aperez@zaizi.com> wrote:
>>>>>
>>>>>> Hi all
>>>>>>
>>>>>> After a hard fight this week, I managed to get it work the Florent's
>>>>>> proof of concept code in the Stanbol 1.0 branch [1]
>>>>>> The code is uploaded in my github account [3]. As I said in a previous
>>>>>> mail, I prefer to do it separately and after the project, uploading the
>>>>>> developed code into a Stanbol branch.
>>>>>>
>>>>>> The 1.0.0 version has some changes in how the Jersey endpoints are
>>>>>> registered and also new classes and packages, so it was not a trivial task
>>>>>> to make work the current proof of concept. Moreover I don't like to simply
>>>>>> copy and paste code and make the needed changes. I always want to
>>>>>> understand how the things work and how they are developed in order to be
>>>>>> able to change/modify them or develop new code around them.
>>>>>>
>>>>>> The steps done to achieve it have been the following:
>>>>>> - Updated pom files to the Stanbol 1.0.0-SNAPSHOT version
>>>>>> - Updated bundle levels in bundlelist package to fit the Stanbol 1.0
>>>>>> version levels
>>>>>> - Adapted cameljobmanager package code to Stanbol 1.0.0-SNAPSHOT
>>>>>> classes and using Java OSGI annotations instead of SCR annotations in
>>>>>> Javadoc
>>>>>> - Updated flow web package to Stanbol 1.0.0-SNAPSHOT classes and
>>>>>> modified needed resources
>>>>>> - Added Java OSGI annotations to the route (WeightedChain) instead of
>>>>>> SCR annotations in javadoc
>>>>>> - Updated launcher to use the 1.0.0-SNAPSHOT packages and needed
>>>>>> bundles
>>>>>>
>>>>>> So now, the http://localhost:8080/flow endpoint will use the only
>>>>>> Camel route (defined by WeightedChain) to call all the registered
>>>>>> Enhancement Engines (ordered by EnhancementEngine order property).
>>>>>> For testing purposes, the /flow/{flowName} has been removed, because
>>>>>> all this code needs to be re-designed and re-implemented so I only wanted
>>>>>> to make it work to have a first (simple) integration in Stanbol 1.0. This
>>>>>> functionality will be added again to trigger custom routes once the next
>>>>>> step (defined below) is developed.
>>>>>>
>>>>>> The next step [2] will be support to write and configure routes in XML
>>>>>> format, putting the file in datafiles in order to be loaded by a Felix
>>>>>> custom artifact (as Rupert pointed out in a previous mail) and create a
>>>>>> Maven archetype to create bundles defining routes which will be loaded
>>>>>> using the Felix bundle tab. If necessary, as we talked in previous
>>>>>> messages, a REST endpoint receiving routes in XML can be developed as an
>>>>>> alternative to the first approach. This is my objective for the midterm.
>>>>>>
>>>>>> After the midterm, the new Stanbol components for Apache Camel will be
>>>>>> developed and also the new architecture for Camel in Stanbol.
>>>>>>
>>>>>> Comments on this and for use cases for Stanbol Camel components are
>>>>>> more than welcome.
>>>>>>
>>>>>> Regards
>>>>>>
>>>>>> [1] https://issues.apache.org/jira/browse/STANBOL-1347
>>>>>> [2] https://issues.apache.org/jira/browse/STANBOL-1348
>>>>>> [3] https://github.com/adperezmorales/stanbol-camel-workflow/
>>>>>>
>>>>>>
>>>>>> On Tue, May 27, 2014 at 6:18 PM, Antonio David Perez Morales <
>>>>>> aperez@zaizi.com> wrote:
>>>>>>
>>>>>>> Hi people
>>>>>>>
>>>>>>> I have already started to work on [1] to integrate current Florent's
>>>>>>> code into Stanbol 1.0.
>>>>>>> As a first approach, only changing the dependency versions to new
>>>>>>> Stanbol 1.0, many issues have arisen:
>>>>>>> - Deprecated use of classes
>>>>>>> - Classes which have changed from package
>>>>>>> - Some classes not necessary now
>>>>>>> - Classes not used which were causing conflicts
>>>>>>> - ...
>>>>>>>
>>>>>>> So now I'm trying to resolve all these problems to replicate the same
>>>>>>> behavior from 0.9 into 1.0. I will upload the code to a Github repository
>>>>>>> in my account (which will be pushed later into a Stanbol branch after the
>>>>>>> project) in order to track the advances.
>>>>>>> Once I can resolve all these problems, I will take a look to the
>>>>>>> Felix Custom Artifacts poiinted out by Rupert in a previous message to find
>>>>>>> out the best way to deploy (and manage) route configurations (felix
>>>>>>> artifacts, watchservice java, rest endpoint to receive xml routes, etc).
>>>>>>>
>>>>>>> Comments on this and future tasks are more than welcome.
>>>>>>>
>>>>>>> Regards
>>>>>>>
>>>>>>> [1] https://issues.apache.org/jira/browse/STANBOL-1347
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Tue, May 27, 2014 at 9:53 AM, Rafa Haro <rh...@apache.org> wrote:
>>>>>>>
>>>>>>>> Hi Rupert, Florent and Antonio
>>>>>>>>
>>>>>>>> El 27/05/14 08:51, Rupert Westenthaler escribió:
>>>>>>>>
>>>>>>>> As the result of Enhancement Routes is content + metadata I can not
>>>>>>>>> see what you want to "store" in the Entityhub that is about managing
>>>>>>>>> Entities.
>>>>>>>>>
>>>>>>>>> > - entityhub: To query/update the entityhub component
>>>>>>>>> Maybe. If you can come up with a good use case ^^
>>>>>>>>>
>>>>>>>>> > - contenthub: To develop a new content-hub using chain/engine
>>>>>>>>>> components
>>>>>>>>>>> and solr/elasticsearch/whatever component (solr and elasticsearch
>>>>>>>>>> component
>>>>>>>>>>> already exist in Camel)
>>>>>>>>> IMO implementing a new Contenthub like component is outside the
>>>>>>>>> scope
>>>>>>>>> of this GSoC project. However If there is already Solr/Elasticsearch
>>>>>>>>> component it would be a really useful thing
>>>>>>>>>
>>>>>>>>>
>>>>>>>> Regarding this, in my opinion, the use case of an eventual
>>>>>>>> integration with a Content hub is probably one of the most clear for this
>>>>>>>> project. I'm not sure if that is what Antonio was trying to explain but,
>>>>>>>> with a single route using as last endpoint Solr or any other backend
>>>>>>>> system, we would be almost cloning the same functionality than the previous
>>>>>>>> ContentHub implementation (Stanbol 0.12). Entities could be dereferenced
>>>>>>>> using the EntityHub before storing the content along with the metadata,
>>>>>>>> which is the point of integration of the EntityHub in such use case. And
>>>>>>>> even most interesting, now with the integration of Marmotta contributed by
>>>>>>>> Rupert, it would be possible to use a whole graph for dereferencing, so
>>>>>>>> "simply" routing components like Enhancer->Marmotta->Solr sounds to me like
>>>>>>>> an interesting use case.
>>>>>>>>
>>>>>>>> wdyt?
>>>>>>>>
>>>>>>>> Cheers,
>>>>>>>> Rafa
>>>>>>>>
>>>>>>>
Re: Camel integration (was : Re: Community bonding period started)
Posted by Antonio David Perez Morales <ap...@zaizi.com>.
Hi all
Continuing with the project, I have managed successfully the integration of
activemq camel component (and also jms) into the Stanbol Camel integration.
This has been a hard task due to the dependencies needed by the component
and also due to the fact that we had to provide an activemq component
configurable through Felix web console.
With this addition, we are in the position to integrate business logic into
Stanbol routes through a message service provided by activemq (jms).
As a first test, I have deployed a route which consumes messages (content)
from an activemq queue, enhance them using the default chain and then write
the result into a file. It's a simple test but it works quite well. In this
case, Stanbol is working in a standalone mode, that is to say, we don't
have to explicitly call Stanbol to enhance content but Stanbol is triggered
based on some external events (a new queue message)
As indicated in the previous mail, I still have some pending things to be
done (because I couldn't do them last week) but in order to go forward with
the project I ask you for some interesting use cases where to apply the new
workflow component in order to give added value to it and also in order to
develop and provide more workflow (camel) components useful for those and
other use cases.
Regards
On Mon, Jun 23, 2014 at 6:16 PM, Antonio David Perez Morales <
aperez@zaizi.com> wrote:
> Hi Stanbolers
>
> The GSoC 2014 midterm is here and I want to give you a summary of the work
> already done so far:
>
> - Adapted previous Camel integration PoC done by Florent into Stanbol 1.0
> version.
> - Improved EngineComponent used by Camel to execute Enhancement Engines
> (configured through Stanbol web console as usual) using the engine:// uri
> scheme in routes.
> - Created ChainComponent used by Camel to execute Enhancement Chains using
> the chain:// uri scheme in routes (both Camel components are provided as
> OSGI components, so the uri scheme can be changed through the Stanbol web
> console)
> - Created a custom artifact for Apache Felix Fileinstall in order to be
> able to install routes defined in Camel Spring XML DSL placing a route file
> (with 'route' extension) in the stanbol/fileinstall directory
> - Created a custom archetype to ease the development of bundles containing
> route definitions in Java DSL. The archetype generates a class extending
> 'RouteBuilder' which creates a default Camel direct endpoint used by other
> Stanbol Workflow components to execute the route.
> - Created a first version of Workflow API, which contains different OSGI
> components which allow registering Camel components/routes,
> start/stop/execute routes, add/remove components used in routes, etc.
> - REST endpoint is provided to test the execution of routes using REST
> requests (/flow/{routeId} )
> - Modified the PoC full launcher to use all the new bundles to support the
> workflow feature.
> - Installed JBoss developer studio which comes with Camel support in order
> to create routes in a visual way with the possibility to be exported as
> Spring XML DSL format
>
> Some pending things I will try to do during this week:
> - Improve the web package to create the needed endpoints to query the
> registered routes, registered camel components, etc
> - Improve the web package to remove classes copied from Stanbol jersey
> module used for testing
> - Update README.md files in the repository with all the new information
> - Document the installation and configuration of JBoss developer studio
> for Camel routes creation
> - Create all the JIRA issued related to the work already done
>
>
> For the second part of the project, I would like to read some comments
> about interesting use cases in order to develop the needed Stanbol and
> Camel components to support them.
>
> If you have any comment, please drop some lines in order to discuss the
> new things to be done.
>
> Regards
>
>
>
> On Sat, Jun 14, 2014 at 3:39 PM, Antonio David Perez Morales <
> aperez@zaizi.com> wrote:
>
>> Hi guys
>>
>> Continuing with the project, and as part of the refactoring/new
>> architecture I have started to modify some workflow components in order to
>> create a better API and architecture based on OSGI components. As a first
>> step and in order to have the same behavior than the current one (regarding
>> enhancement process), a chain component has been created to simulate the
>> chain behaviour. This new component uses internally the ChainManager and
>> EnhancementJobManager component to perform the business logic. This way, a
>> new protocol 'chain' can be used in the routes deployed in Stanbol. The
>> chains are configured in the same way, using Stanbol admin console.
>>
>> Now, we can combine single engine executions with chains executions in
>> routes deployed in Stanbol using the alternatives described in previous
>> mails and in the issue [1]. Both engines and chains are configured through
>> Stanbol admin console. You can see the refactoring advances in [2] (a
>> branch used for refactoring the current PoC of Workflow in Stanbol 1.0). Of
>> course, the Camel EIP and other Camel components can be used in the
>> deployed routes as well.
>>
>> With the new Camel routes support, we can have a Stanbol running and
>> enhancing content without receiving any HTTP request to start the
>> enhancement process, because the routes can be triggered by external events
>> ocurred in a queue, database, etc. Moreover the semantic lifting process
>> can be splitted and merged with some application steps, so the issue [3]
>> requesting asynchronous call support for enhancement could be solved.
>>
>> Anyway, if some of you have any suggestions for new components to be
>> deployed for the second part of the project, or another kind of suggestion,
>> please drop here some lines to continue with the discussion.
>>
>> Regards
>>
>> [1] https://issues.apache.org/jira/browse/STANBOL-1348
>> [2]
>> https://github.com/adperezmorales/stanbol-camel-workflow/tree/refactoring
>> [3] https://issues.apache.org/jira/browse/STANBOL-263
>>
>>
>> On Wed, Jun 11, 2014 at 10:01 AM, Antonio David Perez Morales <
>> aperez@zaizi.com> wrote:
>>
>>> Hi people
>>>
>>> As part of the GSoC project for the midterm and according to the issue
>>> [1], a custom Apache Felix Fileinstall artifact has been created in order
>>> to deploy Camel routes defined in XML (Spring DSL) placing a file with
>>> .route extension in a configured directory (like stanbol/fileinstall
>>> directory). Moreover since this artifact depends on Fileinstall bundle, the
>>> created launcher has been modified to have that bundle in the OSGI context
>>> by default.
>>>
>>> So, once the current Camel integration POC has been integrated in
>>> Stanbol 1.0 and extended to support the deployment of routes defined by
>>> Java DSL (through bundles) and XML (route files), the next step will be
>>> thinking and redesigning the current architecture trying to avoid the
>>> duplicated code and providing a more extendable and easy to use Workflow
>>> API, because with the current integration only direct routes can be
>>> triggered using REST API which means that the defined routes must be
>>> configured properly using a direct endpoint consumer. Anyway, routes
>>> starting in some other way like timers are triggered directly in the
>>> deployment, so this has to be taken into account for the new API (and REST
>>> API).
>>>
>>> In parallel and for the second part, new Stanbol Camel components will
>>> be developed in order to be used in new routes. So if any of you have use
>>> cases for this involving Stanbol components, please drop some lines here in
>>> order to prioritize the Stanbol Camel components to be developed.
>>>
>>> Comments and suggestions are more than welcome
>>>
>>> Regards
>>>
>>> [1] https://issues.apache.org/jira/browse/STANBOL-1348
>>> [2] https://github.com/adperezmorales/stanbol-camel-workflow/
>>>
>>>
>>>
>>> On Mon, Jun 2, 2014 at 7:00 PM, Antonio David Perez Morales <
>>> aperez@zaizi.com> wrote:
>>>
>>>> Hi stanbolers
>>>>
>>>> As part of the issue [1] , I have created a maven archetype useful to
>>>> generate Camel routes in Java DSL.
>>>> The archetype generates a Java project with all the dependencies and
>>>> one Java class with a method which has to be filled. In this method, Camel
>>>> Java DSL syntax is used to create the route.
>>>> By default and as a first approach, the class will use the route name
>>>> given during the project creation to enable a Camel direct endpoint with
>>>> such name.
>>>> The code of the first archetype version can be found at [2].
>>>>
>>>> The next task will be providing a Felix custom artifact to be able to
>>>> deploy XML-based routes in Stanbol, placing a custom file in the Stanbol
>>>> datafiles directory.
>>>> After that, it will be time to think and redesign the architecture to
>>>> integrate Camel workflows inside Stanbol in a better way, more configurable
>>>> and extendable.
>>>>
>>>> Comments and suggestions are more than welcome
>>>>
>>>> Regards
>>>>
>>>> [1] https://issues.apache.org/jira/browse/STANBOL-1348
>>>> [2] https://github.com/adperezmorales/stanbol-camel-workflow/
>>>>
>>>>
>>>> On Fri, May 30, 2014 at 8:03 PM, Antonio David Perez Morales <
>>>> aperez@zaizi.com> wrote:
>>>>
>>>>> Hi all
>>>>>
>>>>> After a hard fight this week, I managed to get it work the Florent's
>>>>> proof of concept code in the Stanbol 1.0 branch [1]
>>>>> The code is uploaded in my github account [3]. As I said in a previous
>>>>> mail, I prefer to do it separately and after the project, uploading the
>>>>> developed code into a Stanbol branch.
>>>>>
>>>>> The 1.0.0 version has some changes in how the Jersey endpoints are
>>>>> registered and also new classes and packages, so it was not a trivial task
>>>>> to make work the current proof of concept. Moreover I don't like to simply
>>>>> copy and paste code and make the needed changes. I always want to
>>>>> understand how the things work and how they are developed in order to be
>>>>> able to change/modify them or develop new code around them.
>>>>>
>>>>> The steps done to achieve it have been the following:
>>>>> - Updated pom files to the Stanbol 1.0.0-SNAPSHOT version
>>>>> - Updated bundle levels in bundlelist package to fit the Stanbol 1.0
>>>>> version levels
>>>>> - Adapted cameljobmanager package code to Stanbol 1.0.0-SNAPSHOT
>>>>> classes and using Java OSGI annotations instead of SCR annotations in
>>>>> Javadoc
>>>>> - Updated flow web package to Stanbol 1.0.0-SNAPSHOT classes and
>>>>> modified needed resources
>>>>> - Added Java OSGI annotations to the route (WeightedChain) instead of
>>>>> SCR annotations in javadoc
>>>>> - Updated launcher to use the 1.0.0-SNAPSHOT packages and needed
>>>>> bundles
>>>>>
>>>>> So now, the http://localhost:8080/flow endpoint will use the only
>>>>> Camel route (defined by WeightedChain) to call all the registered
>>>>> Enhancement Engines (ordered by EnhancementEngine order property).
>>>>> For testing purposes, the /flow/{flowName} has been removed, because
>>>>> all this code needs to be re-designed and re-implemented so I only wanted
>>>>> to make it work to have a first (simple) integration in Stanbol 1.0. This
>>>>> functionality will be added again to trigger custom routes once the next
>>>>> step (defined below) is developed.
>>>>>
>>>>> The next step [2] will be support to write and configure routes in XML
>>>>> format, putting the file in datafiles in order to be loaded by a Felix
>>>>> custom artifact (as Rupert pointed out in a previous mail) and create a
>>>>> Maven archetype to create bundles defining routes which will be loaded
>>>>> using the Felix bundle tab. If necessary, as we talked in previous
>>>>> messages, a REST endpoint receiving routes in XML can be developed as an
>>>>> alternative to the first approach. This is my objective for the midterm.
>>>>>
>>>>> After the midterm, the new Stanbol components for Apache Camel will be
>>>>> developed and also the new architecture for Camel in Stanbol.
>>>>>
>>>>> Comments on this and for use cases for Stanbol Camel components are
>>>>> more than welcome.
>>>>>
>>>>> Regards
>>>>>
>>>>> [1] https://issues.apache.org/jira/browse/STANBOL-1347
>>>>> [2] https://issues.apache.org/jira/browse/STANBOL-1348
>>>>> [3] https://github.com/adperezmorales/stanbol-camel-workflow/
>>>>>
>>>>>
>>>>> On Tue, May 27, 2014 at 6:18 PM, Antonio David Perez Morales <
>>>>> aperez@zaizi.com> wrote:
>>>>>
>>>>>> Hi people
>>>>>>
>>>>>> I have already started to work on [1] to integrate current Florent's
>>>>>> code into Stanbol 1.0.
>>>>>> As a first approach, only changing the dependency versions to new
>>>>>> Stanbol 1.0, many issues have arisen:
>>>>>> - Deprecated use of classes
>>>>>> - Classes which have changed from package
>>>>>> - Some classes not necessary now
>>>>>> - Classes not used which were causing conflicts
>>>>>> - ...
>>>>>>
>>>>>> So now I'm trying to resolve all these problems to replicate the same
>>>>>> behavior from 0.9 into 1.0. I will upload the code to a Github repository
>>>>>> in my account (which will be pushed later into a Stanbol branch after the
>>>>>> project) in order to track the advances.
>>>>>> Once I can resolve all these problems, I will take a look to the
>>>>>> Felix Custom Artifacts poiinted out by Rupert in a previous message to find
>>>>>> out the best way to deploy (and manage) route configurations (felix
>>>>>> artifacts, watchservice java, rest endpoint to receive xml routes, etc).
>>>>>>
>>>>>> Comments on this and future tasks are more than welcome.
>>>>>>
>>>>>> Regards
>>>>>>
>>>>>> [1] https://issues.apache.org/jira/browse/STANBOL-1347
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Tue, May 27, 2014 at 9:53 AM, Rafa Haro <rh...@apache.org> wrote:
>>>>>>
>>>>>>> Hi Rupert, Florent and Antonio
>>>>>>>
>>>>>>> El 27/05/14 08:51, Rupert Westenthaler escribió:
>>>>>>>
>>>>>>> As the result of Enhancement Routes is content + metadata I can not
>>>>>>>> see what you want to "store" in the Entityhub that is about managing
>>>>>>>> Entities.
>>>>>>>>
>>>>>>>> > - entityhub: To query/update the entityhub component
>>>>>>>>>
>>>>>>>> Maybe. If you can come up with a good use case ^^
>>>>>>>>
>>>>>>>> > - contenthub: To develop a new content-hub using chain/engine
>>>>>>>>> components
>>>>>>>>> >and solr/elasticsearch/whatever component (solr and elasticsearch
>>>>>>>>> component
>>>>>>>>> >already exist in Camel)
>>>>>>>>>
>>>>>>>> IMO implementing a new Contenthub like component is outside the
>>>>>>>> scope
>>>>>>>> of this GSoC project. However If there is already Solr/Elasticsearch
>>>>>>>> component it would be a really useful thing
>>>>>>>>
>>>>>>>>
>>>>>>> Regarding this, in my opinion, the use case of an eventual
>>>>>>> integration with a Content hub is probably one of the most clear for this
>>>>>>> project. I'm not sure if that is what Antonio was trying to explain but,
>>>>>>> with a single route using as last endpoint Solr or any other backend
>>>>>>> system, we would be almost cloning the same functionality than the previous
>>>>>>> ContentHub implementation (Stanbol 0.12). Entities could be dereferenced
>>>>>>> using the EntityHub before storing the content along with the metadata,
>>>>>>> which is the point of integration of the EntityHub in such use case. And
>>>>>>> even most interesting, now with the integration of Marmotta contributed by
>>>>>>> Rupert, it would be possible to use a whole graph for dereferencing, so
>>>>>>> "simply" routing components like Enhancer->Marmotta->Solr sounds to me like
>>>>>>> an interesting use case.
>>>>>>>
>>>>>>> wdyt?
>>>>>>>
>>>>>>> Cheers,
>>>>>>> Rafa
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
--
------------------------------
This message should be regarded as confidential. If you have received this
email in error please notify the sender and destroy it immediately.
Statements of intent shall only become binding when confirmed in hard copy
by an authorised signatory.
Zaizi Ltd is registered in England and Wales with the registration number
6440931. The Registered Office is Brook House, 229 Shepherds Bush Road,
London W6 7AN.
Re: Camel integration (was : Re: Community bonding period started)
Posted by Antonio David Perez Morales <ap...@zaizi.com>.
Hi Stanbolers
The GSoC 2014 midterm is here and I want to give you a summary of the work
already done so far:
- Adapted previous Camel integration PoC done by Florent into Stanbol 1.0
version.
- Improved EngineComponent used by Camel to execute Enhancement Engines
(configured through Stanbol web console as usual) using the engine:// uri
scheme in routes.
- Created ChainComponent used by Camel to execute Enhancement Chains using
the chain:// uri scheme in routes (both Camel components are provided as
OSGI components, so the uri scheme can be changed through the Stanbol web
console)
- Created a custom artifact for Apache Felix Fileinstall in order to be
able to install routes defined in Camel Spring XML DSL placing a route file
(with 'route' extension) in the stanbol/fileinstall directory
- Created a custom archetype to ease the development of bundles containing
route definitions in Java DSL. The archetype generates a class extending
'RouteBuilder' which creates a default Camel direct endpoint used by other
Stanbol Workflow components to execute the route.
- Created a first version of Workflow API, which contains different OSGI
components which allow registering Camel components/routes,
start/stop/execute routes, add/remove components used in routes, etc.
- REST endpoint is provided to test the execution of routes using REST
requests (/flow/{routeId} )
- Modified the PoC full launcher to use all the new bundles to support the
workflow feature.
- Installed JBoss developer studio which comes with Camel support in order
to create routes in a visual way with the possibility to be exported as
Spring XML DSL format
Some pending things I will try to do during this week:
- Improve the web package to create the needed endpoints to query the
registered routes, registered camel components, etc
- Improve the web package to remove classes copied from Stanbol jersey
module used for testing
- Update README.md files in the repository with all the new information
- Document the installation and configuration of JBoss developer studio for
Camel routes creation
- Create all the JIRA issued related to the work already done
For the second part of the project, I would like to read some comments
about interesting use cases in order to develop the needed Stanbol and
Camel components to support them.
If you have any comment, please drop some lines in order to discuss the new
things to be done.
Regards
On Sat, Jun 14, 2014 at 3:39 PM, Antonio David Perez Morales <
aperez@zaizi.com> wrote:
> Hi guys
>
> Continuing with the project, and as part of the refactoring/new
> architecture I have started to modify some workflow components in order to
> create a better API and architecture based on OSGI components. As a first
> step and in order to have the same behavior than the current one (regarding
> enhancement process), a chain component has been created to simulate the
> chain behaviour. This new component uses internally the ChainManager and
> EnhancementJobManager component to perform the business logic. This way, a
> new protocol 'chain' can be used in the routes deployed in Stanbol. The
> chains are configured in the same way, using Stanbol admin console.
>
> Now, we can combine single engine executions with chains executions in
> routes deployed in Stanbol using the alternatives described in previous
> mails and in the issue [1]. Both engines and chains are configured through
> Stanbol admin console. You can see the refactoring advances in [2] (a
> branch used for refactoring the current PoC of Workflow in Stanbol 1.0). Of
> course, the Camel EIP and other Camel components can be used in the
> deployed routes as well.
>
> With the new Camel routes support, we can have a Stanbol running and
> enhancing content without receiving any HTTP request to start the
> enhancement process, because the routes can be triggered by external events
> ocurred in a queue, database, etc. Moreover the semantic lifting process
> can be splitted and merged with some application steps, so the issue [3]
> requesting asynchronous call support for enhancement could be solved.
>
> Anyway, if some of you have any suggestions for new components to be
> deployed for the second part of the project, or another kind of suggestion,
> please drop here some lines to continue with the discussion.
>
> Regards
>
> [1] https://issues.apache.org/jira/browse/STANBOL-1348
> [2]
> https://github.com/adperezmorales/stanbol-camel-workflow/tree/refactoring
> [3] https://issues.apache.org/jira/browse/STANBOL-263
>
>
> On Wed, Jun 11, 2014 at 10:01 AM, Antonio David Perez Morales <
> aperez@zaizi.com> wrote:
>
>> Hi people
>>
>> As part of the GSoC project for the midterm and according to the issue
>> [1], a custom Apache Felix Fileinstall artifact has been created in order
>> to deploy Camel routes defined in XML (Spring DSL) placing a file with
>> .route extension in a configured directory (like stanbol/fileinstall
>> directory). Moreover since this artifact depends on Fileinstall bundle, the
>> created launcher has been modified to have that bundle in the OSGI context
>> by default.
>>
>> So, once the current Camel integration POC has been integrated in Stanbol
>> 1.0 and extended to support the deployment of routes defined by Java DSL
>> (through bundles) and XML (route files), the next step will be thinking and
>> redesigning the current architecture trying to avoid the duplicated code
>> and providing a more extendable and easy to use Workflow API, because with
>> the current integration only direct routes can be triggered using REST API
>> which means that the defined routes must be configured properly using a
>> direct endpoint consumer. Anyway, routes starting in some other way like
>> timers are triggered directly in the deployment, so this has to be taken
>> into account for the new API (and REST API).
>>
>> In parallel and for the second part, new Stanbol Camel components will be
>> developed in order to be used in new routes. So if any of you have use
>> cases for this involving Stanbol components, please drop some lines here in
>> order to prioritize the Stanbol Camel components to be developed.
>>
>> Comments and suggestions are more than welcome
>>
>> Regards
>>
>> [1] https://issues.apache.org/jira/browse/STANBOL-1348
>> [2] https://github.com/adperezmorales/stanbol-camel-workflow/
>>
>>
>>
>> On Mon, Jun 2, 2014 at 7:00 PM, Antonio David Perez Morales <
>> aperez@zaizi.com> wrote:
>>
>>> Hi stanbolers
>>>
>>> As part of the issue [1] , I have created a maven archetype useful to
>>> generate Camel routes in Java DSL.
>>> The archetype generates a Java project with all the dependencies and one
>>> Java class with a method which has to be filled. In this method, Camel Java
>>> DSL syntax is used to create the route.
>>> By default and as a first approach, the class will use the route name
>>> given during the project creation to enable a Camel direct endpoint with
>>> such name.
>>> The code of the first archetype version can be found at [2].
>>>
>>> The next task will be providing a Felix custom artifact to be able to
>>> deploy XML-based routes in Stanbol, placing a custom file in the Stanbol
>>> datafiles directory.
>>> After that, it will be time to think and redesign the architecture to
>>> integrate Camel workflows inside Stanbol in a better way, more configurable
>>> and extendable.
>>>
>>> Comments and suggestions are more than welcome
>>>
>>> Regards
>>>
>>> [1] https://issues.apache.org/jira/browse/STANBOL-1348
>>> [2] https://github.com/adperezmorales/stanbol-camel-workflow/
>>>
>>>
>>> On Fri, May 30, 2014 at 8:03 PM, Antonio David Perez Morales <
>>> aperez@zaizi.com> wrote:
>>>
>>>> Hi all
>>>>
>>>> After a hard fight this week, I managed to get it work the Florent's
>>>> proof of concept code in the Stanbol 1.0 branch [1]
>>>> The code is uploaded in my github account [3]. As I said in a previous
>>>> mail, I prefer to do it separately and after the project, uploading the
>>>> developed code into a Stanbol branch.
>>>>
>>>> The 1.0.0 version has some changes in how the Jersey endpoints are
>>>> registered and also new classes and packages, so it was not a trivial task
>>>> to make work the current proof of concept. Moreover I don't like to simply
>>>> copy and paste code and make the needed changes. I always want to
>>>> understand how the things work and how they are developed in order to be
>>>> able to change/modify them or develop new code around them.
>>>>
>>>> The steps done to achieve it have been the following:
>>>> - Updated pom files to the Stanbol 1.0.0-SNAPSHOT version
>>>> - Updated bundle levels in bundlelist package to fit the Stanbol 1.0
>>>> version levels
>>>> - Adapted cameljobmanager package code to Stanbol 1.0.0-SNAPSHOT
>>>> classes and using Java OSGI annotations instead of SCR annotations in
>>>> Javadoc
>>>> - Updated flow web package to Stanbol 1.0.0-SNAPSHOT classes and
>>>> modified needed resources
>>>> - Added Java OSGI annotations to the route (WeightedChain) instead of
>>>> SCR annotations in javadoc
>>>> - Updated launcher to use the 1.0.0-SNAPSHOT packages and needed bundles
>>>>
>>>> So now, the http://localhost:8080/flow endpoint will use the only
>>>> Camel route (defined by WeightedChain) to call all the registered
>>>> Enhancement Engines (ordered by EnhancementEngine order property).
>>>> For testing purposes, the /flow/{flowName} has been removed, because
>>>> all this code needs to be re-designed and re-implemented so I only wanted
>>>> to make it work to have a first (simple) integration in Stanbol 1.0. This
>>>> functionality will be added again to trigger custom routes once the next
>>>> step (defined below) is developed.
>>>>
>>>> The next step [2] will be support to write and configure routes in XML
>>>> format, putting the file in datafiles in order to be loaded by a Felix
>>>> custom artifact (as Rupert pointed out in a previous mail) and create a
>>>> Maven archetype to create bundles defining routes which will be loaded
>>>> using the Felix bundle tab. If necessary, as we talked in previous
>>>> messages, a REST endpoint receiving routes in XML can be developed as an
>>>> alternative to the first approach. This is my objective for the midterm.
>>>>
>>>> After the midterm, the new Stanbol components for Apache Camel will be
>>>> developed and also the new architecture for Camel in Stanbol.
>>>>
>>>> Comments on this and for use cases for Stanbol Camel components are
>>>> more than welcome.
>>>>
>>>> Regards
>>>>
>>>> [1] https://issues.apache.org/jira/browse/STANBOL-1347
>>>> [2] https://issues.apache.org/jira/browse/STANBOL-1348
>>>> [3] https://github.com/adperezmorales/stanbol-camel-workflow/
>>>>
>>>>
>>>> On Tue, May 27, 2014 at 6:18 PM, Antonio David Perez Morales <
>>>> aperez@zaizi.com> wrote:
>>>>
>>>>> Hi people
>>>>>
>>>>> I have already started to work on [1] to integrate current Florent's
>>>>> code into Stanbol 1.0.
>>>>> As a first approach, only changing the dependency versions to new
>>>>> Stanbol 1.0, many issues have arisen:
>>>>> - Deprecated use of classes
>>>>> - Classes which have changed from package
>>>>> - Some classes not necessary now
>>>>> - Classes not used which were causing conflicts
>>>>> - ...
>>>>>
>>>>> So now I'm trying to resolve all these problems to replicate the same
>>>>> behavior from 0.9 into 1.0. I will upload the code to a Github repository
>>>>> in my account (which will be pushed later into a Stanbol branch after the
>>>>> project) in order to track the advances.
>>>>> Once I can resolve all these problems, I will take a look to the Felix
>>>>> Custom Artifacts poiinted out by Rupert in a previous message to find out
>>>>> the best way to deploy (and manage) route configurations (felix artifacts,
>>>>> watchservice java, rest endpoint to receive xml routes, etc).
>>>>>
>>>>> Comments on this and future tasks are more than welcome.
>>>>>
>>>>> Regards
>>>>>
>>>>> [1] https://issues.apache.org/jira/browse/STANBOL-1347
>>>>>
>>>>>
>>>>>
>>>>> On Tue, May 27, 2014 at 9:53 AM, Rafa Haro <rh...@apache.org> wrote:
>>>>>
>>>>>> Hi Rupert, Florent and Antonio
>>>>>>
>>>>>> El 27/05/14 08:51, Rupert Westenthaler escribió:
>>>>>>
>>>>>> As the result of Enhancement Routes is content + metadata I can not
>>>>>>> see what you want to "store" in the Entityhub that is about managing
>>>>>>> Entities.
>>>>>>>
>>>>>>> > - entityhub: To query/update the entityhub component
>>>>>>>>
>>>>>>> Maybe. If you can come up with a good use case ^^
>>>>>>>
>>>>>>> > - contenthub: To develop a new content-hub using chain/engine
>>>>>>>> components
>>>>>>>> >and solr/elasticsearch/whatever component (solr and elasticsearch
>>>>>>>> component
>>>>>>>> >already exist in Camel)
>>>>>>>>
>>>>>>> IMO implementing a new Contenthub like component is outside the scope
>>>>>>> of this GSoC project. However If there is already Solr/Elasticsearch
>>>>>>> component it would be a really useful thing
>>>>>>>
>>>>>>>
>>>>>> Regarding this, in my opinion, the use case of an eventual
>>>>>> integration with a Content hub is probably one of the most clear for this
>>>>>> project. I'm not sure if that is what Antonio was trying to explain but,
>>>>>> with a single route using as last endpoint Solr or any other backend
>>>>>> system, we would be almost cloning the same functionality than the previous
>>>>>> ContentHub implementation (Stanbol 0.12). Entities could be dereferenced
>>>>>> using the EntityHub before storing the content along with the metadata,
>>>>>> which is the point of integration of the EntityHub in such use case. And
>>>>>> even most interesting, now with the integration of Marmotta contributed by
>>>>>> Rupert, it would be possible to use a whole graph for dereferencing, so
>>>>>> "simply" routing components like Enhancer->Marmotta->Solr sounds to me like
>>>>>> an interesting use case.
>>>>>>
>>>>>> wdyt?
>>>>>>
>>>>>> Cheers,
>>>>>> Rafa
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>
--
------------------------------
This message should be regarded as confidential. If you have received this
email in error please notify the sender and destroy it immediately.
Statements of intent shall only become binding when confirmed in hard copy
by an authorised signatory.
Zaizi Ltd is registered in England and Wales with the registration number
6440931. The Registered Office is Brook House, 229 Shepherds Bush Road,
London W6 7AN.
Re: Camel integration (was : Re: Community bonding period started)
Posted by Antonio David Perez Morales <ap...@zaizi.com>.
Hi guys
Continuing with the project, and as part of the refactoring/new
architecture I have started to modify some workflow components in order to
create a better API and architecture based on OSGI components. As a first
step and in order to have the same behavior than the current one (regarding
enhancement process), a chain component has been created to simulate the
chain behaviour. This new component uses internally the ChainManager and
EnhancementJobManager component to perform the business logic. This way, a
new protocol 'chain' can be used in the routes deployed in Stanbol. The
chains are configured in the same way, using Stanbol admin console.
Now, we can combine single engine executions with chains executions in
routes deployed in Stanbol using the alternatives described in previous
mails and in the issue [1]. Both engines and chains are configured through
Stanbol admin console. You can see the refactoring advances in [2] (a
branch used for refactoring the current PoC of Workflow in Stanbol 1.0). Of
course, the Camel EIP and other Camel components can be used in the
deployed routes as well.
With the new Camel routes support, we can have a Stanbol running and
enhancing content without receiving any HTTP request to start the
enhancement process, because the routes can be triggered by external events
ocurred in a queue, database, etc. Moreover the semantic lifting process
can be splitted and merged with some application steps, so the issue [3]
requesting asynchronous call support for enhancement could be solved.
Anyway, if some of you have any suggestions for new components to be
deployed for the second part of the project, or another kind of suggestion,
please drop here some lines to continue with the discussion.
Regards
[1] https://issues.apache.org/jira/browse/STANBOL-1348
[2]
https://github.com/adperezmorales/stanbol-camel-workflow/tree/refactoring
[3] https://issues.apache.org/jira/browse/STANBOL-263
On Wed, Jun 11, 2014 at 10:01 AM, Antonio David Perez Morales <
aperez@zaizi.com> wrote:
> Hi people
>
> As part of the GSoC project for the midterm and according to the issue
> [1], a custom Apache Felix Fileinstall artifact has been created in order
> to deploy Camel routes defined in XML (Spring DSL) placing a file with
> .route extension in a configured directory (like stanbol/fileinstall
> directory). Moreover since this artifact depends on Fileinstall bundle, the
> created launcher has been modified to have that bundle in the OSGI context
> by default.
>
> So, once the current Camel integration POC has been integrated in Stanbol
> 1.0 and extended to support the deployment of routes defined by Java DSL
> (through bundles) and XML (route files), the next step will be thinking and
> redesigning the current architecture trying to avoid the duplicated code
> and providing a more extendable and easy to use Workflow API, because with
> the current integration only direct routes can be triggered using REST API
> which means that the defined routes must be configured properly using a
> direct endpoint consumer. Anyway, routes starting in some other way like
> timers are triggered directly in the deployment, so this has to be taken
> into account for the new API (and REST API).
>
> In parallel and for the second part, new Stanbol Camel components will be
> developed in order to be used in new routes. So if any of you have use
> cases for this involving Stanbol components, please drop some lines here in
> order to prioritize the Stanbol Camel components to be developed.
>
> Comments and suggestions are more than welcome
>
> Regards
>
> [1] https://issues.apache.org/jira/browse/STANBOL-1348
> [2] https://github.com/adperezmorales/stanbol-camel-workflow/
>
>
>
> On Mon, Jun 2, 2014 at 7:00 PM, Antonio David Perez Morales <
> aperez@zaizi.com> wrote:
>
>> Hi stanbolers
>>
>> As part of the issue [1] , I have created a maven archetype useful to
>> generate Camel routes in Java DSL.
>> The archetype generates a Java project with all the dependencies and one
>> Java class with a method which has to be filled. In this method, Camel Java
>> DSL syntax is used to create the route.
>> By default and as a first approach, the class will use the route name
>> given during the project creation to enable a Camel direct endpoint with
>> such name.
>> The code of the first archetype version can be found at [2].
>>
>> The next task will be providing a Felix custom artifact to be able to
>> deploy XML-based routes in Stanbol, placing a custom file in the Stanbol
>> datafiles directory.
>> After that, it will be time to think and redesign the architecture to
>> integrate Camel workflows inside Stanbol in a better way, more configurable
>> and extendable.
>>
>> Comments and suggestions are more than welcome
>>
>> Regards
>>
>> [1] https://issues.apache.org/jira/browse/STANBOL-1348
>> [2] https://github.com/adperezmorales/stanbol-camel-workflow/
>>
>>
>> On Fri, May 30, 2014 at 8:03 PM, Antonio David Perez Morales <
>> aperez@zaizi.com> wrote:
>>
>>> Hi all
>>>
>>> After a hard fight this week, I managed to get it work the Florent's
>>> proof of concept code in the Stanbol 1.0 branch [1]
>>> The code is uploaded in my github account [3]. As I said in a previous
>>> mail, I prefer to do it separately and after the project, uploading the
>>> developed code into a Stanbol branch.
>>>
>>> The 1.0.0 version has some changes in how the Jersey endpoints are
>>> registered and also new classes and packages, so it was not a trivial task
>>> to make work the current proof of concept. Moreover I don't like to simply
>>> copy and paste code and make the needed changes. I always want to
>>> understand how the things work and how they are developed in order to be
>>> able to change/modify them or develop new code around them.
>>>
>>> The steps done to achieve it have been the following:
>>> - Updated pom files to the Stanbol 1.0.0-SNAPSHOT version
>>> - Updated bundle levels in bundlelist package to fit the Stanbol 1.0
>>> version levels
>>> - Adapted cameljobmanager package code to Stanbol 1.0.0-SNAPSHOT classes
>>> and using Java OSGI annotations instead of SCR annotations in Javadoc
>>> - Updated flow web package to Stanbol 1.0.0-SNAPSHOT classes and
>>> modified needed resources
>>> - Added Java OSGI annotations to the route (WeightedChain) instead of
>>> SCR annotations in javadoc
>>> - Updated launcher to use the 1.0.0-SNAPSHOT packages and needed bundles
>>>
>>> So now, the http://localhost:8080/flow endpoint will use the only Camel
>>> route (defined by WeightedChain) to call all the registered Enhancement
>>> Engines (ordered by EnhancementEngine order property).
>>> For testing purposes, the /flow/{flowName} has been removed, because all
>>> this code needs to be re-designed and re-implemented so I only wanted to
>>> make it work to have a first (simple) integration in Stanbol 1.0. This
>>> functionality will be added again to trigger custom routes once the next
>>> step (defined below) is developed.
>>>
>>> The next step [2] will be support to write and configure routes in XML
>>> format, putting the file in datafiles in order to be loaded by a Felix
>>> custom artifact (as Rupert pointed out in a previous mail) and create a
>>> Maven archetype to create bundles defining routes which will be loaded
>>> using the Felix bundle tab. If necessary, as we talked in previous
>>> messages, a REST endpoint receiving routes in XML can be developed as an
>>> alternative to the first approach. This is my objective for the midterm.
>>>
>>> After the midterm, the new Stanbol components for Apache Camel will be
>>> developed and also the new architecture for Camel in Stanbol.
>>>
>>> Comments on this and for use cases for Stanbol Camel components are more
>>> than welcome.
>>>
>>> Regards
>>>
>>> [1] https://issues.apache.org/jira/browse/STANBOL-1347
>>> [2] https://issues.apache.org/jira/browse/STANBOL-1348
>>> [3] https://github.com/adperezmorales/stanbol-camel-workflow/
>>>
>>>
>>> On Tue, May 27, 2014 at 6:18 PM, Antonio David Perez Morales <
>>> aperez@zaizi.com> wrote:
>>>
>>>> Hi people
>>>>
>>>> I have already started to work on [1] to integrate current Florent's
>>>> code into Stanbol 1.0.
>>>> As a first approach, only changing the dependency versions to new
>>>> Stanbol 1.0, many issues have arisen:
>>>> - Deprecated use of classes
>>>> - Classes which have changed from package
>>>> - Some classes not necessary now
>>>> - Classes not used which were causing conflicts
>>>> - ...
>>>>
>>>> So now I'm trying to resolve all these problems to replicate the same
>>>> behavior from 0.9 into 1.0. I will upload the code to a Github repository
>>>> in my account (which will be pushed later into a Stanbol branch after the
>>>> project) in order to track the advances.
>>>> Once I can resolve all these problems, I will take a look to the Felix
>>>> Custom Artifacts poiinted out by Rupert in a previous message to find out
>>>> the best way to deploy (and manage) route configurations (felix artifacts,
>>>> watchservice java, rest endpoint to receive xml routes, etc).
>>>>
>>>> Comments on this and future tasks are more than welcome.
>>>>
>>>> Regards
>>>>
>>>> [1] https://issues.apache.org/jira/browse/STANBOL-1347
>>>>
>>>>
>>>>
>>>> On Tue, May 27, 2014 at 9:53 AM, Rafa Haro <rh...@apache.org> wrote:
>>>>
>>>>> Hi Rupert, Florent and Antonio
>>>>>
>>>>> El 27/05/14 08:51, Rupert Westenthaler escribió:
>>>>>
>>>>> As the result of Enhancement Routes is content + metadata I can not
>>>>>> see what you want to "store" in the Entityhub that is about managing
>>>>>> Entities.
>>>>>>
>>>>>> > - entityhub: To query/update the entityhub component
>>>>>>>
>>>>>> Maybe. If you can come up with a good use case ^^
>>>>>>
>>>>>> > - contenthub: To develop a new content-hub using chain/engine
>>>>>>> components
>>>>>>> >and solr/elasticsearch/whatever component (solr and elasticsearch
>>>>>>> component
>>>>>>> >already exist in Camel)
>>>>>>>
>>>>>> IMO implementing a new Contenthub like component is outside the scope
>>>>>> of this GSoC project. However If there is already Solr/Elasticsearch
>>>>>> component it would be a really useful thing
>>>>>>
>>>>>>
>>>>> Regarding this, in my opinion, the use case of an eventual integration
>>>>> with a Content hub is probably one of the most clear for this project. I'm
>>>>> not sure if that is what Antonio was trying to explain but, with a single
>>>>> route using as last endpoint Solr or any other backend system, we would be
>>>>> almost cloning the same functionality than the previous ContentHub
>>>>> implementation (Stanbol 0.12). Entities could be dereferenced using the
>>>>> EntityHub before storing the content along with the metadata, which is the
>>>>> point of integration of the EntityHub in such use case. And even most
>>>>> interesting, now with the integration of Marmotta contributed by Rupert, it
>>>>> would be possible to use a whole graph for dereferencing, so "simply"
>>>>> routing components like Enhancer->Marmotta->Solr sounds to me like an
>>>>> interesting use case.
>>>>>
>>>>> wdyt?
>>>>>
>>>>> Cheers,
>>>>> Rafa
>>>>>
>>>>
>>>>
>>>
>>
>
--
------------------------------
This message should be regarded as confidential. If you have received this
email in error please notify the sender and destroy it immediately.
Statements of intent shall only become binding when confirmed in hard copy
by an authorised signatory.
Zaizi Ltd is registered in England and Wales with the registration number
6440931. The Registered Office is Brook House, 229 Shepherds Bush Road,
London W6 7AN.
Re: Camel integration (was : Re: Community bonding period started)
Posted by Antonio David Perez Morales <ap...@zaizi.com>.
Hi people
As part of the GSoC project for the midterm and according to the issue [1],
a custom Apache Felix Fileinstall artifact has been created in order to
deploy Camel routes defined in XML (Spring DSL) placing a file with .route
extension in a configured directory (like stanbol/fileinstall directory).
Moreover since this artifact depends on Fileinstall bundle, the created
launcher has been modified to have that bundle in the OSGI context by
default.
So, once the current Camel integration POC has been integrated in Stanbol
1.0 and extended to support the deployment of routes defined by Java DSL
(through bundles) and XML (route files), the next step will be thinking and
redesigning the current architecture trying to avoid the duplicated code
and providing a more extendable and easy to use Workflow API, because with
the current integration only direct routes can be triggered using REST API
which means that the defined routes must be configured properly using a
direct endpoint consumer. Anyway, routes starting in some other way like
timers are triggered directly in the deployment, so this has to be taken
into account for the new API (and REST API).
In parallel and for the second part, new Stanbol Camel components will be
developed in order to be used in new routes. So if any of you have use
cases for this involving Stanbol components, please drop some lines here in
order to prioritize the Stanbol Camel components to be developed.
Comments and suggestions are more than welcome
Regards
[1] https://issues.apache.org/jira/browse/STANBOL-1348
[2] https://github.com/adperezmorales/stanbol-camel-workflow/
On Mon, Jun 2, 2014 at 7:00 PM, Antonio David Perez Morales <
aperez@zaizi.com> wrote:
> Hi stanbolers
>
> As part of the issue [1] , I have created a maven archetype useful to
> generate Camel routes in Java DSL.
> The archetype generates a Java project with all the dependencies and one
> Java class with a method which has to be filled. In this method, Camel Java
> DSL syntax is used to create the route.
> By default and as a first approach, the class will use the route name
> given during the project creation to enable a Camel direct endpoint with
> such name.
> The code of the first archetype version can be found at [2].
>
> The next task will be providing a Felix custom artifact to be able to
> deploy XML-based routes in Stanbol, placing a custom file in the Stanbol
> datafiles directory.
> After that, it will be time to think and redesign the architecture to
> integrate Camel workflows inside Stanbol in a better way, more configurable
> and extendable.
>
> Comments and suggestions are more than welcome
>
> Regards
>
> [1] https://issues.apache.org/jira/browse/STANBOL-1348
> [2] https://github.com/adperezmorales/stanbol-camel-workflow/
>
>
> On Fri, May 30, 2014 at 8:03 PM, Antonio David Perez Morales <
> aperez@zaizi.com> wrote:
>
>> Hi all
>>
>> After a hard fight this week, I managed to get it work the Florent's
>> proof of concept code in the Stanbol 1.0 branch [1]
>> The code is uploaded in my github account [3]. As I said in a previous
>> mail, I prefer to do it separately and after the project, uploading the
>> developed code into a Stanbol branch.
>>
>> The 1.0.0 version has some changes in how the Jersey endpoints are
>> registered and also new classes and packages, so it was not a trivial task
>> to make work the current proof of concept. Moreover I don't like to simply
>> copy and paste code and make the needed changes. I always want to
>> understand how the things work and how they are developed in order to be
>> able to change/modify them or develop new code around them.
>>
>> The steps done to achieve it have been the following:
>> - Updated pom files to the Stanbol 1.0.0-SNAPSHOT version
>> - Updated bundle levels in bundlelist package to fit the Stanbol 1.0
>> version levels
>> - Adapted cameljobmanager package code to Stanbol 1.0.0-SNAPSHOT classes
>> and using Java OSGI annotations instead of SCR annotations in Javadoc
>> - Updated flow web package to Stanbol 1.0.0-SNAPSHOT classes and modified
>> needed resources
>> - Added Java OSGI annotations to the route (WeightedChain) instead of SCR
>> annotations in javadoc
>> - Updated launcher to use the 1.0.0-SNAPSHOT packages and needed bundles
>>
>> So now, the http://localhost:8080/flow endpoint will use the only Camel
>> route (defined by WeightedChain) to call all the registered Enhancement
>> Engines (ordered by EnhancementEngine order property).
>> For testing purposes, the /flow/{flowName} has been removed, because all
>> this code needs to be re-designed and re-implemented so I only wanted to
>> make it work to have a first (simple) integration in Stanbol 1.0. This
>> functionality will be added again to trigger custom routes once the next
>> step (defined below) is developed.
>>
>> The next step [2] will be support to write and configure routes in XML
>> format, putting the file in datafiles in order to be loaded by a Felix
>> custom artifact (as Rupert pointed out in a previous mail) and create a
>> Maven archetype to create bundles defining routes which will be loaded
>> using the Felix bundle tab. If necessary, as we talked in previous
>> messages, a REST endpoint receiving routes in XML can be developed as an
>> alternative to the first approach. This is my objective for the midterm.
>>
>> After the midterm, the new Stanbol components for Apache Camel will be
>> developed and also the new architecture for Camel in Stanbol.
>>
>> Comments on this and for use cases for Stanbol Camel components are more
>> than welcome.
>>
>> Regards
>>
>> [1] https://issues.apache.org/jira/browse/STANBOL-1347
>> [2] https://issues.apache.org/jira/browse/STANBOL-1348
>> [3] https://github.com/adperezmorales/stanbol-camel-workflow/
>>
>>
>> On Tue, May 27, 2014 at 6:18 PM, Antonio David Perez Morales <
>> aperez@zaizi.com> wrote:
>>
>>> Hi people
>>>
>>> I have already started to work on [1] to integrate current Florent's
>>> code into Stanbol 1.0.
>>> As a first approach, only changing the dependency versions to new
>>> Stanbol 1.0, many issues have arisen:
>>> - Deprecated use of classes
>>> - Classes which have changed from package
>>> - Some classes not necessary now
>>> - Classes not used which were causing conflicts
>>> - ...
>>>
>>> So now I'm trying to resolve all these problems to replicate the same
>>> behavior from 0.9 into 1.0. I will upload the code to a Github repository
>>> in my account (which will be pushed later into a Stanbol branch after the
>>> project) in order to track the advances.
>>> Once I can resolve all these problems, I will take a look to the Felix
>>> Custom Artifacts poiinted out by Rupert in a previous message to find out
>>> the best way to deploy (and manage) route configurations (felix artifacts,
>>> watchservice java, rest endpoint to receive xml routes, etc).
>>>
>>> Comments on this and future tasks are more than welcome.
>>>
>>> Regards
>>>
>>> [1] https://issues.apache.org/jira/browse/STANBOL-1347
>>>
>>>
>>>
>>> On Tue, May 27, 2014 at 9:53 AM, Rafa Haro <rh...@apache.org> wrote:
>>>
>>>> Hi Rupert, Florent and Antonio
>>>>
>>>> El 27/05/14 08:51, Rupert Westenthaler escribió:
>>>>
>>>> As the result of Enhancement Routes is content + metadata I can not
>>>>> see what you want to "store" in the Entityhub that is about managing
>>>>> Entities.
>>>>>
>>>>> > - entityhub: To query/update the entityhub component
>>>>>>
>>>>> Maybe. If you can come up with a good use case ^^
>>>>>
>>>>> > - contenthub: To develop a new content-hub using chain/engine
>>>>>> components
>>>>>> >and solr/elasticsearch/whatever component (solr and elasticsearch
>>>>>> component
>>>>>> >already exist in Camel)
>>>>>>
>>>>> IMO implementing a new Contenthub like component is outside the scope
>>>>> of this GSoC project. However If there is already Solr/Elasticsearch
>>>>> component it would be a really useful thing
>>>>>
>>>>>
>>>> Regarding this, in my opinion, the use case of an eventual integration
>>>> with a Content hub is probably one of the most clear for this project. I'm
>>>> not sure if that is what Antonio was trying to explain but, with a single
>>>> route using as last endpoint Solr or any other backend system, we would be
>>>> almost cloning the same functionality than the previous ContentHub
>>>> implementation (Stanbol 0.12). Entities could be dereferenced using the
>>>> EntityHub before storing the content along with the metadata, which is the
>>>> point of integration of the EntityHub in such use case. And even most
>>>> interesting, now with the integration of Marmotta contributed by Rupert, it
>>>> would be possible to use a whole graph for dereferencing, so "simply"
>>>> routing components like Enhancer->Marmotta->Solr sounds to me like an
>>>> interesting use case.
>>>>
>>>> wdyt?
>>>>
>>>> Cheers,
>>>> Rafa
>>>>
>>>
>>>
>>
>
--
------------------------------
This message should be regarded as confidential. If you have received this
email in error please notify the sender and destroy it immediately.
Statements of intent shall only become binding when confirmed in hard copy
by an authorised signatory.
Zaizi Ltd is registered in England and Wales with the registration number
6440931. The Registered Office is Brook House, 229 Shepherds Bush Road,
London W6 7AN.