You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@stanbol.apache.org by Ezequiel Foncubierta <ef...@zaizi.com> on 2011/07/06 23:54:40 UTC
Asynchronous calls support for the engines resource
Hi,
I'm not sure if this option is currently supported or there is a plan to do it. I think that would be interesting to enable an async option when you call the engines/ resource. If the system is overloaded, or some engines take a long time to process the content, it should be an option to run asynchronous transactions. Some integrations requires synchronous calls, because they want to tag contents in the same transactions. But, could be some others in which the synchronous calls are not a required feature.
This proposal is because, almost the all the current integration uses synchronous calls. It means that, the content creation process is as following:
1. Create the content in the CMS
2. Send the content to the enhancer
3. Write the enhancer results
4. Relate the content with the extracted entities
So, the CMS performance depends on the Apache Stanbol performance. An alternative, would be creating the content and run a background process to extract the enhancements (using different transactions).
A first way to get it, is by sending a url parameter (e.g. referer=http://system/listener/service). If this parameter is present, then run the enhancements in a background thread and, once finished, then send the results to the specified url in the referer parameter.
You know better than me the pros and cons of the current implementation, so... what do you think about the asynchronous calls?
Thanks. Regards,
--
Ezequiel Foncubierta
This message should be regarded as confidential. If you have received this email in error please notify the sender and destroy it immediately. Statements of intent shall only become binding when confirmed in hard copy by an authorised signatory.
Zaizi Ltd is registered in England and Wales with the registration number 6440931. The Registered Office is 203 Westbourne Studios, 242 Acklam Road, London W10 5JJ, UK.
Re: Asynchronous calls support for the engines resource
Posted by Rupert Westenthaler <ru...@gmail.com>.
Hallo
On Thu, Jul 7, 2011 at 5:07 PM, Aingaran Pillai <ap...@zaizi.com> wrote:
> Hi,
>
> I think a better option would be to follow the Solr Data Import handler
> example and create a CMIS Import Handler. For example CMIS spec supports
> Change logs - list of content created, updated, deleted. Stanbol and track
> the changes in a CMIS repository and do extraction.
>
You might be interested in the work on the CMS Adapter component.
This already supports this kind of feature for CMIS and JCR.
In the coming month it is the plan to deeply integrate it to the other
components so that you can use it to
* send Content to the Enhancer
* semantically index Content in the Contenthub
* synchronies Entities (Persons, Customers, Taxonomies) stored in the
CMS with the Entityhub
* extract knowledge present in the hierarchy and properties of the CMS
and use it in combination with the rules and inferencing functionality of
Stanbol.
best
Rupert
--
| Rupert Westenthaler rupert.westenthaler@gmail.com
| Bodenlehenstraße 11 ++43-699-11108907
| A-5500 Bischofshofen
Re: Asynchronous calls support for the engines resource
Posted by Aingaran Pillai <ap...@zaizi.com>.
Hi,
I think a better option would be to follow the Solr Data Import handler example and create a CMIS Import Handler. For example CMIS spec supports Change logs - list of content created, updated, deleted. Stanbol and track the changes in a CMIS repository and do extraction.
Ainga
On 7 Jul 2011, at 05:06, Olivier Grisel wrote:
> 2011/7/6 Ezequiel Foncubierta <ef...@zaizi.com>:
>> Hi,
>>
>> I'm not sure if this option is currently supported or there is a plan to do it. I think that would be interesting to enable an async option when you call the engines/ resource. If the system is overloaded, or some engines take a long time to process the content, it should be an option to run asynchronous transactions. Some integrations requires synchronous calls, because they want to tag contents in the same transactions. But, could be some others in which the synchronous calls are not a required feature.
>>
>> This proposal is because, almost the all the current integration uses synchronous calls. It means that, the content creation process is as following:
>>
>> 1. Create the content in the CMS
>> 2. Send the content to the enhancer
>> 3. Write the enhancer results
>> 4. Relate the content with the extracted entities
>>
>> So, the CMS performance depends on the Apache Stanbol performance. An alternative, would be creating the content and run a background process to extract the enhancements (using different transactions).
>>
>> A first way to get it, is by sending a url parameter (e.g. referer=http://system/listener/service). If this parameter is present, then run the enhancements in a background thread and, once finished, then send the results to the specified url in the referer parameter.
>
> Sounds like a good approach but nothing is implemented yet. The Async
> stuff that occurs in the source code is a left over of early code
> prototyping that happened during the first sprint and was never lead
> to it's term.
>
>> You know better than me the pros and cons of the current implementation, so... what do you think about the asynchronous calls?
>
> They would be useful. Please file a new jira for this.
>
> We need to extend the JobManager to fork threads for such stuff. We
> could use the JDK ThreadPoolExecutor API to do this quite easily (in
> memory queued tasks with basic multicore parallelism).
>
> Later we might also want to provide a way to query for some monitoring
> / progress info: the first query returns a token id that could be used
> to query for a description of the progress of the job.
>
> --
> Olivier
> http://twitter.com/ogrisel - http://github.com/ogrisel
This message should be regarded as confidential. If you have received this email in error please notify the sender and destroy it immediately. Statements of intent shall only become binding when confirmed in hard copy by an authorised signatory.
Zaizi Ltd is registered in England and Wales with the registration number 6440931. The Registered Office is 203 Westbourne Studios, 242 Acklam Road, London W10 5JJ, UK.
Re: Asynchronous calls support for the engines resource
Posted by Olivier Grisel <ol...@ensta.org>.
2011/7/6 Ezequiel Foncubierta <ef...@zaizi.com>:
> Hi,
>
> I'm not sure if this option is currently supported or there is a plan to do it. I think that would be interesting to enable an async option when you call the engines/ resource. If the system is overloaded, or some engines take a long time to process the content, it should be an option to run asynchronous transactions. Some integrations requires synchronous calls, because they want to tag contents in the same transactions. But, could be some others in which the synchronous calls are not a required feature.
>
> This proposal is because, almost the all the current integration uses synchronous calls. It means that, the content creation process is as following:
>
> 1. Create the content in the CMS
> 2. Send the content to the enhancer
> 3. Write the enhancer results
> 4. Relate the content with the extracted entities
>
> So, the CMS performance depends on the Apache Stanbol performance. An alternative, would be creating the content and run a background process to extract the enhancements (using different transactions).
>
> A first way to get it, is by sending a url parameter (e.g. referer=http://system/listener/service). If this parameter is present, then run the enhancements in a background thread and, once finished, then send the results to the specified url in the referer parameter.
Sounds like a good approach but nothing is implemented yet. The Async
stuff that occurs in the source code is a left over of early code
prototyping that happened during the first sprint and was never lead
to it's term.
> You know better than me the pros and cons of the current implementation, so... what do you think about the asynchronous calls?
They would be useful. Please file a new jira for this.
We need to extend the JobManager to fork threads for such stuff. We
could use the JDK ThreadPoolExecutor API to do this quite easily (in
memory queued tasks with basic multicore parallelism).
Later we might also want to provide a way to query for some monitoring
/ progress info: the first query returns a token id that could be used
to query for a description of the progress of the job.
--
Olivier
http://twitter.com/ogrisel - http://github.com/ogrisel
Re: Asynchronous calls support for the engines resource
Posted by Ezequiel Foncubierta <ef...@zaizi.com>.
Hi
I'm going to create the ticket about async calls, and then discuss the architecture.
Regards,
On 11 Jul 2011, at 09:13, Bertrand Delacretaz wrote:
> Hi,
>
> On Sun, Jul 10, 2011 at 9:46 PM, Ezequiel Foncubierta
> <ef...@zaizi.com> wrote:
>> ,,,If you need to process large files, an asynchronous option would be nice to avoid having connections waiting for a
>> response. I think that both, configurable chains and asynchronous calls, ought to be implemented...
>
> I agree with the need for async enhancement engines, one of the ideas
> that we had when discussing the initial FISE design (that led to the
> Stanbol enhancer) was to have some metadata that says "we're still
> working on some parts of this content and might get some more metadata
> later" in the content items to handle status info about asynchronous
> enhancement.
>
> Async processing should IMO take things like Mechanical Turk into
> account, i.e. extremely slow processing, using things like
> http://groups.csail.mit.edu/uid/turkit/ maybe.
>
> Another idea, more complicated but potentially much more powerful, is
> to use a kind of tuple space for enhancement engines to collaborate,
> something like:
>
> -Content item CI is added to the space
> -Engine A sees CI, works on it and as a result adds triple A1 to the space
> -Engine B sees triple A1, works on it and adds triple B1 to the space
> -Engine A sees triple B1 and adds more metadata, based on it an CI, to the space
>
> Engines can then work iteratively on finding out more things about
> content items, and this would also allow for correlating metadata
> supplied by several engines to improve the metadata quality.
>
> I'm basically just dreaming outloud here, but I think we should take
> this into account if we introduce multiple processing chains in the
> enhancer: some chains might use a totally different engine
> collaboration mechanism like the above, instead of the current
> sequential processing.
>
> -Bertrand
This message should be regarded as confidential. If you have received this email in error please notify the sender and destroy it immediately. Statements of intent shall only become binding when confirmed in hard copy by an authorised signatory.
Zaizi Ltd is registered in England and Wales with the registration number 6440931. The Registered Office is 203 Westbourne Studios, 242 Acklam Road, London W10 5JJ, UK.
Re: Asynchronous calls support for the engines resource
Posted by Bertrand Delacretaz <bd...@apache.org>.
Hi,
On Sun, Jul 10, 2011 at 9:46 PM, Ezequiel Foncubierta
<ef...@zaizi.com> wrote:
> ,,,If you need to process large files, an asynchronous option would be nice to avoid having connections waiting for a
> response. I think that both, configurable chains and asynchronous calls, ought to be implemented...
I agree with the need for async enhancement engines, one of the ideas
that we had when discussing the initial FISE design (that led to the
Stanbol enhancer) was to have some metadata that says "we're still
working on some parts of this content and might get some more metadata
later" in the content items to handle status info about asynchronous
enhancement.
Async processing should IMO take things like Mechanical Turk into
account, i.e. extremely slow processing, using things like
http://groups.csail.mit.edu/uid/turkit/ maybe.
Another idea, more complicated but potentially much more powerful, is
to use a kind of tuple space for enhancement engines to collaborate,
something like:
-Content item CI is added to the space
-Engine A sees CI, works on it and as a result adds triple A1 to the space
-Engine B sees triple A1, works on it and adds triple B1 to the space
-Engine A sees triple B1 and adds more metadata, based on it an CI, to the space
Engines can then work iteratively on finding out more things about
content items, and this would also allow for correlating metadata
supplied by several engines to improve the metadata quality.
I'm basically just dreaming outloud here, but I think we should take
this into account if we introduce multiple processing chains in the
enhancer: some chains might use a totally different engine
collaboration mechanism like the above, instead of the current
sequential processing.
-Bertrand
Re: Asynchronous calls support for the engines resource
Posted by Rupert Westenthaler <ru...@gmail.com>.
Hi
Can someone please open JIRA issues for both. I would be interested to
work on the multiple chains.
best
Rupert
On Sun, Jul 10, 2011 at 9:46 PM, Ezequiel Foncubierta
<ef...@zaizi.com> wrote:
> Hi,
>
> Nice, thats a good idea. But anyway, if you need to process large files, an asynchronous option would be nice to avoid having connections waiting for a response. I think that both, configurable chains and asynchronous calls, ought to be implemented.
>
> Regards,
>
> On 9 Jul 2011, at 12:05, Fabian Christ wrote:
>
>> Hi,
>>
>> yes we discussed such chains a while ago and they are definitely on
>> the roadmap. If I remember correctly, the idea was to have different
>> chains at different URLs.
>>
>> /engines/fastchain
>> /engines/mytestchain
>> ...
>>
>> something like that
>>
>> Best,
>> - Fabian
>>
>> 2011/7/9 florent andré <fl...@4sengines.com>:
>>> Near this idea, I think that the need of multiple processing chains in
>>> Stanbol will appear.
>>>
>>> As for example, when upload a document, (at least) two types of processing
>>> chain could be useful :
>>>
>>> - The first one - fast - with just one or two "essentials" engines, provide
>>> entities to the users in (near) "real time"
>>>
>>> - The second one - slower -with a lot of engines, provided a detailed
>>> analysis, and computations that help for classification, summary, detailed
>>> enhancements, etc... and are displayed to the user later or when reopen
>>> document.
>>>
>>> As I know, for now, the only way to do that is to set up two different
>>> Stanbol servers, with a different engine stack for each...
>>>
>>> An opportunity to define different processing chain is something in the
>>> roadmap ?
>>>
>>> ++
>>>
>>>
>>>
>>>
>>> On 07/06/2011 11:54 PM, Ezequiel Foncubierta wrote:
>>>>
>>>> Hi,
>>>>
>>>> I'm not sure if this option is currently supported or there is a plan to
>>>> do it. I think that would be interesting to enable an async option when you
>>>> call the engines/ resource. If the system is overloaded, or some engines
>>>> take a long time to process the content, it should be an option to run
>>>> asynchronous transactions. Some integrations requires synchronous calls,
>>>> because they want to tag contents in the same transactions. But, could be
>>>> some others in which the synchronous calls are not a required feature.
>>>>
>>>> This proposal is because, almost the all the current integration uses
>>>> synchronous calls. It means that, the content creation process is as
>>>> following:
>>>>
>>>> 1. Create the content in the CMS
>>>> 2. Send the content to the enhancer
>>>> 3. Write the enhancer results
>>>> 4. Relate the content with the extracted entities
>>>>
>>>> So, the CMS performance depends on the Apache Stanbol performance. An
>>>> alternative, would be creating the content and run a background process to
>>>> extract the enhancements (using different transactions).
>>>>
>>>> A first way to get it, is by sending a url parameter (e.g.
>>>> referer=http://system/listener/service). If this parameter is present, then
>>>> run the enhancements in a background thread and, once finished, then send
>>>> the results to the specified url in the referer parameter.
>>>>
>>>> You know better than me the pros and cons of the current implementation,
>>>> so... what do you think about the asynchronous calls?
>>>>
>>>> Thanks. Regards,
>>>>
>>>> --
>>>> Ezequiel Foncubierta
>>>> This message should be regarded as confidential. If you have received this
>>>> email in error please notify the sender and destroy it immediately.
>>>> Statements of intent shall only become binding when confirmed in hard copy
>>>> by an authorised signatory.
>>>>
>>>> Zaizi Ltd is registered in England and Wales with the registration number
>>>> 6440931. The Registered Office is 203 Westbourne Studios, 242 Acklam Road,
>>>> London W10 5JJ, UK.
>>>>
>>>
>>
>>
>>
>> --
>> Fabian
>
>
>
> This message should be regarded as confidential. If you have received this email in error please notify the sender and destroy it immediately. Statements of intent shall only become binding when confirmed in hard copy by an authorised signatory.
>
> Zaizi Ltd is registered in England and Wales with the registration number 6440931. The Registered Office is 203 Westbourne Studios, 242 Acklam Road, London W10 5JJ, UK.
>
>
--
| Rupert Westenthaler rupert.westenthaler@gmail.com
| Bodenlehenstraße 11 ++43-699-11108907
| A-5500 Bischofshofen
Re: Asynchronous calls support for the engines resource
Posted by Ezequiel Foncubierta <ef...@zaizi.com>.
Hi,
Nice, thats a good idea. But anyway, if you need to process large files, an asynchronous option would be nice to avoid having connections waiting for a response. I think that both, configurable chains and asynchronous calls, ought to be implemented.
Regards,
On 9 Jul 2011, at 12:05, Fabian Christ wrote:
> Hi,
>
> yes we discussed such chains a while ago and they are definitely on
> the roadmap. If I remember correctly, the idea was to have different
> chains at different URLs.
>
> /engines/fastchain
> /engines/mytestchain
> ...
>
> something like that
>
> Best,
> - Fabian
>
> 2011/7/9 florent andré <fl...@4sengines.com>:
>> Near this idea, I think that the need of multiple processing chains in
>> Stanbol will appear.
>>
>> As for example, when upload a document, (at least) two types of processing
>> chain could be useful :
>>
>> - The first one - fast - with just one or two "essentials" engines, provide
>> entities to the users in (near) "real time"
>>
>> - The second one - slower -with a lot of engines, provided a detailed
>> analysis, and computations that help for classification, summary, detailed
>> enhancements, etc... and are displayed to the user later or when reopen
>> document.
>>
>> As I know, for now, the only way to do that is to set up two different
>> Stanbol servers, with a different engine stack for each...
>>
>> An opportunity to define different processing chain is something in the
>> roadmap ?
>>
>> ++
>>
>>
>>
>>
>> On 07/06/2011 11:54 PM, Ezequiel Foncubierta wrote:
>>>
>>> Hi,
>>>
>>> I'm not sure if this option is currently supported or there is a plan to
>>> do it. I think that would be interesting to enable an async option when you
>>> call the engines/ resource. If the system is overloaded, or some engines
>>> take a long time to process the content, it should be an option to run
>>> asynchronous transactions. Some integrations requires synchronous calls,
>>> because they want to tag contents in the same transactions. But, could be
>>> some others in which the synchronous calls are not a required feature.
>>>
>>> This proposal is because, almost the all the current integration uses
>>> synchronous calls. It means that, the content creation process is as
>>> following:
>>>
>>> 1. Create the content in the CMS
>>> 2. Send the content to the enhancer
>>> 3. Write the enhancer results
>>> 4. Relate the content with the extracted entities
>>>
>>> So, the CMS performance depends on the Apache Stanbol performance. An
>>> alternative, would be creating the content and run a background process to
>>> extract the enhancements (using different transactions).
>>>
>>> A first way to get it, is by sending a url parameter (e.g.
>>> referer=http://system/listener/service). If this parameter is present, then
>>> run the enhancements in a background thread and, once finished, then send
>>> the results to the specified url in the referer parameter.
>>>
>>> You know better than me the pros and cons of the current implementation,
>>> so... what do you think about the asynchronous calls?
>>>
>>> Thanks. Regards,
>>>
>>> --
>>> Ezequiel Foncubierta
>>> This message should be regarded as confidential. If you have received this
>>> email in error please notify the sender and destroy it immediately.
>>> Statements of intent shall only become binding when confirmed in hard copy
>>> by an authorised signatory.
>>>
>>> Zaizi Ltd is registered in England and Wales with the registration number
>>> 6440931. The Registered Office is 203 Westbourne Studios, 242 Acklam Road,
>>> London W10 5JJ, UK.
>>>
>>
>
>
>
> --
> Fabian
This message should be regarded as confidential. If you have received this email in error please notify the sender and destroy it immediately. Statements of intent shall only become binding when confirmed in hard copy by an authorised signatory.
Zaizi Ltd is registered in England and Wales with the registration number 6440931. The Registered Office is 203 Westbourne Studios, 242 Acklam Road, London W10 5JJ, UK.
Re: Asynchronous calls support for the engines resource
Posted by Fabian Christ <ch...@googlemail.com>.
Hi,
yes we discussed such chains a while ago and they are definitely on
the roadmap. If I remember correctly, the idea was to have different
chains at different URLs.
/engines/fastchain
/engines/mytestchain
...
something like that
Best,
- Fabian
2011/7/9 florent andré <fl...@4sengines.com>:
> Near this idea, I think that the need of multiple processing chains in
> Stanbol will appear.
>
> As for example, when upload a document, (at least) two types of processing
> chain could be useful :
>
> - The first one - fast - with just one or two "essentials" engines, provide
> entities to the users in (near) "real time"
>
> - The second one - slower -with a lot of engines, provided a detailed
> analysis, and computations that help for classification, summary, detailed
> enhancements, etc... and are displayed to the user later or when reopen
> document.
>
> As I know, for now, the only way to do that is to set up two different
> Stanbol servers, with a different engine stack for each...
>
> An opportunity to define different processing chain is something in the
> roadmap ?
>
> ++
>
>
>
>
> On 07/06/2011 11:54 PM, Ezequiel Foncubierta wrote:
>>
>> Hi,
>>
>> I'm not sure if this option is currently supported or there is a plan to
>> do it. I think that would be interesting to enable an async option when you
>> call the engines/ resource. If the system is overloaded, or some engines
>> take a long time to process the content, it should be an option to run
>> asynchronous transactions. Some integrations requires synchronous calls,
>> because they want to tag contents in the same transactions. But, could be
>> some others in which the synchronous calls are not a required feature.
>>
>> This proposal is because, almost the all the current integration uses
>> synchronous calls. It means that, the content creation process is as
>> following:
>>
>> 1. Create the content in the CMS
>> 2. Send the content to the enhancer
>> 3. Write the enhancer results
>> 4. Relate the content with the extracted entities
>>
>> So, the CMS performance depends on the Apache Stanbol performance. An
>> alternative, would be creating the content and run a background process to
>> extract the enhancements (using different transactions).
>>
>> A first way to get it, is by sending a url parameter (e.g.
>> referer=http://system/listener/service). If this parameter is present, then
>> run the enhancements in a background thread and, once finished, then send
>> the results to the specified url in the referer parameter.
>>
>> You know better than me the pros and cons of the current implementation,
>> so... what do you think about the asynchronous calls?
>>
>> Thanks. Regards,
>>
>> --
>> Ezequiel Foncubierta
>> This message should be regarded as confidential. If you have received this
>> email in error please notify the sender and destroy it immediately.
>> Statements of intent shall only become binding when confirmed in hard copy
>> by an authorised signatory.
>>
>> Zaizi Ltd is registered in England and Wales with the registration number
>> 6440931. The Registered Office is 203 Westbourne Studios, 242 Acklam Road,
>> London W10 5JJ, UK.
>>
>
--
Fabian
Re: Asynchronous calls support for the engines resource
Posted by florent andré <fl...@4sengines.com>.
Near this idea, I think that the need of multiple processing chains in
Stanbol will appear.
As for example, when upload a document, (at least) two types of
processing chain could be useful :
- The first one - fast - with just one or two "essentials" engines,
provide entities to the users in (near) "real time"
- The second one - slower -with a lot of engines, provided a detailed
analysis, and computations that help for classification, summary,
detailed enhancements, etc... and are displayed to the user later or
when reopen document.
As I know, for now, the only way to do that is to set up two different
Stanbol servers, with a different engine stack for each...
An opportunity to define different processing chain is something in the
roadmap ?
++
On 07/06/2011 11:54 PM, Ezequiel Foncubierta wrote:
> Hi,
>
> I'm not sure if this option is currently supported or there is a plan to do it. I think that would be interesting to enable an async option when you call the engines/ resource. If the system is overloaded, or some engines take a long time to process the content, it should be an option to run asynchronous transactions. Some integrations requires synchronous calls, because they want to tag contents in the same transactions. But, could be some others in which the synchronous calls are not a required feature.
>
> This proposal is because, almost the all the current integration uses synchronous calls. It means that, the content creation process is as following:
>
> 1. Create the content in the CMS
> 2. Send the content to the enhancer
> 3. Write the enhancer results
> 4. Relate the content with the extracted entities
>
> So, the CMS performance depends on the Apache Stanbol performance. An alternative, would be creating the content and run a background process to extract the enhancements (using different transactions).
>
> A first way to get it, is by sending a url parameter (e.g. referer=http://system/listener/service). If this parameter is present, then run the enhancements in a background thread and, once finished, then send the results to the specified url in the referer parameter.
>
> You know better than me the pros and cons of the current implementation, so... what do you think about the asynchronous calls?
>
> Thanks. Regards,
>
> --
> Ezequiel Foncubierta
> This message should be regarded as confidential. If you have received this email in error please notify the sender and destroy it immediately. Statements of intent shall only become binding when confirmed in hard copy by an authorised signatory.
>
> Zaizi Ltd is registered in England and Wales with the registration number 6440931. The Registered Office is 203 Westbourne Studios, 242 Acklam Road, London W10 5JJ, UK.
>