You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@manifoldcf.apache.org by "Christian M. Rieck" <ch...@comperiosearch.com> on 2013/11/21 12:05:00 UTC

RabbitMQ output connector and content processing integration - RFC

Hello,

The need for a content processing pipeline has led me to implement an output connector for RabbitMQ. In my current setup Logstash reads from the queue, does some simple processing and feeds the document to Elasticsearch. Scaling is pretty straight forward by adding more Logstash processes. Of all the content processing frameworks I've looked at lately Logstash seems like the only one that has the most traction at the moment. Support for delete operations (in Logstash) is still missing, but is being addressed.

There are two issues that I've identified that I'd like the community to comment on: (I write RabbitMQ and Logstash explicitly, but the argument holds for any queue/processing framework)

-          Data format. How should the document look when it is handed over to RabbitMQ. Is there a standard format than should be used? OData? By using a standard format it would be easier for other systems to consume data from Manifold as well. For my particular upcoming project I will probably send the content in the JSON format used by Logstash.

-          Error handling. Who is responsible should a document fail during content processing? A fire-and-forget approach from Manifold with leaving the doc in the message queue is the simplest. Logstash is responsible for handling the error and refeed/fix the error until success. If there is an API for telling Manifold about failing docs Logstash can write status messages to a separate queue.  Reading that queue and updating doc status in Manifold should be easy.





Any thoughts on this, or interest in general for a RabbitMQ output connector?





Regards,

Christian.

Re: RabbitMQ output connector and content processing integration - RFC

Posted by Karl Wright <da...@gmail.com>.
>>>>>>What is the best way going forward, practically? Right now I have
integrated the connector in the build.xmls locally and it seems to work
correctly. The connector itself is however not finished. I am right now
working on moving the HTML from being returned as a string in the java
class to being based on .html-files in main/resources , for instance. When
and how would you like the code? How "finished" does it have to be?
<<<<<<

Hi Christian,
Usually for co-development of this kind the best approach is to create a
ticket and a corresponding ManifoldCF branch.  Then, at least initially,
you submit patches to the ticket and one of the committers commits them to
the branch.  This also gives us a chance to review things and recommend
modifications, or even chip in and help.  Things may proceed further from
there.

If you go ahead and create a ticket, I'd be happy to set up a branch right
now.

Thanks!
Karl



On Thu, Nov 21, 2013 at 7:33 AM, Christian M. Rieck <
christian.rieck@comperiosearch.com> wrote:

> >>>>>>>>>
> Hi Christian,
> Of course there is interest in a RabbitMQ output connector.  Please let us
> know any way in which we can help.
> <<<<<<<<
> Showing interest helps :)
>
> >>>>>>
> Basically you have free reign in your output connector to decide what data
> format makes sense.  The output you will get from ManifoldCF will be a
> RepositoryDocument java object, which consists of one (1) binary chunk of
> data, in stream form, and (2) multiple pieces of metadata, which are
> presumed to be bounded in size, and (3) some well-known metadata, such as
> the content-type of the binary data you are dealing with, if known.  You
> should *not* in general presume you can fit the entire binary file in
> memory.
> <<<<<<
>
> Ok, I thought maybe it would be wise and see if ManifoldCF could
> coordinate somewhat with other open source initiatives. I'll go with
> Logstash format for now and it will be easy to add some radiobuttons in the
> config to choose other formats later. As for not fitting the entire binary
> file in memory, I hadn't thought of that. Any tips on how to do this? I'll
> see what RabbitMQ supports.
>
> >>>>>>>>
> Output connectors have the following error handling:
>
> (1) ServiceInterruption exception, which indicates that the document
> cannot be accepted at the time, but may be accepted if tried again later
> (and you can signal how long to wait before retrying);
> (2) "Document accepted" return code, indicating that the document was
> successfully received
> (3) "Document rejected" return code, indicating that the document has been
> determined to be unsuitable, and will thus NOT be retried;
> (4) ManifoldCFException exception, which indicates that something fatal
> and unexpected happened.  (Code ManifoldCFException.INTERRUPTED should be
> thrown only on
> thread interruption during shutdown.)
> <<<<<<<<
>
> These are all thrown or returned from addOrReplaceDocument() I presume,
> and as such its either fire-and-forget with either "document accepted" or
> exceptions or have the method wait/sleep until RabbitMQ says otherwise. I
> will go for fire-and-forget to start with and look more into RabbitMQ's
> Acknowledgements, Confirms and Dead-Letter-Exchanges and see where that
> takes me. If anyone on this mailing list have extensive rabbitmq
> experience, please let me know.
>
>
> >>>>>>>>
> Hope this helps.
> <<<<<<<<
>
> Helped a lot :)
> What is the best way going forward, practically? Right now I have
> integrated the connector in the build.xmls locally and it seems to work
> correctly. The connector itself is however not finished. I am right now
> working on moving the HTML from being returned as a string in the java
> class to being based on .html-files in main/resources , for instance. When
> and how would you like the code? How "finished" does it have to be?
>
>
> Christian.
>
>
>
> -----Original Message-----
> From: Karl Wright [mailto:daddywri@gmail.com]
> Sent: 21. november 2013 12:34
> To: dev
> Subject: Re: RabbitMQ output connector and content processing integration
> - RFC
>
> Hi Christian,
>
> Of course there is interest in a RabbitMQ output connector.  Please let us
> know any way in which we can help.
>
> >>>>>>
> -          Data format. How should the document look when it is handed over
> to RabbitMQ. Is there a standard format than should be used? OData? By
> using a standard format it would be easier for other systems to consume
> data from Manifold as well. For my particular upcoming project I will
> probably send the content in the JSON format used by Logstash.
> <<<<<<
>
> Basically you have free reign in your output connector to decide what data
> format makes sense.  The output you will get from ManifoldCF will be a
> RepositoryDocument java object, which consists of one (1) binary chunk of
> data, in stream form, and (2) multiple pieces of metadata, which are
> presumed to be bounded in size, and (3) some well-known metadata, such as
> the content-type of the binary data you are dealing with, if known.  You
> should *not* in general presume you can fit the entire binary file in
> memory.
>
> >>>>>>
> -          Error handling. Who is responsible should a document fail during
> content processing? A fire-and-forget approach from Manifold with leaving
> the doc in the message queue is the simplest. Logstash is responsible for
> handling the error and refeed/fix the error until success. If there is an
> API for telling Manifold about failing docs Logstash can write status
> messages to a separate queue.  Reading that queue and updating doc status
> in Manifold should be easy.
>
> <<<<<<
>
> Output connectors have the following error handling:
>
> (1) ServiceInterruption exception, which indicates that the document
> cannot be accepted at the time, but may be accepted if tried again later
> (and you can signal how long to wait before retrying);
> (2) "Document accepted" return code, indicating that the document was
> successfully received
> (3) "Document rejected" return code, indicating that the document has been
> determined to be unsuitable, and will thus NOT be retried;
> (4) ManifoldCFException exception, which indicates that something fatal
> and unexpected happened.  (Code ManifoldCFException.INTERRUPTED should be
> thrown only on thread interruption during shutdown.)
>
> Hope this helps.
>
> Karl
>
>
>
>
> On Thu, Nov 21, 2013 at 6:05 AM, Christian M. Rieck <
> christian.rieck@comperiosearch.com> wrote:
>
> > Hello,
> >
> > The need for a content processing pipeline has led me to implement an
> > output connector for RabbitMQ. In my current setup Logstash reads from
> > the queue, does some simple processing and feeds the document to
> Elasticsearch.
> > Scaling is pretty straight forward by adding more Logstash processes.
> > Of all the content processing frameworks I've looked at lately
> > Logstash seems like the only one that has the most traction at the
> > moment. Support for delete operations (in Logstash) is still missing,
> but is being addressed.
> >
> > There are two issues that I've identified that I'd like the community
> > to comment on: (I write RabbitMQ and Logstash explicitly, but the
> > argument holds for any queue/processing framework)
> >
> > -          Data format. How should the document look when it is handed
> > over to RabbitMQ. Is there a standard format than should be used?
> > OData? By using a standard format it would be easier for other systems
> > to consume data from Manifold as well. For my particular upcoming
> > project I will probably send the content in the JSON format used by
> Logstash.
> >
> > -          Error handling. Who is responsible should a document fail
> > during content processing? A fire-and-forget approach from Manifold
> > with leaving the doc in the message queue is the simplest. Logstash is
> > responsible for handling the error and refeed/fix the error until
> success.
> > If there is an API for telling Manifold about failing docs Logstash
> > can write status messages to a separate queue.  Reading that queue and
> > updating doc status in Manifold should be easy.
> >
> >
> >
> >
> >
> > Any thoughts on this, or interest in general for a RabbitMQ output
> > connector?
> >
> >
> >
> >
> >
> > Regards,
> >
> > Christian.
> >
>

RE: RabbitMQ output connector and content processing integration - RFC

Posted by "Christian M. Rieck" <ch...@comperiosearch.com>.
>>>>>>>>>
Hi Christian,
Of course there is interest in a RabbitMQ output connector.  Please let us know any way in which we can help.
<<<<<<<<
Showing interest helps :) 

>>>>>>
Basically you have free reign in your output connector to decide what data format makes sense.  The output you will get from ManifoldCF will be a RepositoryDocument java object, which consists of one (1) binary chunk of data, in stream form, and (2) multiple pieces of metadata, which are presumed to be bounded in size, and (3) some well-known metadata, such as the content-type of the binary data you are dealing with, if known.  You should *not* in general presume you can fit the entire binary file in memory.
<<<<<<

Ok, I thought maybe it would be wise and see if ManifoldCF could coordinate somewhat with other open source initiatives. I'll go with Logstash format for now and it will be easy to add some radiobuttons in the config to choose other formats later. As for not fitting the entire binary file in memory, I hadn't thought of that. Any tips on how to do this? I'll see what RabbitMQ supports. 

>>>>>>>>
Output connectors have the following error handling:

(1) ServiceInterruption exception, which indicates that the document cannot be accepted at the time, but may be accepted if tried again later (and you can signal how long to wait before retrying);
(2) "Document accepted" return code, indicating that the document was successfully received
(3) "Document rejected" return code, indicating that the document has been determined to be unsuitable, and will thus NOT be retried;
(4) ManifoldCFException exception, which indicates that something fatal and unexpected happened.  (Code ManifoldCFException.INTERRUPTED should be thrown only on 
thread interruption during shutdown.)
<<<<<<<<

These are all thrown or returned from addOrReplaceDocument() I presume, and as such its either fire-and-forget with either "document accepted" or exceptions or have the method wait/sleep until RabbitMQ says otherwise. I will go for fire-and-forget to start with and look more into RabbitMQ's Acknowledgements, Confirms and Dead-Letter-Exchanges and see where that takes me. If anyone on this mailing list have extensive rabbitmq experience, please let me know.


>>>>>>>>
Hope this helps.
<<<<<<<<

Helped a lot :)
What is the best way going forward, practically? Right now I have integrated the connector in the build.xmls locally and it seems to work correctly. The connector itself is however not finished. I am right now working on moving the HTML from being returned as a string in the java class to being based on .html-files in main/resources , for instance. When and how would you like the code? How "finished" does it have to be? 


Christian.



-----Original Message-----
From: Karl Wright [mailto:daddywri@gmail.com] 
Sent: 21. november 2013 12:34
To: dev
Subject: Re: RabbitMQ output connector and content processing integration - RFC

Hi Christian,

Of course there is interest in a RabbitMQ output connector.  Please let us know any way in which we can help.

>>>>>>
-          Data format. How should the document look when it is handed over
to RabbitMQ. Is there a standard format than should be used? OData? By using a standard format it would be easier for other systems to consume data from Manifold as well. For my particular upcoming project I will probably send the content in the JSON format used by Logstash.
<<<<<<

Basically you have free reign in your output connector to decide what data format makes sense.  The output you will get from ManifoldCF will be a RepositoryDocument java object, which consists of one (1) binary chunk of data, in stream form, and (2) multiple pieces of metadata, which are presumed to be bounded in size, and (3) some well-known metadata, such as the content-type of the binary data you are dealing with, if known.  You should *not* in general presume you can fit the entire binary file in memory.

>>>>>>
-          Error handling. Who is responsible should a document fail during
content processing? A fire-and-forget approach from Manifold with leaving the doc in the message queue is the simplest. Logstash is responsible for handling the error and refeed/fix the error until success. If there is an API for telling Manifold about failing docs Logstash can write status messages to a separate queue.  Reading that queue and updating doc status in Manifold should be easy.

<<<<<<

Output connectors have the following error handling:

(1) ServiceInterruption exception, which indicates that the document cannot be accepted at the time, but may be accepted if tried again later (and you can signal how long to wait before retrying);
(2) "Document accepted" return code, indicating that the document was successfully received
(3) "Document rejected" return code, indicating that the document has been determined to be unsuitable, and will thus NOT be retried;
(4) ManifoldCFException exception, which indicates that something fatal and unexpected happened.  (Code ManifoldCFException.INTERRUPTED should be thrown only on thread interruption during shutdown.)

Hope this helps.

Karl




On Thu, Nov 21, 2013 at 6:05 AM, Christian M. Rieck < christian.rieck@comperiosearch.com> wrote:

> Hello,
>
> The need for a content processing pipeline has led me to implement an 
> output connector for RabbitMQ. In my current setup Logstash reads from 
> the queue, does some simple processing and feeds the document to Elasticsearch.
> Scaling is pretty straight forward by adding more Logstash processes. 
> Of all the content processing frameworks I've looked at lately 
> Logstash seems like the only one that has the most traction at the 
> moment. Support for delete operations (in Logstash) is still missing, but is being addressed.
>
> There are two issues that I've identified that I'd like the community 
> to comment on: (I write RabbitMQ and Logstash explicitly, but the 
> argument holds for any queue/processing framework)
>
> -          Data format. How should the document look when it is handed
> over to RabbitMQ. Is there a standard format than should be used? 
> OData? By using a standard format it would be easier for other systems 
> to consume data from Manifold as well. For my particular upcoming 
> project I will probably send the content in the JSON format used by Logstash.
>
> -          Error handling. Who is responsible should a document fail
> during content processing? A fire-and-forget approach from Manifold 
> with leaving the doc in the message queue is the simplest. Logstash is 
> responsible for handling the error and refeed/fix the error until success.
> If there is an API for telling Manifold about failing docs Logstash 
> can write status messages to a separate queue.  Reading that queue and 
> updating doc status in Manifold should be easy.
>
>
>
>
>
> Any thoughts on this, or interest in general for a RabbitMQ output 
> connector?
>
>
>
>
>
> Regards,
>
> Christian.
>

Re: RabbitMQ output connector and content processing integration - RFC

Posted by Karl Wright <da...@gmail.com>.
Hi Christian,

Of course there is interest in a RabbitMQ output connector.  Please let us
know any way in which we can help.

>>>>>>
-          Data format. How should the document look when it is handed over
to RabbitMQ. Is there a standard format than should be used? OData? By
using a standard format it would be easier for other systems to consume
data from Manifold as well. For my particular upcoming project I will
probably send the content in the JSON format used by Logstash.
<<<<<<

Basically you have free reign in your output connector to decide what data
format makes sense.  The output you will get from ManifoldCF will be a
RepositoryDocument java object, which consists of one (1) binary chunk of
data, in stream form, and (2) multiple pieces of metadata, which are
presumed to be bounded in size, and (3) some well-known metadata, such as
the content-type of the binary data you are dealing with, if known.  You
should *not* in general presume you can fit the entire binary file in
memory.

>>>>>>
-          Error handling. Who is responsible should a document fail during
content processing? A fire-and-forget approach from Manifold with leaving
the doc in the message queue is the simplest. Logstash is responsible for
handling the error and refeed/fix the error until success. If there is an
API for telling Manifold about failing docs Logstash can write status
messages to a separate queue.  Reading that queue and updating doc status
in Manifold should be easy.

<<<<<<

Output connectors have the following error handling:

(1) ServiceInterruption exception, which indicates that the document cannot
be accepted at the time, but may be accepted if tried again later (and you
can signal how long to wait before retrying);
(2) "Document accepted" return code, indicating that the document was
successfully received
(3) "Document rejected" return code, indicating that the document has been
determined to be unsuitable, and will thus NOT be retried;
(4) ManifoldCFException exception, which indicates that something fatal and
unexpected happened.  (Code ManifoldCFException.INTERRUPTED should be
thrown only on thread interruption during shutdown.)

Hope this helps.

Karl




On Thu, Nov 21, 2013 at 6:05 AM, Christian M. Rieck <
christian.rieck@comperiosearch.com> wrote:

> Hello,
>
> The need for a content processing pipeline has led me to implement an
> output connector for RabbitMQ. In my current setup Logstash reads from the
> queue, does some simple processing and feeds the document to Elasticsearch.
> Scaling is pretty straight forward by adding more Logstash processes. Of
> all the content processing frameworks I've looked at lately Logstash seems
> like the only one that has the most traction at the moment. Support for
> delete operations (in Logstash) is still missing, but is being addressed.
>
> There are two issues that I've identified that I'd like the community to
> comment on: (I write RabbitMQ and Logstash explicitly, but the argument
> holds for any queue/processing framework)
>
> -          Data format. How should the document look when it is handed
> over to RabbitMQ. Is there a standard format than should be used? OData? By
> using a standard format it would be easier for other systems to consume
> data from Manifold as well. For my particular upcoming project I will
> probably send the content in the JSON format used by Logstash.
>
> -          Error handling. Who is responsible should a document fail
> during content processing? A fire-and-forget approach from Manifold with
> leaving the doc in the message queue is the simplest. Logstash is
> responsible for handling the error and refeed/fix the error until success.
> If there is an API for telling Manifold about failing docs Logstash can
> write status messages to a separate queue.  Reading that queue and updating
> doc status in Manifold should be easy.
>
>
>
>
>
> Any thoughts on this, or interest in general for a RabbitMQ output
> connector?
>
>
>
>
>
> Regards,
>
> Christian.
>