You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Brandon Fish <br...@gmail.com> on 2011/07/15 19:52:22 UTC

Data Import from a Queue

Does anyone know of any existing examples of importing data from a queue
into Solr?

Thank you.

Re: Data Import from a Queue

Posted by Bill Bell <bi...@gmail.com>.
Yes this is a good reason for using a queue. I have used Amazon SQS this way and it was simple to set up.

Bill Bell
Sent from mobile


On Jul 20, 2011, at 2:59 AM, Stefan Matheis <ma...@googlemail.com> wrote:

> Brandon,
> 
> i don't know how they are using it in detail, but Part of Chef's Architecture is this one:
> 
> Chef Server -> RabbitMQ -> Chef Solr Indexer -> Solr
> http://wiki.opscode.com/download/attachments/7274878/chef-server-arch.png
> 
> Perhaps not exactly, what you're looking for - but may give you an idea?
> 
> Regards
> Stefan
> 
> Am 19.07.2011 19:04, schrieb Brandon Fish:
>> Let me provide some more details to the question:
>> 
>> I was unable to find any example implementations where individual documents
>> (single document per message) are read from a message queue (like ActiveMQ
>> or RabbitMQ) and then added to Solr via SolrJ, a HTTP POST or another
>> method. Does anyone know of any available examples for this type of import?
>> 
>> If no examples exist, what would be a recommended commit strategy for
>> performance? My best guess for this would be to have a queue per core and
>> commit once the queue is empty.
>> 
>> Thanks.
>> 
>> On Mon, Jul 18, 2011 at 6:52 PM, Erick Erickson<er...@gmail.com>wrote:
>> 
>>> This is a really cryptic problem statement.
>>> 
>>> you might want to review:
>>> 
>>> http://wiki.apache.org/solr/UsingMailingLists
>>> 
>>> Best
>>> Erick
>>> 
>>> On Fri, Jul 15, 2011 at 1:52 PM, Brandon Fish<br...@gmail.com>
>>> wrote:
>>>> Does anyone know of any existing examples of importing data from a queue
>>>> into Solr?
>>>> 
>>>> Thank you.
>>>> 
>>> 
>>

Re: Data Import from a Queue

Posted by Stefan Matheis <ma...@googlemail.com>.
Brandon,

i don't know how they are using it in detail, but Part of Chef's 
Architecture is this one:

Chef Server -> RabbitMQ -> Chef Solr Indexer -> Solr
http://wiki.opscode.com/download/attachments/7274878/chef-server-arch.png

Perhaps not exactly, what you're looking for - but may give you an idea?

Regards
Stefan

Am 19.07.2011 19:04, schrieb Brandon Fish:
> Let me provide some more details to the question:
>
> I was unable to find any example implementations where individual documents
> (single document per message) are read from a message queue (like ActiveMQ
> or RabbitMQ) and then added to Solr via SolrJ, a HTTP POST or another
> method. Does anyone know of any available examples for this type of import?
>
> If no examples exist, what would be a recommended commit strategy for
> performance? My best guess for this would be to have a queue per core and
> commit once the queue is empty.
>
> Thanks.
>
> On Mon, Jul 18, 2011 at 6:52 PM, Erick Erickson<er...@gmail.com>wrote:
>
>> This is a really cryptic problem statement.
>>
>> you might want to review:
>>
>> http://wiki.apache.org/solr/UsingMailingLists
>>
>> Best
>> Erick
>>
>> On Fri, Jul 15, 2011 at 1:52 PM, Brandon Fish<br...@gmail.com>
>> wrote:
>>> Does anyone know of any existing examples of importing data from a queue
>>> into Solr?
>>>
>>> Thank you.
>>>
>>
>

Re: Data Import from a Queue

Posted by Brandon Fish <br...@gmail.com>.
Let me provide some more details to the question:

I was unable to find any example implementations where individual documents
(single document per message) are read from a message queue (like ActiveMQ
or RabbitMQ) and then added to Solr via SolrJ, a HTTP POST or another
method. Does anyone know of any available examples for this type of import?

If no examples exist, what would be a recommended commit strategy for
performance? My best guess for this would be to have a queue per core and
commit once the queue is empty.

Thanks.

On Mon, Jul 18, 2011 at 6:52 PM, Erick Erickson <er...@gmail.com>wrote:

> This is a really cryptic problem statement.
>
> you might want to review:
>
> http://wiki.apache.org/solr/UsingMailingLists
>
> Best
> Erick
>
> On Fri, Jul 15, 2011 at 1:52 PM, Brandon Fish <br...@gmail.com>
> wrote:
> > Does anyone know of any existing examples of importing data from a queue
> > into Solr?
> >
> > Thank you.
> >
>

Re: Data Import from a Queue

Posted by Erick Erickson <er...@gmail.com>.
This is a really cryptic problem statement.

you might want to review:

http://wiki.apache.org/solr/UsingMailingLists

Best
Erick

On Fri, Jul 15, 2011 at 1:52 PM, Brandon Fish <br...@gmail.com> wrote:
> Does anyone know of any existing examples of importing data from a queue
> into Solr?
>
> Thank you.
>