You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Tri Nguyen <tr...@yahoo.com> on 2010/10/13 23:51:40 UTC
searching while importing
Hi,
Can I perform searches against the index while it is being imported?
Does importing add 1 document at a time or will solr make a temporary index and
switch to that index when indexing is done?
Thanks,
Tri
Re: searching while importing
Posted by "scott chu (朱炎詹)" <sc...@udngroup.com>.
You can build 2 similiar Solr cores. One for service & one for importing.
When the importing is done, you can do either MERGE or SWAP actions,
depending on how you put your data on these 2 cores.
----- Original Message -----
From: "Tri Nguyen" <tr...@yahoo.com>
To: <so...@lucene.apache.org>
Sent: Thursday, October 14, 2010 5:51 AM
Subject: searching while importing
Hi,
Can I perform searches against the index while it is being imported?
Does importing add 1 document at a time or will solr make a temporary index
and
switch to that index when indexing is done?
Thanks,
Tri
--------------------------------------------------------------------------------
%<&b6G$J0T.'$$'d(l/f,r!C
Checked by AVG - www.avg.com
Version: 9.0.862 / Virus Database: 271.1.1/3193 - Release Date: 10/13/10
02:37:00
Re: searching while importing
Posted by Shawn Heisey <so...@elyograg.org>.
If I haven't deleted the index for some reason before doing the full
import, then I can search the old data.
On 10/13/2010 4:41 PM, Tri Nguyen wrote:
> Hi,
>
> As long as I can search on the current ("older") index while importing, I'm
> good. I've tested this and I can search the older index while data-importing
> the newer index.
>
> So you can search the older index in your 5 hour wait?
Re: searching while importing
Posted by Shawn Heisey <el...@elyograg.org>.
If I haven't deleted the index for some reason before doing the full
import, then I can search the old data.
On 10/13/2010 4:41 PM, Tri Nguyen wrote:
> Hi,
>
> As long as I can search on the current ("older") index while importing, I'm
> good. I've tested this and I can search the older index while data-importing
> the newer index.
>
> So you can search the older index in your 5 hour wait?
Re: searching while importing
Posted by Tri Nguyen <tr...@yahoo.com>.
Hi,
As long as I can search on the current ("older") index while importing, I'm
good. I've tested this and I can search the older index while data-importing
the newer index.
So you can search the older index in your 5 hour wait?
Thanks,
Tri
________________________________
From: Shawn Heisey <so...@elyograg.org>
To: solr-user@lucene.apache.org
Sent: Wed, October 13, 2010 3:38:48 PM
Subject: Re: searching while importing
If you are using the DataImportHandler, you will not be able to search new data
until the full-import or delta-import is complete and the update is committed.
When I do a full reindex, it takes about 5 hours, and until it is finished, I
cannot search it.
I have not tried to issue a manual commit in the middle of an import to see
whether that makes data inserted up to that point searchable, but I would not
expect that to work.
If you need this kind of functionality, you may need to change your build system
so that a full import clears the index manually and then does a series of
delta-import batches.
On 10/13/2010 3:51 PM, Tri Nguyen wrote:
> Hi,
> Can I perform searches against the index while it is being imported?
> Does importing add 1 document at a time or will solr make a temporary index
>and
> switch to that index when indexing is done?
> Thanks,
> Tri
Re: searching while importing
Posted by Ken Stanley <do...@gmail.com>.
On Wed, Oct 13, 2010 at 6:38 PM, Shawn Heisey <so...@elyograg.org> wrote:
> If you are using the DataImportHandler, you will not be able to search new
> data until the full-import or delta-import is complete and the update is
> committed. When I do a full reindex, it takes about 5 hours, and until it
> is finished, I cannot search it.
>
> This is not true; when I use the DIH to do a full-import, I (and my team)
are still able to search on the already-indexed data that exists.
> I have not tried to issue a manual commit in the middle of an import to see
> whether that makes data inserted up to that point searchable, but I would
> not expect that to work.
>
> If you set the autoCommit properties maxDocs and maxTime to reasonable
values, then once those limits are reached, I suspect that SOLR would commit
and continue indexing; however, I have not had the chance to use those
features in solrconfig.xml.
> If you need this kind of functionality, you may need to change your build
> system so that a full import clears the index manually and then does a
> series of delta-import batches.
>
> The only time I've had an issue with being able to search while indexing is
when my DIH had mis-configuration that caused the import to finish without
indexing anything, thus wiping out my data. Aside of that, I continually
index and search at the same time almost every day (using 1.4.1).
>
>
> On 10/13/2010 3:51 PM, Tri Nguyen wrote:
>
>> Hi,
>> Can I perform searches against the index while it is being imported?
>> Does importing add 1 document at a time or will solr make a temporary
>> index and
>> switch to that index when indexing is done?
>> Thanks,
>> Tri
>>
>
>
Re: searching while importing
Posted by Gora Mohanty <go...@mimirtech.com>.
On Thu, Oct 14, 2010 at 4:08 AM, Shawn Heisey <so...@elyograg.org> wrote:
> If you are using the DataImportHandler, you will not be able to search new
> data until the full-import or delta-import is complete and the update is
> committed. When I do a full reindex, it takes about 5 hours, and until it
> is finished, I cannot search it.
>
> I have not tried to issue a manual commit in the middle of an import to see
> whether that makes data inserted up to that point searchable, but I would
> not expect that to work.
[...]
Just as a data point, we have done this, and yes it is possible to do a commit
in the middle of an import, and have the documents that have already been
indexed be available for search.
Regards,
Gora
Re: searching while importing
Posted by Shawn Heisey <so...@elyograg.org>.
If you are using the DataImportHandler, you will not be able to search
new data until the full-import or delta-import is complete and the
update is committed. When I do a full reindex, it takes about 5 hours,
and until it is finished, I cannot search it.
I have not tried to issue a manual commit in the middle of an import to
see whether that makes data inserted up to that point searchable, but I
would not expect that to work.
If you need this kind of functionality, you may need to change your
build system so that a full import clears the index manually and then
does a series of delta-import batches.
On 10/13/2010 3:51 PM, Tri Nguyen wrote:
> Hi,
>
> Can I perform searches against the index while it is being imported?
>
> Does importing add 1 document at a time or will solr make a temporary index and
> switch to that index when indexing is done?
>
> Thanks,
>
> Tri