You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Claudio Atzori <cl...@isti.cnr.it> on 2010/03/31 12:51:36 UTC
Shred queries on EmbeddedSolrServer
In my application I need to create and destroy indexes via java code, so
to bypass the http requests I'm using the EmbeddedSolrServer, and I am
creating different SolrCore(s) one per every index I need.
Now the point is that a requirement of my application is the capability
to perfom a query on a specific index, on a subset of indexes, or on
every index.
I have been looking to the "shred" parameter:
<http://localhost:8080/solr/core1/select?shards=localhost:8080/solr/core1,localhost:8080/solr/core2&q=some>http://localhost:8080/solr/core1/select?shards=localhost:8080/solr/core1,localhost:8080/solr/core2&q=some
query...
...and ok, but my solr core instances doesn't expose an http interface,
so how can I shred a query between all my solr cores?
Thanks in advance,
Claudio
Re: Shred queries on EmbeddedSolrServer
Posted by Lance Norskog <go...@gmail.com>.
You can create and destroy cores over the HTTP interface:
http://www.lucidimagination.com/search/document/CDRG_ch08_8.2.5
But you are right, the Embedded Solr API does not support Distributed
Search across multiple cores. See:
org.apache.solr.handler.component.SearchHandler.submit() which very
definitly only does HTTP requests.
https://issues.apache.org/jira/browse/SOLR-1858 requests this feature.
On Wed, Mar 31, 2010 at 3:51 AM, Claudio Atzori
<cl...@isti.cnr.it> wrote:
> In my application I need to create and destroy indexes via java code, so to
> bypass the http requests I'm using the EmbeddedSolrServer, and I am creating
> different SolrCore(s) one per every index I need.
> Now the point is that a requirement of my application is the capability to
> perfom a query on a specific index, on a subset of indexes, or on every
> index.
>
> I have been looking to the "shred" parameter:
>
> <http://localhost:8080/solr/core1/select?shards=localhost:8080/solr/core1,localhost:8080/solr/core2&q=some>http://localhost:8080/solr/core1/select?shards=localhost:8080/solr/core1,localhost:8080/solr/core2&q=some
> query...
>
> ...and ok, but my solr core instances doesn't expose an http interface, so
> how can I shred a query between all my solr cores?
>
> Thanks in advance,
> Claudio
>
--
Lance Norskog
goksron@gmail.com
Re: deleteByQuery issue
Posted by Claudio Atzori <cl...@isti.cnr.it>.
Hi Eick, thanks for replying.
Yes I do commit after deleting-by-query, but since the IndexReader is an
internal (at lest for my app point of view), is there a way to reopen it
(or a new one) when I do a commit?
Claudio
On 10/11/2010 10:08 PM, Erick Erickson wrote:
> I'd guess that after you delete your documents and commit, you're still
> using
> an IndexReader that you haven't reopened when you search. WARNING:
> I'm not all that familiar with EmbeddedSolrServer, so this may be waaaaay
> off
> base.
>
> HTH
> Erick
>
> On Mon, Oct 11, 2010 at 12:04 PM, Claudio Atzori<claudio.atzori@isti.cnr.it
>> wrote:
>
>> On 10/11/2010 04:06 PM, Ahmet Arslan wrote:
>>
>>>
>>> --- On Mon, 10/11/10, Claudio Atzori<cl...@isti.cnr.it> wrote:
>>>
>>> From: Claudio Atzori<cl...@isti.cnr.it>
>>>> Subject: deleteByQuery issue
>>>> To: solr-user@lucene.apache.org
>>>> Date: Monday, October 11, 2010, 10:38 AM
>>>> Hi everybody,
>>>> in my application I use an instance of EmbeddedSolrServer
>>>> (solr 1.4.1), the following snippet shows how I am
>>>> instantiating it:
>>>>
>>>> File home = new
>>>>>
>>>> File(indexDataPath(solrDataDir, indexName));
>>>>
>>>>> container = new
>>>>>
>>>> CoreContainer(indexDataPath(solrDataDir, indexName));
>>>>
>>>>>
>>>>> container.load(indexDataPath(solrDataDir,
>>>> indexName), new File(home, "solr.xml"));
>>>>
>>>>> return new
>>>>>
>>>> EmbeddedSolrServer(container, indexName);
>>>>
>>>> and I'm going through some issues using deleteByQuery
>>>> method, in fact, when I try to delete a subset of documents,
>>>> or even all the documents from the index, I see as they are
>>>> correctly marked for deletion on the luke inspector (
>>>> http://code.google.com/p/luke/), but after a commit I
>>>> can still retrieve them, just like they haven't been
>>>> removed...
>>>>
>>>> I can see the difference and see the documents disappear
>>>> only when I restart my jetty application, but obviously this
>>>> cannot be a feature... any idea?
>>>>
>>> I think you are accessing same solr index using both embedded server and
>>> http.
>>> The changes that you made using embedded server won't be reflected to http
>>> until a commit issued from http. I mean if you hit this url:
>>>
>>> http://localhost:8983/solr/update?commit=true
>>>
>>> the deleted documents won't be retrieved anymore.
>>>
>>> P.s. if you want to expunge deleted docs completely you can either
>>> optimize or commit with expungeDeletes = "true".
>>>
>>>
>> Thanks for your reply.
>> Alright I'll better explain my scenario. I'm not exposing any http
>> interface of the index. I handle the whole index 'life cycle' via java code
>> with the EmbeddedSolrServer instance, so I'm handling commits,
>> optimizations, feedings, index creation, all through that instance, moreover
>> my client application calls embeddedSolrServerInstance.commit() after
>> deleteByQuery, but the documents are still there....
>>
>>
>
Re: deleteByQuery issue
Posted by Erick Erickson <er...@gmail.com>.
I'd guess that after you delete your documents and commit, you're still
using
an IndexReader that you haven't reopened when you search. WARNING:
I'm not all that familiar with EmbeddedSolrServer, so this may be waaaaay
off
base.
HTH
Erick
On Mon, Oct 11, 2010 at 12:04 PM, Claudio Atzori <claudio.atzori@isti.cnr.it
> wrote:
> On 10/11/2010 04:06 PM, Ahmet Arslan wrote:
>
>>
>> --- On Mon, 10/11/10, Claudio Atzori<cl...@isti.cnr.it> wrote:
>>
>> From: Claudio Atzori<cl...@isti.cnr.it>
>>> Subject: deleteByQuery issue
>>> To: solr-user@lucene.apache.org
>>> Date: Monday, October 11, 2010, 10:38 AM
>>> Hi everybody,
>>> in my application I use an instance of EmbeddedSolrServer
>>> (solr 1.4.1), the following snippet shows how I am
>>> instantiating it:
>>>
>>> File home = new
>>>>
>>> File(indexDataPath(solrDataDir, indexName));
>>>
>>>> container = new
>>>>
>>> CoreContainer(indexDataPath(solrDataDir, indexName));
>>>
>>>>
>>>> container.load(indexDataPath(solrDataDir,
>>> indexName), new File(home, "solr.xml"));
>>>
>>>> return new
>>>>
>>> EmbeddedSolrServer(container, indexName);
>>>
>>> and I'm going through some issues using deleteByQuery
>>> method, in fact, when I try to delete a subset of documents,
>>> or even all the documents from the index, I see as they are
>>> correctly marked for deletion on the luke inspector (
>>> http://code.google.com/p/luke/), but after a commit I
>>> can still retrieve them, just like they haven't been
>>> removed...
>>>
>>> I can see the difference and see the documents disappear
>>> only when I restart my jetty application, but obviously this
>>> cannot be a feature... any idea?
>>>
>> I think you are accessing same solr index using both embedded server and
>> http.
>> The changes that you made using embedded server won't be reflected to http
>> until a commit issued from http. I mean if you hit this url:
>>
>> http://localhost:8983/solr/update?commit=true
>>
>> the deleted documents won't be retrieved anymore.
>>
>> P.s. if you want to expunge deleted docs completely you can either
>> optimize or commit with expungeDeletes = "true".
>>
>>
> Thanks for your reply.
> Alright I'll better explain my scenario. I'm not exposing any http
> interface of the index. I handle the whole index 'life cycle' via java code
> with the EmbeddedSolrServer instance, so I'm handling commits,
> optimizations, feedings, index creation, all through that instance, moreover
> my client application calls embeddedSolrServerInstance.commit() after
> deleteByQuery, but the documents are still there....
>
>
Re: deleteByQuery issue
Posted by Claudio Atzori <cl...@isti.cnr.it>.
On 10/11/2010 04:06 PM, Ahmet Arslan wrote:
>
> --- On Mon, 10/11/10, Claudio Atzori<cl...@isti.cnr.it> wrote:
>
>> From: Claudio Atzori<cl...@isti.cnr.it>
>> Subject: deleteByQuery issue
>> To: solr-user@lucene.apache.org
>> Date: Monday, October 11, 2010, 10:38 AM
>> Hi everybody,
>> in my application I use an instance of EmbeddedSolrServer
>> (solr 1.4.1), the following snippet shows how I am
>> instantiating it:
>>
>>> File home = new
>> File(indexDataPath(solrDataDir, indexName));
>>> container = new
>> CoreContainer(indexDataPath(solrDataDir, indexName));
>>>
>> container.load(indexDataPath(solrDataDir,
>> indexName), new File(home, "solr.xml"));
>>> return new
>> EmbeddedSolrServer(container, indexName);
>>
>> and I'm going through some issues using deleteByQuery
>> method, in fact, when I try to delete a subset of documents,
>> or even all the documents from the index, I see as they are
>> correctly marked for deletion on the luke inspector (http://code.google.com/p/luke/), but after a commit I
>> can still retrieve them, just like they haven't been
>> removed...
>>
>> I can see the difference and see the documents disappear
>> only when I restart my jetty application, but obviously this
>> cannot be a feature... any idea?
> I think you are accessing same solr index using both embedded server and http.
> The changes that you made using embedded server won't be reflected to http until a commit issued from http. I mean if you hit this url:
>
> http://localhost:8983/solr/update?commit=true
>
> the deleted documents won't be retrieved anymore.
>
> P.s. if you want to expunge deleted docs completely you can either optimize or commit with expungeDeletes = "true".
>
Thanks for your reply.
Alright I'll better explain my scenario. I'm not exposing any http
interface of the index. I handle the whole index 'life cycle' via java
code with the EmbeddedSolrServer instance, so I'm handling commits,
optimizations, feedings, index creation, all through that instance,
moreover my client application calls embeddedSolrServerInstance.commit()
after deleteByQuery, but the documents are still there....
Re: deleteByQuery issue
Posted by Ahmet Arslan <io...@yahoo.com>.
--- On Mon, 10/11/10, Claudio Atzori <cl...@isti.cnr.it> wrote:
> From: Claudio Atzori <cl...@isti.cnr.it>
> Subject: deleteByQuery issue
> To: solr-user@lucene.apache.org
> Date: Monday, October 11, 2010, 10:38 AM
> Hi everybody,
> in my application I use an instance of EmbeddedSolrServer
> (solr 1.4.1), the following snippet shows how I am
> instantiating it:
>
> > File home = new
> File(indexDataPath(solrDataDir, indexName));
> >
> > container = new
> CoreContainer(indexDataPath(solrDataDir, indexName));
> >
> container.load(indexDataPath(solrDataDir,
> indexName), new File(home, "solr.xml"));
> >
> > return new
> EmbeddedSolrServer(container, indexName);
>
> and I'm going through some issues using deleteByQuery
> method, in fact, when I try to delete a subset of documents,
> or even all the documents from the index, I see as they are
> correctly marked for deletion on the luke inspector (http://code.google.com/p/luke/), but after a commit I
> can still retrieve them, just like they haven't been
> removed...
>
> I can see the difference and see the documents disappear
> only when I restart my jetty application, but obviously this
> cannot be a feature... any idea?
I think you are accessing same solr index using both embedded server and http.
The changes that you made using embedded server won't be reflected to http until a commit issued from http. I mean if you hit this url:
http://localhost:8983/solr/update?commit=true
the deleted documents won't be retrieved anymore.
P.s. if you want to expunge deleted docs completely you can either optimize or commit with expungeDeletes = "true".
deleteByQuery issue
Posted by Claudio Atzori <cl...@isti.cnr.it>.
Hi everybody,
in my application I use an instance of EmbeddedSolrServer (solr 1.4.1),
the following snippet shows how I am instantiating it:
> File home = new File(indexDataPath(solrDataDir, indexName));
>
> container = new CoreContainer(indexDataPath(solrDataDir,
> indexName));
> container.load(indexDataPath(solrDataDir, indexName), new
> File(home, "solr.xml"));
>
> return new EmbeddedSolrServer(container, indexName);
and I'm going through some issues using deleteByQuery method, in fact,
when I try to delete a subset of documents, or even all the documents
from the index, I see as they are correctly marked for deletion on the
luke inspector (http://code.google.com/p/luke/), but after a commit I
can still retrieve them, just like they haven't been removed...
I can see the difference and see the documents disappear only when I
restart my jetty application, but obviously this cannot be a feature...
any idea?