You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Furkan KAMACI <fu...@gmail.com> on 2014/05/26 18:57:15 UTC
How to Configure Solr For Test Purposes?
Hi;
I run Solr within my Test Suite. I delete documents or atomically update
them and check whether if it works or not. I know that I have to setup a
hard/soft commit timing for my test Solr. However even I have that settings:
<autoCommit>
<maxTime>1</maxTime>
<openSearcher>true</openSearcher>
</autoCommit>
<autoSoftCommit>
<maxTime>1</maxTime>
</autoSoftCommit>
and even I wait (Thread.sleep()) for a time to wait Solr *sometimes* my
tests are failed. I get fail error even I increase wait time. Example of a
sometimes failed code piece:
for (int i = 0; i < dummyDocumentSize; i++) {
deleteById("id" + i);
dummyDocumentSize--;
queryResponse = query(solrParams);
assertTrue(queryResponse.getResults().size() == dummyDocumentSize);
}
at debug mode if I wait for Solr to reflect changes I see that I do not get
error. What do you think, what kind of configuration I should have for such
kind of purposes?
Thanks;
Furkan KAMACI
Re: How to Configure Solr For Test Purposes?
Posted by Furkan KAMACI <fu...@gmail.com>.
Hi;
I've developed a Proxy application that takes request from clients and
sends them to Solr then gets response and sends them to client as response.
So, I am testing my application, my proxy for Solr.
Thanks;
Furkan KAMACI
2014-05-27 14:52 GMT+03:00 Tomás Fernández Löbbe <to...@gmail.com>:
> > What do you suggest for my purpose? If a test case fails re-running it
> for
> > some times maybe a solution? What kind of configuration do you suggest
> for
> > my Solr configuration?
> >
> >
> From the snippet of test that you showed, it looks like it's testing only
> Solr functionality. So, first make sure this is a test that you really
> need. Solr has it's own tests, and it you feel it could use more (for some
> specific case or context), I'd open a Jira and try to get the test inside
> Solr.
> If my impression is wrong and your test is actually testing your code, then
> I'd suggest you to use a specific soft commit call with waitSearcher = true
> on your test instead of relying on the autocommit (and remove the
> autocommit completely from your solrconfig).
>
> Tomás
>
>
>
> Thanks;
> > Furkan KAMACI
> > 26 May 2014 21:03 tarihinde "Shawn Heisey" <so...@elyograg.org> yazdı:
> >
> > > On 5/26/2014 10:57 AM, Furkan KAMACI wrote:
> > > > Hi;
> > > >
> > > > I run Solr within my Test Suite. I delete documents or atomically
> > update
> > > > them and check whether if it works or not. I know that I have to
> setup
> > a
> > > > hard/soft commit timing for my test Solr. However even I have that
> > > settings:
> > > >
> > > > <autoCommit>
> > > > <maxTime>1</maxTime>
> > > > <openSearcher>true</openSearcher>
> > > > </autoCommit>
> > > >
> > > > <autoSoftCommit>
> > > > <maxTime>1</maxTime>
> > > > </autoSoftCommit>
> > >
> > > I hope you know that this is BAD configuration. Doing automatic
> commits
> > > on an interval of 1 millisecond is asking for a whole host of problems.
> > > In some cases, this could do a commit after every single document that
> > > is indexed, which is NOT recommended at all. The openSearcher setting
> > > of "true" on autoCommit makes it even worse. There's no reason to do
> > > both autoSoftCommit and autoCommit with openSearcher=true. I don't
> know
> > > which one "wins" between autoCommit and autoSoftCommit if they both
> have
> > > the same config, but I would guess the hard commit does.
> > >
> > > > and even I wait (Thread.sleep()) for a time to wait Solr *sometimes*
> my
> > > > tests are failed. I get fail error even I increase wait time.
> Example
> > > of a
> > > > sometimes failed code piece:
> > > >
> > > > for (int i = 0; i < dummyDocumentSize; i++) {
> > > > deleteById("id" + i);
> > > > dummyDocumentSize--;
> > > > queryResponse = query(solrParams);
> > > > assertTrue(queryResponse.getResults().size() ==
> > > dummyDocumentSize);
> > > > }
> > > >
> > > > at debug mode if I wait for Solr to reflect changes I see that I do
> not
> > > get
> > > > error. What do you think, what kind of configuration I should have
> for
> > > such
> > > > kind of purposes?
> > >
> > > Chances are that commits are going to take longer than 1 millisecond.
> > > If you're actively indexing, the system is going to be trying to stack
> > > up lots of commits at the same time. The maxWarmingSearchers value
> will
> > > limit the number of new searchers that can be opened, but it will not
> > > stop the commits themselves. When lots of commits are going on, each
> > > one will take *even longer* to complete, which probably explains the
> > > problem.
> > >
> > > Thanks,
> > > Shawn
> > >
> > >
> >
>
Re: How to Configure Solr For Test Purposes?
Posted by Tomás Fernández Löbbe <to...@gmail.com>.
> What do you suggest for my purpose? If a test case fails re-running it for
> some times maybe a solution? What kind of configuration do you suggest for
> my Solr configuration?
>
>
>From the snippet of test that you showed, it looks like it's testing only
Solr functionality. So, first make sure this is a test that you really
need. Solr has it's own tests, and it you feel it could use more (for some
specific case or context), I'd open a Jira and try to get the test inside
Solr.
If my impression is wrong and your test is actually testing your code, then
I'd suggest you to use a specific soft commit call with waitSearcher = true
on your test instead of relying on the autocommit (and remove the
autocommit completely from your solrconfig).
Tomás
Thanks;
> Furkan KAMACI
> 26 May 2014 21:03 tarihinde "Shawn Heisey" <so...@elyograg.org> yazdı:
>
> > On 5/26/2014 10:57 AM, Furkan KAMACI wrote:
> > > Hi;
> > >
> > > I run Solr within my Test Suite. I delete documents or atomically
> update
> > > them and check whether if it works or not. I know that I have to setup
> a
> > > hard/soft commit timing for my test Solr. However even I have that
> > settings:
> > >
> > > <autoCommit>
> > > <maxTime>1</maxTime>
> > > <openSearcher>true</openSearcher>
> > > </autoCommit>
> > >
> > > <autoSoftCommit>
> > > <maxTime>1</maxTime>
> > > </autoSoftCommit>
> >
> > I hope you know that this is BAD configuration. Doing automatic commits
> > on an interval of 1 millisecond is asking for a whole host of problems.
> > In some cases, this could do a commit after every single document that
> > is indexed, which is NOT recommended at all. The openSearcher setting
> > of "true" on autoCommit makes it even worse. There's no reason to do
> > both autoSoftCommit and autoCommit with openSearcher=true. I don't know
> > which one "wins" between autoCommit and autoSoftCommit if they both have
> > the same config, but I would guess the hard commit does.
> >
> > > and even I wait (Thread.sleep()) for a time to wait Solr *sometimes* my
> > > tests are failed. I get fail error even I increase wait time. Example
> > of a
> > > sometimes failed code piece:
> > >
> > > for (int i = 0; i < dummyDocumentSize; i++) {
> > > deleteById("id" + i);
> > > dummyDocumentSize--;
> > > queryResponse = query(solrParams);
> > > assertTrue(queryResponse.getResults().size() ==
> > dummyDocumentSize);
> > > }
> > >
> > > at debug mode if I wait for Solr to reflect changes I see that I do not
> > get
> > > error. What do you think, what kind of configuration I should have for
> > such
> > > kind of purposes?
> >
> > Chances are that commits are going to take longer than 1 millisecond.
> > If you're actively indexing, the system is going to be trying to stack
> > up lots of commits at the same time. The maxWarmingSearchers value will
> > limit the number of new searchers that can be opened, but it will not
> > stop the commits themselves. When lots of commits are going on, each
> > one will take *even longer* to complete, which probably explains the
> > problem.
> >
> > Thanks,
> > Shawn
> >
> >
>
Re: How to Configure Solr For Test Purposes?
Posted by Furkan KAMACI <fu...@gmail.com>.
Hi All;
I have defined just that:
<autoSoftCommit>
<maxTime>1</maxTime>
</autoSoftCommit>
then turned of hard commit. I've run my tests time and time again and did
not get any error. Who wants to write a unit test that interacts with Solr
as like my situation can use it.
Thanks;
Furkan KAMACI
2014-05-26 23:37 GMT+03:00 Furkan KAMACI <fu...@gmail.com>:
> Hi Shawn;
>
> I know that it is a bad practise but I just commit up to 5 documents and
> there will not be more than 5 documents at any time at any test method. It
> is just for test purpose to see that my API works. I want to have
> automatic tests.
>
> What do you suggest for my purpose? If a test case fails re-running it for
> some times maybe a solution? What kind of configuration do you suggest for
> my Solr configuration?
>
> Thanks;
> Furkan KAMACI
> 26 May 2014 21:03 tarihinde "Shawn Heisey" <so...@elyograg.org> yazdı:
>
> On 5/26/2014 10:57 AM, Furkan KAMACI wrote:
>> > Hi;
>> >
>> > I run Solr within my Test Suite. I delete documents or atomically update
>> > them and check whether if it works or not. I know that I have to setup a
>> > hard/soft commit timing for my test Solr. However even I have that
>> settings:
>> >
>> > <autoCommit>
>> > <maxTime>1</maxTime>
>> > <openSearcher>true</openSearcher>
>> > </autoCommit>
>> >
>> > <autoSoftCommit>
>> > <maxTime>1</maxTime>
>> > </autoSoftCommit>
>>
>> I hope you know that this is BAD configuration. Doing automatic commits
>> on an interval of 1 millisecond is asking for a whole host of problems.
>> In some cases, this could do a commit after every single document that
>> is indexed, which is NOT recommended at all. The openSearcher setting
>> of "true" on autoCommit makes it even worse. There's no reason to do
>> both autoSoftCommit and autoCommit with openSearcher=true. I don't know
>> which one "wins" between autoCommit and autoSoftCommit if they both have
>> the same config, but I would guess the hard commit does.
>>
>> > and even I wait (Thread.sleep()) for a time to wait Solr *sometimes* my
>> > tests are failed. I get fail error even I increase wait time. Example
>> of a
>> > sometimes failed code piece:
>> >
>> > for (int i = 0; i < dummyDocumentSize; i++) {
>> > deleteById("id" + i);
>> > dummyDocumentSize--;
>> > queryResponse = query(solrParams);
>> > assertTrue(queryResponse.getResults().size() ==
>> dummyDocumentSize);
>> > }
>> >
>> > at debug mode if I wait for Solr to reflect changes I see that I do not
>> get
>> > error. What do you think, what kind of configuration I should have for
>> such
>> > kind of purposes?
>>
>> Chances are that commits are going to take longer than 1 millisecond.
>> If you're actively indexing, the system is going to be trying to stack
>> up lots of commits at the same time. The maxWarmingSearchers value will
>> limit the number of new searchers that can be opened, but it will not
>> stop the commits themselves. When lots of commits are going on, each
>> one will take *even longer* to complete, which probably explains the
>> problem.
>>
>> Thanks,
>> Shawn
>>
>>
Re: How to Configure Solr For Test Purposes?
Posted by Furkan KAMACI <fu...@gmail.com>.
Hi Shawn;
I know that it is a bad practise but I just commit up to 5 documents and
there will not be more than 5 documents at any time at any test method. It
is just for test purpose to see that my API works. I want to have
automatic tests.
What do you suggest for my purpose? If a test case fails re-running it for
some times maybe a solution? What kind of configuration do you suggest for
my Solr configuration?
Thanks;
Furkan KAMACI
26 May 2014 21:03 tarihinde "Shawn Heisey" <so...@elyograg.org> yazdı:
> On 5/26/2014 10:57 AM, Furkan KAMACI wrote:
> > Hi;
> >
> > I run Solr within my Test Suite. I delete documents or atomically update
> > them and check whether if it works or not. I know that I have to setup a
> > hard/soft commit timing for my test Solr. However even I have that
> settings:
> >
> > <autoCommit>
> > <maxTime>1</maxTime>
> > <openSearcher>true</openSearcher>
> > </autoCommit>
> >
> > <autoSoftCommit>
> > <maxTime>1</maxTime>
> > </autoSoftCommit>
>
> I hope you know that this is BAD configuration. Doing automatic commits
> on an interval of 1 millisecond is asking for a whole host of problems.
> In some cases, this could do a commit after every single document that
> is indexed, which is NOT recommended at all. The openSearcher setting
> of "true" on autoCommit makes it even worse. There's no reason to do
> both autoSoftCommit and autoCommit with openSearcher=true. I don't know
> which one "wins" between autoCommit and autoSoftCommit if they both have
> the same config, but I would guess the hard commit does.
>
> > and even I wait (Thread.sleep()) for a time to wait Solr *sometimes* my
> > tests are failed. I get fail error even I increase wait time. Example
> of a
> > sometimes failed code piece:
> >
> > for (int i = 0; i < dummyDocumentSize; i++) {
> > deleteById("id" + i);
> > dummyDocumentSize--;
> > queryResponse = query(solrParams);
> > assertTrue(queryResponse.getResults().size() ==
> dummyDocumentSize);
> > }
> >
> > at debug mode if I wait for Solr to reflect changes I see that I do not
> get
> > error. What do you think, what kind of configuration I should have for
> such
> > kind of purposes?
>
> Chances are that commits are going to take longer than 1 millisecond.
> If you're actively indexing, the system is going to be trying to stack
> up lots of commits at the same time. The maxWarmingSearchers value will
> limit the number of new searchers that can be opened, but it will not
> stop the commits themselves. When lots of commits are going on, each
> one will take *even longer* to complete, which probably explains the
> problem.
>
> Thanks,
> Shawn
>
>
Re: How to Configure Solr For Test Purposes?
Posted by Shawn Heisey <so...@elyograg.org>.
On 5/26/2014 10:57 AM, Furkan KAMACI wrote:
> Hi;
>
> I run Solr within my Test Suite. I delete documents or atomically update
> them and check whether if it works or not. I know that I have to setup a
> hard/soft commit timing for my test Solr. However even I have that settings:
>
> <autoCommit>
> <maxTime>1</maxTime>
> <openSearcher>true</openSearcher>
> </autoCommit>
>
> <autoSoftCommit>
> <maxTime>1</maxTime>
> </autoSoftCommit>
I hope you know that this is BAD configuration. Doing automatic commits
on an interval of 1 millisecond is asking for a whole host of problems.
In some cases, this could do a commit after every single document that
is indexed, which is NOT recommended at all. The openSearcher setting
of "true" on autoCommit makes it even worse. There's no reason to do
both autoSoftCommit and autoCommit with openSearcher=true. I don't know
which one "wins" between autoCommit and autoSoftCommit if they both have
the same config, but I would guess the hard commit does.
> and even I wait (Thread.sleep()) for a time to wait Solr *sometimes* my
> tests are failed. I get fail error even I increase wait time. Example of a
> sometimes failed code piece:
>
> for (int i = 0; i < dummyDocumentSize; i++) {
> deleteById("id" + i);
> dummyDocumentSize--;
> queryResponse = query(solrParams);
> assertTrue(queryResponse.getResults().size() == dummyDocumentSize);
> }
>
> at debug mode if I wait for Solr to reflect changes I see that I do not get
> error. What do you think, what kind of configuration I should have for such
> kind of purposes?
Chances are that commits are going to take longer than 1 millisecond.
If you're actively indexing, the system is going to be trying to stack
up lots of commits at the same time. The maxWarmingSearchers value will
limit the number of new searchers that can be opened, but it will not
stop the commits themselves. When lots of commits are going on, each
one will take *even longer* to complete, which probably explains the
problem.
Thanks,
Shawn