You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Pratik Patel <pr...@semandex.net> on 2019/05/10 20:35:54 UTC
Writing unit tests to test complex solr queries
Hello Everyone,
I want to write unit tests for some solr queries which are being triggered
through java code. These queries includes complex streaming expressions and
faceting queries which requires large number of documents to be present in
solr index. I can not create and push so many documents programmatically
through my tests.
I am trying to find a way to test these queries without depending on
externally running solr instance. I found following approach which is using
classes like EmbeddedSolrServer and CoreContainer. We can put index files
and solr configuration on classpath and run the tests against them.
https://dzone.com/articles/junit-testing-for-solr-6
However, this seems to be an old approach and I am trying to find a way to
do it using latest solr-test-framework. I also can not use old approach
because I want to test Streaming Expressions as well and I need
SolrCloudClient for that.
In solr-test-framework, I found MiniSolrCloudCluster class but I don't know
how to use pre-created index files and configuration with that.
Does anyone know how we can use pre-created index files and configuration
with latest test-framework? What is the recommended way to do such kind of
testing? Any direction with this would be really helpful.
Thanks!
Pratik
Re: Writing unit tests to test complex solr queries
Posted by Pratik Patel <pr...@semandex.net>.
Thanks a lot for the response Mikhail and Angie!
I did go through most of the test classes in solr before posting here but
couldn't find anything which is close to what I want to do which is to load
pre-created index files and configuration or at least index files.
However, the class HelloWorldSolrCloudTestCase.java class pointed out by
Angie put together with his code that he has shared seems to be completing
the picture and looks spot on! Thanks a lot.
I will try to re-write my unit tests with this approach and will post an
update soon.
@Angie, can you please share the format of data in your
"testdata/test-data.json" file? I want to be sure about using the correct
format.
Thanks!
Pratik
On Tue, May 14, 2019 at 1:14 PM Angie Rabelero <ma...@oracle.com>
wrote:
> Hi, I’ll advised you to extend the class SolrCloudTestCase, which extends
> the MiniSolrCloudCluster. Theres a hello world example in the solr source
> at
> https://github.com/apache/lucene-solr/blob/master/solr/core/src/test/org/apache/solr/HelloWorldSolrCloudTestCase.java
> .
>
> Here’s how I setup a cluster, create a collection with my ConfigSet, and
> index a file.
>
> @BeforeClass
> public static void setupCluster() throws Exception {
>
> // Create and configure cluster
> configureCluster(nodeCount)
> .addConfig(CONFIG_NAME, getFile(CONFIG_DIR).toPath())
> .configure();
>
> // Create an empty collection
> Create.createCollection(COLLECTION, CONFIG_NAME, numShards,
> numReplicas)
> .setMaxShardsPerNode(maxShardsPerNode)
> .process(cluster.getSolrClient(), COLLECTION);
> AbstractDistribZkTestBase
> .waitForRecoveriesToFinish(COLLECTION,
> cluster.getSolrClient().getZkStateReader(), true, true, 120);
>
> // Set default collection
> cluster.getSolrClient().setDefaultCollection(COLLECTION);
>
> // Add documents to collection
> ContentStreamUpdateRequest up = new
> ContentStreamUpdateRequest("/update");
> up.addFile(getFile("testdata/test-data.json"), "application/json");
> up.setAction(AbstractUpdateRequest.ACTION.COMMIT, true, true);
> NamedList<Object> result = cluster.getSolrClient().request(up);
>
> // Print cluster status
> System.out.println("Default Collection: " +
> cluster.getSolrClient().getDefaultCollection());
> System.out.println("Cluster State: " +
> cluster.getSolrClient().getZkStateReader().getClusterState());
> System.out.println("Update Result: " + result);
>
> }
>
> I copy the configset to the resources dir in the pom using a mauven
> plugin. And the test file is already in the resources dir.
>
>
>
>
> > On May 14, 2019, at 04:01, Mikhail Khludnev <mk...@apache.org> wrote:
> >
> > Hello, Pratick.
> > Welcome to mysterious world of Solr testing. The best way is to find
> > existing test closest to your problem field, copy in and amend
> necessarily.
> > What about
> >
> https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_lucene-2Dsolr_blob_master_solr_solrj_src_test_org_apache_solr_client_solrj_io_stream_StreamExpressionTest.java&d=DwIBaQ&c=RoP1YumCXCgaWHvlZYR8PZh8Bv7qIrMUB65eapI_JnE&r=lUsTzFRk0CX38HvagQ0wd52D67dA0fx_D6M6F3LHzAU&m=9tFliF4KA1tiG2lGmDJWO34hyq9-Sz1inAxRPVKkz78&s=KjveDzxzQAKRmvzPYk2y1FQ-w6yAGWuwfTVGHMQP2ZA&e=
> > ?
> >
> > On Fri, May 10, 2019 at 11:36 PM Pratik Patel <pr...@semandex.net>
> wrote:
> >
> >> Hello Everyone,
> >>
> >> I want to write unit tests for some solr queries which are being
> triggered
> >> through java code. These queries includes complex streaming expressions
> and
> >> faceting queries which requires large number of documents to be present
> in
> >> solr index. I can not create and push so many documents programmatically
> >> through my tests.
> >>
> >> I am trying to find a way to test these queries without depending on
> >> externally running solr instance. I found following approach which is
> using
> >> classes like EmbeddedSolrServer and CoreContainer. We can put index
> files
> >> and solr configuration on classpath and run the tests against them.
> >>
> >>
> https://urldefense.proofpoint.com/v2/url?u=https-3A__dzone.com_articles_junit-2Dtesting-2Dfor-2Dsolr-2D6&d=DwIBaQ&c=RoP1YumCXCgaWHvlZYR8PZh8Bv7qIrMUB65eapI_JnE&r=lUsTzFRk0CX38HvagQ0wd52D67dA0fx_D6M6F3LHzAU&m=9tFliF4KA1tiG2lGmDJWO34hyq9-Sz1inAxRPVKkz78&s=K4vPwvz9h9H8s-nsZTbkmCvTh002RP3CHcpbb9IOrpw&e=
> >>
> >> However, this seems to be an old approach and I am trying to find a way
> to
> >> do it using latest solr-test-framework. I also can not use old approach
> >> because I want to test Streaming Expressions as well and I need
> >> SolrCloudClient for that.
> >> In solr-test-framework, I found MiniSolrCloudCluster class but I don't
> know
> >> how to use pre-created index files and configuration with that.
> >>
> >> Does anyone know how we can use pre-created index files and
> configuration
> >> with latest test-framework? What is the recommended way to do such kind
> of
> >> testing? Any direction with this would be really helpful.
> >>
> >> Thanks!
> >> Pratik
> >>
> >
> >
> > --
> > Sincerely yours
> > Mikhail Khludnev
>
>
Re: Writing unit tests to test complex solr queries
Posted by Angie Rabelero <ma...@oracle.com>.
Hi, I’ll advised you to extend the class SolrCloudTestCase, which extends the MiniSolrCloudCluster. Theres a hello world example in the solr source at https://github.com/apache/lucene-solr/blob/master/solr/core/src/test/org/apache/solr/HelloWorldSolrCloudTestCase.java.
Here’s how I setup a cluster, create a collection with my ConfigSet, and index a file.
@BeforeClass
public static void setupCluster() throws Exception {
// Create and configure cluster
configureCluster(nodeCount)
.addConfig(CONFIG_NAME, getFile(CONFIG_DIR).toPath())
.configure();
// Create an empty collection
Create.createCollection(COLLECTION, CONFIG_NAME, numShards, numReplicas)
.setMaxShardsPerNode(maxShardsPerNode)
.process(cluster.getSolrClient(), COLLECTION);
AbstractDistribZkTestBase
.waitForRecoveriesToFinish(COLLECTION, cluster.getSolrClient().getZkStateReader(), true, true, 120);
// Set default collection
cluster.getSolrClient().setDefaultCollection(COLLECTION);
// Add documents to collection
ContentStreamUpdateRequest up = new ContentStreamUpdateRequest("/update");
up.addFile(getFile("testdata/test-data.json"), "application/json");
up.setAction(AbstractUpdateRequest.ACTION.COMMIT, true, true);
NamedList<Object> result = cluster.getSolrClient().request(up);
// Print cluster status
System.out.println("Default Collection: " + cluster.getSolrClient().getDefaultCollection());
System.out.println("Cluster State: " + cluster.getSolrClient().getZkStateReader().getClusterState());
System.out.println("Update Result: " + result);
}
I copy the configset to the resources dir in the pom using a mauven plugin. And the test file is already in the resources dir.
> On May 14, 2019, at 04:01, Mikhail Khludnev <mk...@apache.org> wrote:
>
> Hello, Pratick.
> Welcome to mysterious world of Solr testing. The best way is to find
> existing test closest to your problem field, copy in and amend necessarily.
> What about
> https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_lucene-2Dsolr_blob_master_solr_solrj_src_test_org_apache_solr_client_solrj_io_stream_StreamExpressionTest.java&d=DwIBaQ&c=RoP1YumCXCgaWHvlZYR8PZh8Bv7qIrMUB65eapI_JnE&r=lUsTzFRk0CX38HvagQ0wd52D67dA0fx_D6M6F3LHzAU&m=9tFliF4KA1tiG2lGmDJWO34hyq9-Sz1inAxRPVKkz78&s=KjveDzxzQAKRmvzPYk2y1FQ-w6yAGWuwfTVGHMQP2ZA&e=
> ?
>
> On Fri, May 10, 2019 at 11:36 PM Pratik Patel <pr...@semandex.net> wrote:
>
>> Hello Everyone,
>>
>> I want to write unit tests for some solr queries which are being triggered
>> through java code. These queries includes complex streaming expressions and
>> faceting queries which requires large number of documents to be present in
>> solr index. I can not create and push so many documents programmatically
>> through my tests.
>>
>> I am trying to find a way to test these queries without depending on
>> externally running solr instance. I found following approach which is using
>> classes like EmbeddedSolrServer and CoreContainer. We can put index files
>> and solr configuration on classpath and run the tests against them.
>>
>> https://urldefense.proofpoint.com/v2/url?u=https-3A__dzone.com_articles_junit-2Dtesting-2Dfor-2Dsolr-2D6&d=DwIBaQ&c=RoP1YumCXCgaWHvlZYR8PZh8Bv7qIrMUB65eapI_JnE&r=lUsTzFRk0CX38HvagQ0wd52D67dA0fx_D6M6F3LHzAU&m=9tFliF4KA1tiG2lGmDJWO34hyq9-Sz1inAxRPVKkz78&s=K4vPwvz9h9H8s-nsZTbkmCvTh002RP3CHcpbb9IOrpw&e=
>>
>> However, this seems to be an old approach and I am trying to find a way to
>> do it using latest solr-test-framework. I also can not use old approach
>> because I want to test Streaming Expressions as well and I need
>> SolrCloudClient for that.
>> In solr-test-framework, I found MiniSolrCloudCluster class but I don't know
>> how to use pre-created index files and configuration with that.
>>
>> Does anyone know how we can use pre-created index files and configuration
>> with latest test-framework? What is the recommended way to do such kind of
>> testing? Any direction with this would be really helpful.
>>
>> Thanks!
>> Pratik
>>
>
>
> --
> Sincerely yours
> Mikhail Khludnev
Re: Writing unit tests to test complex solr queries
Posted by Mikhail Khludnev <mk...@apache.org>.
Hello, Pratick.
Welcome to mysterious world of Solr testing. The best way is to find
existing test closest to your problem field, copy in and amend necessarily.
What about
https://github.com/apache/lucene-solr/blob/master/solr/solrj/src/test/org/apache/solr/client/solrj/io/stream/StreamExpressionTest.java
?
On Fri, May 10, 2019 at 11:36 PM Pratik Patel <pr...@semandex.net> wrote:
> Hello Everyone,
>
> I want to write unit tests for some solr queries which are being triggered
> through java code. These queries includes complex streaming expressions and
> faceting queries which requires large number of documents to be present in
> solr index. I can not create and push so many documents programmatically
> through my tests.
>
> I am trying to find a way to test these queries without depending on
> externally running solr instance. I found following approach which is using
> classes like EmbeddedSolrServer and CoreContainer. We can put index files
> and solr configuration on classpath and run the tests against them.
>
> https://dzone.com/articles/junit-testing-for-solr-6
>
> However, this seems to be an old approach and I am trying to find a way to
> do it using latest solr-test-framework. I also can not use old approach
> because I want to test Streaming Expressions as well and I need
> SolrCloudClient for that.
> In solr-test-framework, I found MiniSolrCloudCluster class but I don't know
> how to use pre-created index files and configuration with that.
>
> Does anyone know how we can use pre-created index files and configuration
> with latest test-framework? What is the recommended way to do such kind of
> testing? Any direction with this would be really helpful.
>
> Thanks!
> Pratik
>
--
Sincerely yours
Mikhail Khludnev