You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Soheb Mahmood <so...@gmail.com> on 2010/12/07 21:44:01 UTC

Testing our code for SolrCloud

Hello Solr Devs!

My group and I have been working on implementing tasks from the to-do
lists so that we can give something to the SolrCloud open source
community (as we were introduced to it by Upayavira). So far, like my
teammate Alex mentioned, we have all been working on two tasks:

- Optionally allow user to query by collection
- Optionally allow user to query by multiple collections (assume
schemas are compatible)

from the SolrCloud wiki here: http://wiki.apache.org/solr/SolrCloud#TODO.

So far, we have implemented the following code which addresses both of
the above tasks:

// Stores the comma-separated list of specified collections.
// Eg: "collection1,collection2,collection3"
String collections = params.get("collection");
if (collections != null) {
  // If there were one or more collections specified in the query, split each
  // parameter and store as a seperate member of a List.
  List<String> collectionList = StrUtils.splitSmart(collections, ",", true);

  // Get the slices that cover the specified collection.
  // First create an empty HashMap to add the slice info to.
  slices = new HashMap<String, Slice>();

  // In turn, retrieve the slices that cover each collection from the
cloud state
  // and add them to the map 'slices'.
  for (int i = 0; i < collectionList.size(); i++) {
      appendMap(slices, cloudState.getSlices(collectionList.get(i)));
  }
} else {
  // If no collections were specified, default to the collection for this core.
  slices = cloudState.getSlices(cloudDescriptor.getCollectionName());
}

// Store the logical slices in the ResponseBuilder and create a new
// String array to hold the physical shards (which will be mapped later).
rb.slices = slices.keySet().toArray(new String[slices.size()]);
rb.shards = new String[rb.slices.length];
//------------

And we have been "testing" the code by doing acceptance testing
(running the code and seeing if it works the way we would like it to).
So far, we are pretty confident with this code, but we were wondering
if anyone would like to share some comments about what we have
written.

Secondly, we now really want to have some unit tests written for the
code, and we have been admittedly struggling a bit. As of now, we have
just got a test class written so that it essentially starts some
shards, but what we need is a buildZooKeeper method in
AbstractZkTestCase.java that sets up multiple collections and
configurations. Would anybody who has expertise in the testing area
give us a point in the right direction? From what I've seen, there is
the one line of code which has:

zkClient.makePath("/collections/collection1", props1.store(),
CreateMode.PERSISTENT);

so what I was thinking was having more of these lines of code to
represent different collections (e.g. collection2, collection3). Or is
the problem more complicated than that? (I feel like I'm missing
something, like that solution is too simple).

Finally, after we get these to-do lists working, we would like to work
on a higher level design goal. I was wondering if anyone (or any group
of people) are working on a higher level design goal and wanted some
help with it?

Thanks for reading my email,
Soheb

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Re: Testing our code for SolrCloud

Posted by Soheb Mahmood <so...@gmail.com>.
Hello Mark!

Apologies for the late reply!

> Do you mind creating a JIRA issue and attaching a patch? That is usually the
> best way to go about these discussions.

We have done so here: https://issues.apache.org/jira/browse/SOLR-2287.
Unfortunately, our test cases are incomplete at the moment, but we are
currently working on it right now based on the feedback you gave us.

>> Finally, after we get these to-do lists working, we would like to work
>> on a higher level design goal. I was wondering if anyone (or any group
>> of people) are working on a higher level design goal and wanted some
>> help with it?
>
> Certainly! The more people involved the better.

I must have not worded my sentence correctly, I was wondering if
anyone in the Lucene dev mailing list was working on the higher level
design goals, so we could lend a hand, and thus move that part of
SolrCloud development forward.

Many thanks, and we will keep the mailing list informed of any
changes, including test cases for our code

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Re: Testing our code for SolrCloud

Posted by Mark Miller <ma...@gmail.com>.
On 12/7/10 3:44 PM, Soheb Mahmood wrote:
> Hello Solr Devs!

Hi Soheb!

>
> My group and I have been working on implementing tasks from the to-do
> lists so that we can give something to the SolrCloud open source
> community (as we were introduced to it by Upayavira). So far, like my
> teammate Alex mentioned, we have all been working on two tasks:
>
> - Optionally allow user to query by collection
> - Optionally allow user to query by multiple collections (assume
> schemas are compatible)

That's great!

>
> from the SolrCloud wiki here: http://wiki.apache.org/solr/SolrCloud#TODO.
>
> So far, we have implemented the following code which addresses both of
> the above tasks:
>
> // Stores the comma-separated list of specified collections.
> // Eg: "collection1,collection2,collection3"
> String collections = params.get("collection");
> if (collections != null) {
>    // If there were one or more collections specified in the query, split each
>    // parameter and store as a seperate member of a List.
>    List<String>  collectionList = StrUtils.splitSmart(collections, ",", true);
>
>    // Get the slices that cover the specified collection.
>    // First create an empty HashMap to add the slice info to.
>    slices = new HashMap<String, Slice>();
>
>    // In turn, retrieve the slices that cover each collection from the
> cloud state
>    // and add them to the map 'slices'.
>    for (int i = 0; i<  collectionList.size(); i++) {
>        appendMap(slices, cloudState.getSlices(collectionList.get(i)));
>    }
> } else {
>    // If no collections were specified, default to the collection for this core.
>    slices = cloudState.getSlices(cloudDescriptor.getCollectionName());
> }
>
> // Store the logical slices in the ResponseBuilder and create a new
> // String array to hold the physical shards (which will be mapped later).
> rb.slices = slices.keySet().toArray(new String[slices.size()]);
> rb.shards = new String[rb.slices.length];
> //------------
>
> And we have been "testing" the code by doing acceptance testing
> (running the code and seeing if it works the way we would like it to).
> So far, we are pretty confident with this code, but we were wondering
> if anyone would like to share some comments about what we have
> written.

Do you mind creating a JIRA issue and attaching a patch? That is usually 
the best way to go about these discussions.

>
> Secondly, we now really want to have some unit tests written for the
> code, and we have been admittedly struggling a bit. As of now, we have
> just got a test class written so that it essentially starts some
> shards, but what we need is a buildZooKeeper method in
> AbstractZkTestCase.java that sets up multiple collections and
> configurations. Would anybody who has expertise in the testing area
> give us a point in the right direction? From what I've seen, there is
> the one line of code which has:
>
> zkClient.makePath("/collections/collection1", props1.store(),
> CreateMode.PERSISTENT);
>
> so what I was thinking was having more of these lines of code to
> represent different collections (e.g. collection2, collection3). Or is
> the problem more complicated than that? (I feel like I'm missing
> something, like that solution is too simple).

Nope - that should be about all there is to it. Just creating a core 
with a different 'collection' value (which defaults to the SolrCore 
name) will cause the new collection to be created in zookeeper if I 
remember right. The reason for that line is not mainly to create the 
zookeeper path, but to set which named configuration set to use. props1 
has configName=conf1. Right now the tests just use one config set called 
conf1. Other collections could share that same configuration, or you 
might add others.

>
> Finally, after we get these to-do lists working, we would like to work
> on a higher level design goal. I was wondering if anyone (or any group
> of people) are working on a higher level design goal and wanted some
> help with it?

Certainly! The more people involved the better.

>
> Thanks for reading my email,
> Soheb
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>

- Mark



---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org