multiple indices

[Jack L <jl...@yahoo.ca>]

I was going through some old emails on this topic. Rafael Rossini figured
out how to run multiple indices on single instance of jetty but it has to
be jetty plus. I guess jetty doesn't allow this? I suppose I can add
additional jars and make it work but I haven't tried that. It'll
always be much safer/simpler/less playing around if a feature is
available out of box.

I'm mentioning this again because I really think it's a desirable feature,
especially because each JVM uses a lot of memory and sometimes it's
not possible to start a new jetty for each index due to memory
limitation.

I understand I can use a type field and mix doc types but this is not
ideal for two reasons:

1. it's easier to maintain separate indices. I can just wipe out all
the files and re-post an individual index. Much less posting work to
do as opposed to re-posting all docs. Or I can move one index to
another partition, or even to another server to run separately in
order to scale up. It'll be a problem (although solvable by deleting
and re-posting) with a mixed index.

2. my understanding is that mixed index means larger index files and
slower performance

JettyPlus's download links seem to be broken so I wasn't able to check
its download size. If not too big, maybe JettyPlus is an option?
If not, there should be a way to have this feature implemented on solr
side? Maybe by prefixing the REST URLs with index names...

-- 
Thanks,
Jack

Re: Re[2]: multiple indices

[Matt Kangas <ka...@gmail.com>]

Jack, the JNDI-enabling jarfiles now ship as part of the main .zip  
distribution. There is no need for a separate "JettyPlus" download as  
of Jetty 6.

I used Jetty 6.1.3 (http://dist.codehaus.org/jetty/jetty-6.1.x/ 
jetty-6.1.3.zip) at the time, and I am using only these jarfiles from  
the main distribution. I stripped everything else out that seemed  
unnecessary for running Solr.

lib/jetty-6.1.3.jar
lib/jetty-util-6.1.3.jar
lib/jsp-2.1/ant-1.6.5.jar
lib/jsp-2.1/core-3.1.1.jar
lib/jsp-2.1/jsp-2.1.jar
lib/jsp-2.1/jsp-api-2.1.jar
lib/naming/jetty-naming-6.1.3.jar
lib/plus/jetty-plus-6.1.3.jar
lib/servlet-api-2.5-6.1.3.jar

--Matt

On Sep 13, 2007, at 11:44 AM, Jack L wrote:

> Thanks Matt, I'll give it a try! So this requires JettyPlus?
>
> -- 
> Best regards,
> Jack
>
> Wednesday, September 12, 2007, 5:14:32 AM, you wrote:
>
>> Jack, I've posted a complete recipe for running two Solr indices
>> within one Jetty 6 container:
>
>> http://wiki.apache.org/solr/SolrJetty
>
>> Scroll down to the part that says:
>>> (7/2007 MattKangas) The recipe above didn't work for me with Jetty
>>> 6.1.3.
>>>
>>> ...
>>>
>>> I'm glossing over a lot of details, so attached is a tarball with a
>>> known-good configuration that runs two Solr instances inside one
>>> Jetty container. I'm using Solr 1.2.0 and Jetty 6.1.3 respectively.
>>>
>>>
>
>> Hope this helps,
>> --matt
>
>> On Sep 11, 2007, at 11:52 AM, Jack L wrote:
>
>>> I was going through some old emails on this topic. Rafael Rossini
>>> figured
>>> out how to run multiple indices on single instance of jetty but it
>>> has to
>>> be jetty plus. I guess jetty doesn't allow this? I suppose I can add
>>> additional jars and make it work but I haven't tried that. It'll
>>> always be much safer/simpler/less playing around if a feature is
>>> available out of box.
>>>
>>> I'm mentioning this again because I really think it's a desirable
>>> feature,
>>> especially because each JVM uses a lot of memory and sometimes it's
>>> not possible to start a new jetty for each index due to memory
>>> limitation.
>>>
>>> I understand I can use a type field and mix doc types but this is  
>>> not
>>> ideal for two reasons:
>>>
>>> 1. it's easier to maintain separate indices. I can just wipe out all
>>> the files and re-post an individual index. Much less posting work to
>>> do as opposed to re-posting all docs. Or I can move one index to
>>> another partition, or even to another server to run separately in
>>> order to scale up. It'll be a problem (although solvable by deleting
>>> and re-posting) with a mixed index.
>>>
>>> 2. my understanding is that mixed index means larger index files and
>>> slower performance
>>>
>>> JettyPlus's download links seem to be broken so I wasn't able to  
>>> check
>>> its download size. If not too big, maybe JettyPlus is an option?
>>> If not, there should be a way to have this feature implemented on  
>>> solr
>>> side? Maybe by prefixing the REST URLs with index names...
>>>
>>> -- 
>>> Thanks,
>>> Jack
>>>
>
>> --
>> Matt Kangas / kangas@gmail.com
>
>

--
Matt Kangas / kangas@gmail.com

Re[2]: multiple indices

[Jack L <jl...@yahoo.ca>]

Thanks Matt, I'll give it a try! So this requires JettyPlus?

-- 
Best regards,
Jack

Wednesday, September 12, 2007, 5:14:32 AM, you wrote:

> Jack, I've posted a complete recipe for running two Solr indices  
> within one Jetty 6 container:

> http://wiki.apache.org/solr/SolrJetty

> Scroll down to the part that says:
>> (7/2007 MattKangas) The recipe above didn't work for me with Jetty
>> 6.1.3.
>>
>> ...
>>
>> I'm glossing over a lot of details, so attached is a tarball with a
>> known-good configuration that runs two Solr instances inside one  
>> Jetty container. I'm using Solr 1.2.0 and Jetty 6.1.3 respectively.
>>
>>

> Hope this helps,
> --matt

> On Sep 11, 2007, at 11:52 AM, Jack L wrote:

>> I was going through some old emails on this topic. Rafael Rossini  
>> figured
>> out how to run multiple indices on single instance of jetty but it
>> has to
>> be jetty plus. I guess jetty doesn't allow this? I suppose I can add
>> additional jars and make it work but I haven't tried that. It'll
>> always be much safer/simpler/less playing around if a feature is
>> available out of box.
>>
>> I'm mentioning this again because I really think it's a desirable  
>> feature,
>> especially because each JVM uses a lot of memory and sometimes it's
>> not possible to start a new jetty for each index due to memory
>> limitation.
>>
>> I understand I can use a type field and mix doc types but this is not
>> ideal for two reasons:
>>
>> 1. it's easier to maintain separate indices. I can just wipe out all
>> the files and re-post an individual index. Much less posting work to
>> do as opposed to re-posting all docs. Or I can move one index to
>> another partition, or even to another server to run separately in
>> order to scale up. It'll be a problem (although solvable by deleting
>> and re-posting) with a mixed index.
>>
>> 2. my understanding is that mixed index means larger index files and
>> slower performance
>>
>> JettyPlus's download links seem to be broken so I wasn't able to check
>> its download size. If not too big, maybe JettyPlus is an option?
>> If not, there should be a way to have this feature implemented on solr
>> side? Maybe by prefixing the REST URLs with index names...
>>
>> -- 
>> Thanks,
>> Jack
>>

> --
> Matt Kangas / kangas@gmail.com

Re: multiple indices

[Matt Kangas <ka...@gmail.com>]

Jack, I've posted a complete recipe for running two Solr indices  
within one Jetty 6 container:

http://wiki.apache.org/solr/SolrJetty

Scroll down to the part that says:
> (7/2007 MattKangas) The recipe above didn't work for me with Jetty  
> 6.1.3.
>
> ...
>
> I'm glossing over a lot of details, so attached is a tarball with a  
> known-good configuration that runs two Solr instances inside one  
> Jetty container. I'm using Solr 1.2.0 and Jetty 6.1.3 respectively.
>
>

Hope this helps,
--matt

On Sep 11, 2007, at 11:52 AM, Jack L wrote:

> I was going through some old emails on this topic. Rafael Rossini  
> figured
> out how to run multiple indices on single instance of jetty but it  
> has to
> be jetty plus. I guess jetty doesn't allow this? I suppose I can add
> additional jars and make it work but I haven't tried that. It'll
> always be much safer/simpler/less playing around if a feature is
> available out of box.
>
> I'm mentioning this again because I really think it's a desirable  
> feature,
> especially because each JVM uses a lot of memory and sometimes it's
> not possible to start a new jetty for each index due to memory
> limitation.
>
> I understand I can use a type field and mix doc types but this is not
> ideal for two reasons:
>
> 1. it's easier to maintain separate indices. I can just wipe out all
> the files and re-post an individual index. Much less posting work to
> do as opposed to re-posting all docs. Or I can move one index to
> another partition, or even to another server to run separately in
> order to scale up. It'll be a problem (although solvable by deleting
> and re-posting) with a mixed index.
>
> 2. my understanding is that mixed index means larger index files and
> slower performance
>
> JettyPlus's download links seem to be broken so I wasn't able to check
> its download size. If not too big, maybe JettyPlus is an option?
> If not, there should be a way to have this feature implemented on solr
> side? Maybe by prefixing the REST URLs with index names...
>
> -- 
> Thanks,
> Jack
>

--
Matt Kangas / kangas@gmail.com

Re[2]: multiple indices

[Jack L <jl...@yahoo.ca>]

Hello Mike,

> but it is still just an example application.

I think this is a very modest statement. I'd like to say both solr
(including the example) and jetty are production level software.
I suppose many users, like me, will just take it and make minimum
modification of the configs and use it on a production server.
That's exactly what I did and it has been working great.

If multiple indices feature is available out of the box, it'll become
another great feature that's easily accessible for entry level users,
like me :)

> There just might be something like that in 1.3...

I'm looking forward to it!

Jack

Re: multiple indices

[Mike Klaas <mi...@gmail.com>]

On 11-Sep-07, at 3:32 PM, George Aroush wrote:

>>
>> The example that comes with Solr is meant to be a starting
>> point for users.  It is a relatively functional and
>> well-commented example, and its config files are pretty much
>> the canonical documentation for solr config, and for many
>> people they can modifying it for their own production use....
>>
>> but it is still just an example application.
>>
>> By the time people want to do expert-level activities with
>> Solr (multi-index falls into that category), they should be
>> able to configure their own servlet container, whether it be
>> jetty plus, tomcat, resin, etc.
>
> Does this means Solr 1.2 supports MultiSearcher?

No, I'm purely talking about housing two different indices on the  
same machine with Solr.  They may be related, they may not be.

Currently (1.2), the options are:

1. multiple processes/servlet containers/jvms
2. multiple instances of solr webapps within a single container/ 
process/jvm

In the future, (1.3 or farther down the line), another option will be:

3. multiple indices within a single solr webapp, added/removed on the  
fly.

-Mike

RE: multiple indices

[George Aroush <ge...@aroush.net>]

> > I was going through some old emails on this topic. Rafael Rossini 
> > figured out how to run multiple indices on single instance of jetty 
> > but it has to be jetty plus. I guess jetty doesn't allow this? I 
> > suppose I can add additional jars and make it work but I 
> haven't tried 
> > that. It'll always be much safer/simpler/less playing around if a 
> > feature is available out of box.
> 
> The example that comes with Solr is meant to be a starting 
> point for users.  It is a relatively functional and 
> well-commented example, and its config files are pretty much 
> the canonical documentation for solr config, and for many 
> people they can modifying it for their own production use....
> 
> but it is still just an example application.
> 
> By the time people want to do expert-level activities with 
> Solr (multi-index falls into that category), they should be 
> able to configure their own servlet container, whether it be 
> jetty plus, tomcat, resin, etc.

Does this means Solr 1.2 supports MultiSearcher?

-- George

Re: multiple indices

[Mike Klaas <mi...@gmail.com>]

On 11-Sep-07, at 8:52 AM, Jack L wrote:

> I was going through some old emails on this topic. Rafael Rossini  
> figured
> out how to run multiple indices on single instance of jetty but it  
> has to
> be jetty plus. I guess jetty doesn't allow this? I suppose I can add
> additional jars and make it work but I haven't tried that. It'll
> always be much safer/simpler/less playing around if a feature is
> available out of box.

The example that comes with Solr is meant to be a starting point for  
users.  It is a relatively functional and well-commented example, and  
its config files are pretty much the canonical documentation for solr  
config, and for many people they can modifying it for their own  
production use....

but it is still just an example application.

By the time people want to do expert-level activities with Solr  
(multi-index falls into that category), they should be able to  
configure their own servlet container, whether it be jetty plus,  
tomcat, resin, etc.

> 1. it's easier to maintain separate indices. I can just wipe out all
> the files and re-post an individual index. Much less posting work to
> do as opposed to re-posting all docs. Or I can move one index to
> another partition, or even to another server to run separately in
> order to scale up. It'll be a problem (although solvable by deleting
> and re-posting) with a mixed index.

> 2. my understanding is that mixed index means larger index files and
> slower performance

Both of these are true, but do not typically have to be decided at  
minute zero when developing a project with solr.  I recent split our  
main index into two separate solr installations in a single jettyplus  
container and it was less than a day's work (most of it was tweaking  
interface code on our side, not the solr config itsefl).

> JettyPlus's download links seem to be broken so I wasn't able to check
> its download size. If not too big, maybe JettyPlus is an option?
> If not, there should be a way to have this feature implemented on solr
> side? Maybe by prefixing the REST URLs with index names...

There just might be something like that in 1.3...

-Mike