You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Trevor Campbell <tc...@atlassian.com> on 2013/03/25 06:44:08 UTC
Multi-core and replicated Solr cloud testing. Data-directory mis-configures
I have three indexes which I have set up as three separate cores, using this solr.xml config.
<cores adminPath="/admin/cores" host="${host:}" hostPort="${jetty.port:}">
<core name="jira-issue" instanceDir="jira-issue" >
<property name="dataDir" value="jira-issue/data/" />
</core>
<core name="jira-comment" instanceDir="jira-comment" >
<property name="dataDir" value="jira-comment/data/" />
</core>
<core name="jira-change-history" instanceDir="jira-change-history" >
<property name="dataDir" value="jira-change-history/data/" />
</core>
</cores>
This works just fine a standalone solr.
I duplicated this setup on the same machine under a completely separate solr installation (solr-nodeb) and modified all
the data directroies to point to the direstories in nodeb. This all worked fine.
I then connected the 2 instances together with zoo-keeper using settings "-Dbootstrap_conf=true
-Dcollection.configName=jiraCluster -DzkRun -DnumShards=1" for the first intsance and "-DzkHost=localhost:9080" for the
second. (I'm using tomcat and ports 8080 and 8081 for the 2 Solr instances)
Now the data directories of the second node point to the data directories in the first node.
I have tried many settings in the solrconfig.xml for each core but am now using absolute paths, e.g.
<dataDir>/home/xxxxxxxx/solr-4.2.0-nodeb/example/multicore/jira-comment/data</dataDir>
previously I used
${solr.jira-comment.data.dir:/home/tcampbell/solr-4.2.0-nodeb/example/multicore/jira-comment/data}
but that had the same result.
It seems zookeeper is forcing data directory config from the uploaded configuration on all the nodes in the cluster?
How can I do testing on a single machine? Do I really need identical directory layouts on all machines?
Re: Multi-core and replicated Solr cloud testing. Data-directory mis-configures
Posted by Trevor Campbell <tc...@atlassian.com>.
Solved.
I was able to solve this by removing any reference to dataDir from the
solrconfig.xml. So in solr.xml for each node I have:
<cores adminPath="/admin/cores" host="${host:}" hostPort="${jetty.port:}">
<core name="jira-issue" instanceDir="jira-issue" >
<property name="dataDir" value="jira-issue/data/" />
</core>
<core name="jira-comment" instanceDir="jira-comment" >
<property name="dataDir" value="jira-comment/data/" />
</core>
<core name="jira-change-history" instanceDir="jira-change-history" >
<property name="dataDir" value="jira-change-history/data/" />
</core>
</cores>
and in solrconfig.xml in each core I have removed the reference to dataDir
completely.
<!-- <dataDir>${solr.core0.data.dir:}</dataDir> -->
On Tue, Mar 26, 2013 at 8:41 AM, Trevor Campbell <tc...@atlassian.com>wrote:
> That example does not work if you have > 1 collection (core) per node, all
> end up sharing the same index and overwrite one another.
>
>
> On Mon, Mar 25, 2013 at 6:27 PM, Gopal Patwa <go...@gmail.com> wrote:
>
>> if you use default directory then it will use solr.home directory, I have
>> tested solr cloud example on local machine with 5-6 nodes.And data
>> directory was created under core name, like
>>
>> "example2/solr/collection1/data". you could see example startup script
>> from
>> source code "solr/cloud-dev/solrcloud-multi-start.sh"
>>
>> example solrconfig.xml
>>
>> <dataDir>${solr.data.dir:}</dataDir>
>>
>> On Sun, Mar 24, 2013 at 10:44 PM, Trevor Campbell
>> <tc...@atlassian.com>wrote:
>>
>> > I have three indexes which I have set up as three separate cores, using
>> > this solr.xml config.
>> >
>> > <cores adminPath="/admin/cores" host="${host:}"
>> > hostPort="${jetty.port:}">
>> > <core name="jira-issue" instanceDir="jira-issue" >
>> > <property name="dataDir" value="jira-issue/data/" />
>> > </core>
>> > <core name="jira-comment" instanceDir="jira-comment" >
>> > <property name="dataDir" value="jira-comment/data/" />
>> > </core>
>> > <core name="jira-change-history"
>> instanceDir="jira-change-**history" >
>> > <property name="dataDir" value="jira-change-history/**data/" />
>> > </core>
>> > </cores>
>> >
>> > This works just fine a standalone solr.
>> >
>> > I duplicated this setup on the same machine under a completely separate
>> > solr installation (solr-nodeb) and modified all the data directroies to
>> > point to the direstories in nodeb. This all worked fine.
>> >
>> > I then connected the 2 instances together with zoo-keeper using settings
>> > "-Dbootstrap_conf=true -Dcollection.configName=**jiraCluster -DzkRun
>> > -DnumShards=1" for the first intsance and "-DzkHost=localhost:9080" for
>> > the second. (I'm using tomcat and ports 8080 and 8081 for the 2 Solr
>> > instances)
>> >
>> > Now the data directories of the second node point to the data
>> directories
>> > in the first node.
>> >
>> > I have tried many settings in the solrconfig.xml for each core but am
>> now
>> > using absolute paths, e.g.
>> > <dataDir>/home/xxxxxxxx/solr-**4.2.0-nodeb/example/multicore/**
>> > jira-comment/data</dataDir>
>> >
>> > previously I used
>> > ${solr.jira-comment.data.dir:/**home/tcampbell/solr-4.2.0-**
>> > nodeb/example/multicore/jira-**comment/data}
>> > but that had the same result.
>> >
>> > It seems zookeeper is forcing data directory config from the uploaded
>> > configuration on all the nodes in the cluster?
>> >
>> > How can I do testing on a single machine? Do I really need identical
>> > directory layouts on all machines?
>> >
>> >
>> >
>>
>
>
Re: Multi-core and replicated Solr cloud testing. Data-directory mis-configures
Posted by Trevor Campbell <tc...@atlassian.com>.
That example does not work if you have > 1 collection (core) per node, all
end up sharing the same index and overwrite one another.
On Mon, Mar 25, 2013 at 6:27 PM, Gopal Patwa <go...@gmail.com> wrote:
> if you use default directory then it will use solr.home directory, I have
> tested solr cloud example on local machine with 5-6 nodes.And data
> directory was created under core name, like
>
> "example2/solr/collection1/data". you could see example startup script from
> source code "solr/cloud-dev/solrcloud-multi-start.sh"
>
> example solrconfig.xml
>
> <dataDir>${solr.data.dir:}</dataDir>
>
> On Sun, Mar 24, 2013 at 10:44 PM, Trevor Campbell
> <tc...@atlassian.com>wrote:
>
> > I have three indexes which I have set up as three separate cores, using
> > this solr.xml config.
> >
> > <cores adminPath="/admin/cores" host="${host:}"
> > hostPort="${jetty.port:}">
> > <core name="jira-issue" instanceDir="jira-issue" >
> > <property name="dataDir" value="jira-issue/data/" />
> > </core>
> > <core name="jira-comment" instanceDir="jira-comment" >
> > <property name="dataDir" value="jira-comment/data/" />
> > </core>
> > <core name="jira-change-history" instanceDir="jira-change-**history"
> >
> > <property name="dataDir" value="jira-change-history/**data/" />
> > </core>
> > </cores>
> >
> > This works just fine a standalone solr.
> >
> > I duplicated this setup on the same machine under a completely separate
> > solr installation (solr-nodeb) and modified all the data directroies to
> > point to the direstories in nodeb. This all worked fine.
> >
> > I then connected the 2 instances together with zoo-keeper using settings
> > "-Dbootstrap_conf=true -Dcollection.configName=**jiraCluster -DzkRun
> > -DnumShards=1" for the first intsance and "-DzkHost=localhost:9080" for
> > the second. (I'm using tomcat and ports 8080 and 8081 for the 2 Solr
> > instances)
> >
> > Now the data directories of the second node point to the data directories
> > in the first node.
> >
> > I have tried many settings in the solrconfig.xml for each core but am now
> > using absolute paths, e.g.
> > <dataDir>/home/xxxxxxxx/solr-**4.2.0-nodeb/example/multicore/**
> > jira-comment/data</dataDir>
> >
> > previously I used
> > ${solr.jira-comment.data.dir:/**home/tcampbell/solr-4.2.0-**
> > nodeb/example/multicore/jira-**comment/data}
> > but that had the same result.
> >
> > It seems zookeeper is forcing data directory config from the uploaded
> > configuration on all the nodes in the cluster?
> >
> > How can I do testing on a single machine? Do I really need identical
> > directory layouts on all machines?
> >
> >
> >
>
Re: Multi-core and replicated Solr cloud testing. Data-directory mis-configures
Posted by Gopal Patwa <go...@gmail.com>.
if you use default directory then it will use solr.home directory, I have
tested solr cloud example on local machine with 5-6 nodes.And data
directory was created under core name, like
"example2/solr/collection1/data". you could see example startup script from
source code "solr/cloud-dev/solrcloud-multi-start.sh"
example solrconfig.xml
<dataDir>${solr.data.dir:}</dataDir>
On Sun, Mar 24, 2013 at 10:44 PM, Trevor Campbell
<tc...@atlassian.com>wrote:
> I have three indexes which I have set up as three separate cores, using
> this solr.xml config.
>
> <cores adminPath="/admin/cores" host="${host:}"
> hostPort="${jetty.port:}">
> <core name="jira-issue" instanceDir="jira-issue" >
> <property name="dataDir" value="jira-issue/data/" />
> </core>
> <core name="jira-comment" instanceDir="jira-comment" >
> <property name="dataDir" value="jira-comment/data/" />
> </core>
> <core name="jira-change-history" instanceDir="jira-change-**history" >
> <property name="dataDir" value="jira-change-history/**data/" />
> </core>
> </cores>
>
> This works just fine a standalone solr.
>
> I duplicated this setup on the same machine under a completely separate
> solr installation (solr-nodeb) and modified all the data directroies to
> point to the direstories in nodeb. This all worked fine.
>
> I then connected the 2 instances together with zoo-keeper using settings
> "-Dbootstrap_conf=true -Dcollection.configName=**jiraCluster -DzkRun
> -DnumShards=1" for the first intsance and "-DzkHost=localhost:9080" for
> the second. (I'm using tomcat and ports 8080 and 8081 for the 2 Solr
> instances)
>
> Now the data directories of the second node point to the data directories
> in the first node.
>
> I have tried many settings in the solrconfig.xml for each core but am now
> using absolute paths, e.g.
> <dataDir>/home/xxxxxxxx/solr-**4.2.0-nodeb/example/multicore/**
> jira-comment/data</dataDir>
>
> previously I used
> ${solr.jira-comment.data.dir:/**home/tcampbell/solr-4.2.0-**
> nodeb/example/multicore/jira-**comment/data}
> but that had the same result.
>
> It seems zookeeper is forcing data directory config from the uploaded
> configuration on all the nodes in the cluster?
>
> How can I do testing on a single machine? Do I really need identical
> directory layouts on all machines?
>
>
>