You are viewing a plain text version of this content. The canonical link for it is here.

Posted to solr-dev@lucene.apache.org by "Jianhan (JIRA)" <ji...@apache.org> on 2009/05/08 22:00:45 UTC

[jira] Created: (SOLR-1154) allow specifying solr configuration file through system property to simplify deployment procedure in certain cases

allow specifying solr configuration file through system property to simplify deployment procedure in certain cases
------------------------------------------------------------------------------------------------------------------

                 Key: SOLR-1154
                 URL: https://issues.apache.org/jira/browse/SOLR-1154
             Project: Solr
          Issue Type: Improvement
    Affects Versions: 1.4
            Reporter: Jianhan
            Priority: Minor
             Fix For: 1.4


Hi,

I wanted to use this parameter to specify different solr configuration files for master and slave to simplify deployment procedure. Unfortunately, I can't dynamically replace the value of this parameter. Basically, what I want is

  <filter>
    <filter-name>SolrRequestFilter</filter-name>
    <filter-class>org.apache.solr.servlet.SolrDispatchFilter</filter-class>
    <init-param>
      <param-name>solrconfig-filename</param-name>
      <param-value>solrconfig-master.xml</param-value>
    </init-param>
</filter>

for master instance, and

  <filter>
    <filter-name>SolrRequestFilter</filter-name>
    <filter-class>org.apache.solr.servlet.SolrDispatchFilter</filter-class>
    <init-param>
      <param-name>solrconfig-filename</param-name>
      <param-value>solrconfig-slave.xml</param-value>
    </init-param>
</filter>

for slave instance.

Ideally, if I can use system property for its value like in solrconfig.xml. For example,


  <filter>
    <filter-name>SolrRequestFilter</filter-name>
    <filter-class>org.apache.solr.servlet.SolrDispatchFilter</filter-class>
    <init-param>
      <param-name>solrconfig-filename</param-name>
      <param-value>${solr.config.filename: solrconfig.xml}</param-value>
    </init-param>
</filter>

but I learned that in general we can't use system property in web.xml.

I realize that I can use replication of config file to achieve this, but I thought that creates unnecessary dependencies for slaves on master instance.

So here is my proposal:

make SolrDispatchFilter look up another init parameter, say 'solrconfig-filename-property', and its value is a system property name, and if this property is set, we get the file name, otherwise nothing happens (of course, if both exist, 'solrconfig-filename' takes precedence). This will give us maximum flexibility of specifying configuration files for different instances.

Your thoughts?

Thanks,

Jianhan

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

Re: [jira] Commented: (SOLR-1154) allow specifying solr configuration file through system property to simplify deployment procedure in certain cases

Posted by Jian Han Guo <ji...@gmail.com>.

In our environment, master and slave instances are running in different
boxes. We package everything in a single tar file and push it to production
boxes and expand the tar file there. After that, operation team just needs
to run a commend to start/restart the application without making any
changes. There is no difference between master and slave instances other
than a launch argument. In other words, we have to distinguish master from
slave by a launch argument and have master and slave use different
configuration files.

I have never used solr.xml, but it sounds for different purpose, correct?

Thanks,

Jianhan



On Fri, May 8, 2009 at 10:24 PM, Noble Paul (JIRA) <ji...@apache.org> wrote:

>
>    [
> https://issues.apache.org/jira/browse/SOLR-1154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12707610#action_12707610]
>
> Noble Paul commented on SOLR-1154:
> ----------------------------------
>
> is your master and slave running on the same tomcat?
>
> did you try running multicore? solr.xml can let you specify different
> solconfig/schema xml for each core.
>
>
>
>
> > allow specifying solr configuration file through system property to
> simplify deployment procedure in certain cases
> >
> ------------------------------------------------------------------------------------------------------------------
> >
> >                 Key: SOLR-1154
> >                 URL: https://issues.apache.org/jira/browse/SOLR-1154
> >             Project: Solr
> >          Issue Type: Improvement
> >    Affects Versions: 1.4
> >            Reporter: Jianhan
> >            Priority: Minor
> >             Fix For: 1.4
> >
> >         Attachments: SOLR-1154.patch
> >
> >   Original Estimate: 5h
> >  Remaining Estimate: 5h
> >
> > Hi,
> > I wanted to use this parameter to specify different solr configuration
> files for master and slave to simplify deployment procedure. Unfortunately,
> I can't dynamically replace the value of this parameter. Basically, what I
> want is
> >   <filter>
> >     <filter-name>SolrRequestFilter</filter-name>
> >
> <filter-class>org.apache.solr.servlet.SolrDispatchFilter</filter-class>
> >     <init-param>
> >       <param-name>solrconfig-filename</param-name>
> >       <param-value>solrconfig-master.xml</param-value>
> >     </init-param>
> > </filter>
> > for master instance, and
> >   <filter>
> >     <filter-name>SolrRequestFilter</filter-name>
> >
> <filter-class>org.apache.solr.servlet.SolrDispatchFilter</filter-class>
> >     <init-param>
> >       <param-name>solrconfig-filename</param-name>
> >       <param-value>solrconfig-slave.xml</param-value>
> >     </init-param>
> > </filter>
> > for slave instance.
> > Ideally, if I can use system property for its value like in
> solrconfig.xml. For example,
> >   <filter>
> >     <filter-name>SolrRequestFilter</filter-name>
> >
> <filter-class>org.apache.solr.servlet.SolrDispatchFilter</filter-class>
> >     <init-param>
> >       <param-name>solrconfig-filename</param-name>
> >       <param-value>${solr.config.filename: solrconfig.xml}</param-value>
> >     </init-param>
> > </filter>
> > but I learned that in general we can't use system property in web.xml.
> > I realize that I can use replication of config file to achieve this, but
> I thought that creates unnecessary dependencies for slaves on master
> instance.
> > So here is my proposal:
> > make SolrDispatchFilter look up another init parameter, say
> 'solrconfig-filename-property', and its value is a system property name, and
> if this property is set, we get the file name, otherwise nothing happens (of
> course, if both exist, 'solrconfig-filename' takes precedence). This will
> give us maximum flexibility of specifying configuration files for different
> instances.
> > Your thoughts?
> > Thanks,
> > Jianhan
>
> --
> This message is automatically generated by JIRA.
> -
> You can reply to this email to add a comment to the issue online.
>
>

Re: [jira] Commented: (SOLR-1154) allow specifying solr configuration file through system property to simplify deployment procedure in certain cases

Posted by Jian Han Guo <ji...@gmail.com>.

I understand. I'd love to see a different way for us to load sections of
solrconfig.xml dynamically so we don't have to modify web.xml. For now what
I need is to be able to include

this part

<!-- Please refer to http://wiki.apache.org/solr/SolrReplication for details
on configuring replication -->
<!--Master config-->
<!--
<requestHandler name="/replication" class="solr.ReplicationHandler" >
    <lst name="master">
        <str name="replicateAfter">commit</str>
         <str name="confFiles">schema.xml,stopwords.txt</str>
    </lst>
</requestHandler>
-->

for my master instance, and this part


<!-- Slave config-->
<!--
<requestHandler name="/replication" class="solr.ReplicationHandler">
    <lst name="slave">
        <str name="masterUrl">http://localhost:8983/solr/replication</str>
        <str name="pollInterval">00:00:60</str>
     </lst>
</requestHandler>
-->

for my slave instance, but I don't seem to be able to find a way to do it.

Also, yes, other than solrconfig.xml, everything else is a simple copy. The
way I see it is that once solrconfig.xml/solrconfig-slove.xml is configured,
we will have a lot of changes on schema.xml, synonyms.txt, stopwords.txt
etc, and if we have multiple copies of these files, we have to remember to
copy over every time we make changes. That is what I meant when I said
'potential maintenance problem'.

Thanks,

Jianhan






2009/5/11 Noble Paul നോബിള്‍ नोब्ळ् <no...@corp.aol.com>

> in general , modifying web.xml is discouraged. so, this is the first
> time someone is asking for config to be added into the configuration
> of SolrDispatchFilter.
>
> when you say the whole directory ,other than solrconfig.xml everything
> else is a simple copy , right?
>
> On Mon, May 11, 2009 at 10:20 PM, Jian Han Guo <ji...@gmail.com> wrote:
> > You are right that keeping directories for solr home can solve this
> problem.
> > The only issue is that you then have to keep two copies of everything in
> the
> > solr home directory, which is a potential maintenance problem. I still
> yet
> > to figure out how to extract the common part of solrconfig.xml and
> > solrconfig-slave.xml, but at least we only duplicate one file instead of
> the
> > whole directory if this feature is available.
> >
> > Thanks,
> >
> > Jianhan
> >
> > On Sun, May 10, 2009 at 10:09 PM, Noble Paul (JIRA) <ji...@apache.org>
> wrote:
> >
> >>
> >>    [
> >>
> https://issues.apache.org/jira/browse/SOLR-1154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12707885#action_12707885
> ]
> >>
> >> Noble Paul commented on SOLR-1154:
> >> ----------------------------------
> >>
> >> replying to the mail http://markmail.org/message/52vj7lughfplxmks
> >>
> >> it is still possible to achieve what you want to do by setting a
> different
> >> solr.solr.home from the system property. when you package your war keep
> two
> >> instance directories one for master and another for slave. pass
> different
> >> solr.solr.home as command line param when you start the server
> >>
> >> > allow specifying solr configuration file through system property to
> >> simplify deployment procedure in certain cases
> >> >
> >>
> ------------------------------------------------------------------------------------------------------------------
> >> >
> >> >                 Key: SOLR-1154
> >> >                 URL: https://issues.apache.org/jira/browse/SOLR-1154
> >> >             Project: Solr
> >> >          Issue Type: Improvement
> >> >    Affects Versions: 1.4
> >> >            Reporter: Jianhan
> >> >            Priority: Minor
> >> >             Fix For: 1.4
> >> >
> >> >         Attachments: SOLR-1154.patch
> >> >
> >> >   Original Estimate: 5h
> >> >  Remaining Estimate: 5h
> >> >
> >> > Hi,
> >> > I wanted to use this parameter to specify different solr configuration
> >> files for master and slave to simplify deployment procedure.
> Unfortunately,
> >> I can't dynamically replace the value of this parameter. Basically, what
> I
> >> want is
> >> >   <filter>
> >> >     <filter-name>SolrRequestFilter</filter-name>
> >> >
> >> <filter-class>org.apache.solr.servlet.SolrDispatchFilter</filter-class>
> >> >     <init-param>
> >> >       <param-name>solrconfig-filename</param-name>
> >> >       <param-value>solrconfig-master.xml</param-value>
> >> >     </init-param>
> >> > </filter>
> >> > for master instance, and
> >> >   <filter>
> >> >     <filter-name>SolrRequestFilter</filter-name>
> >> >
> >> <filter-class>org.apache.solr.servlet.SolrDispatchFilter</filter-class>
> >> >     <init-param>
> >> >       <param-name>solrconfig-filename</param-name>
> >> >       <param-value>solrconfig-slave.xml</param-value>
> >> >     </init-param>
> >> > </filter>
> >> > for slave instance.
> >> > Ideally, if I can use system property for its value like in
> >> solrconfig.xml. For example,
> >> >   <filter>
> >> >     <filter-name>SolrRequestFilter</filter-name>
> >> >
> >> <filter-class>org.apache.solr.servlet.SolrDispatchFilter</filter-class>
> >> >     <init-param>
> >> >       <param-name>solrconfig-filename</param-name>
> >> >       <param-value>${solr.config.filename:
> solrconfig.xml}</param-value>
> >> >     </init-param>
> >> > </filter>
> >> > but I learned that in general we can't use system property in web.xml.
> >> > I realize that I can use replication of config file to achieve this,
> but
> >> I thought that creates unnecessary dependencies for slaves on master
> >> instance.
> >> > So here is my proposal:
> >> > make SolrDispatchFilter look up another init parameter, say
> >> 'solrconfig-filename-property', and its value is a system property name,
> and
> >> if this property is set, we get the file name, otherwise nothing happens
> (of
> >> course, if both exist, 'solrconfig-filename' takes precedence). This
> will
> >> give us maximum flexibility of specifying configuration files for
> different
> >> instances.
> >> > Your thoughts?
> >> > Thanks,
> >> > Jianhan
> >>
> >> --
> >> This message is automatically generated by JIRA.
> >> -
> >> You can reply to this email to add a comment to the issue online.
> >>
> >>
> >
>
>
>
> --
> -----------------------------------------------------
> Noble Paul | Principal Engineer| AOL | http://aol.com
>

Re: [jira] Commented: (SOLR-1154) allow specifying solr configuration file through system property to simplify deployment procedure in certain cases

Posted by Noble Paul നോബിള്‍ नोब्ळ् <no...@corp.aol.com>.

in general , modifying web.xml is discouraged. so, this is the first
time someone is asking for config to be added into the configuration
of SolrDispatchFilter.

when you say the whole directory ,other than solrconfig.xml everything
else is a simple copy , right?

On Mon, May 11, 2009 at 10:20 PM, Jian Han Guo <ji...@gmail.com> wrote:
> You are right that keeping directories for solr home can solve this problem.
> The only issue is that you then have to keep two copies of everything in the
> solr home directory, which is a potential maintenance problem. I still yet
> to figure out how to extract the common part of solrconfig.xml and
> solrconfig-slave.xml, but at least we only duplicate one file instead of the
> whole directory if this feature is available.
>
> Thanks,
>
> Jianhan
>
> On Sun, May 10, 2009 at 10:09 PM, Noble Paul (JIRA) <ji...@apache.org> wrote:
>
>>
>>    [
>> https://issues.apache.org/jira/browse/SOLR-1154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12707885#action_12707885]
>>
>> Noble Paul commented on SOLR-1154:
>> ----------------------------------
>>
>> replying to the mail http://markmail.org/message/52vj7lughfplxmks
>>
>> it is still possible to achieve what you want to do by setting a different
>> solr.solr.home from the system property. when you package your war keep two
>> instance directories one for master and another for slave. pass different
>> solr.solr.home as command line param when you start the server
>>
>> > allow specifying solr configuration file through system property to
>> simplify deployment procedure in certain cases
>> >
>> ------------------------------------------------------------------------------------------------------------------
>> >
>> >                 Key: SOLR-1154
>> >                 URL: https://issues.apache.org/jira/browse/SOLR-1154
>> >             Project: Solr
>> >          Issue Type: Improvement
>> >    Affects Versions: 1.4
>> >            Reporter: Jianhan
>> >            Priority: Minor
>> >             Fix For: 1.4
>> >
>> >         Attachments: SOLR-1154.patch
>> >
>> >   Original Estimate: 5h
>> >  Remaining Estimate: 5h
>> >
>> > Hi,
>> > I wanted to use this parameter to specify different solr configuration
>> files for master and slave to simplify deployment procedure. Unfortunately,
>> I can't dynamically replace the value of this parameter. Basically, what I
>> want is
>> >   <filter>
>> >     <filter-name>SolrRequestFilter</filter-name>
>> >
>> <filter-class>org.apache.solr.servlet.SolrDispatchFilter</filter-class>
>> >     <init-param>
>> >       <param-name>solrconfig-filename</param-name>
>> >       <param-value>solrconfig-master.xml</param-value>
>> >     </init-param>
>> > </filter>
>> > for master instance, and
>> >   <filter>
>> >     <filter-name>SolrRequestFilter</filter-name>
>> >
>> <filter-class>org.apache.solr.servlet.SolrDispatchFilter</filter-class>
>> >     <init-param>
>> >       <param-name>solrconfig-filename</param-name>
>> >       <param-value>solrconfig-slave.xml</param-value>
>> >     </init-param>
>> > </filter>
>> > for slave instance.
>> > Ideally, if I can use system property for its value like in
>> solrconfig.xml. For example,
>> >   <filter>
>> >     <filter-name>SolrRequestFilter</filter-name>
>> >
>> <filter-class>org.apache.solr.servlet.SolrDispatchFilter</filter-class>
>> >     <init-param>
>> >       <param-name>solrconfig-filename</param-name>
>> >       <param-value>${solr.config.filename: solrconfig.xml}</param-value>
>> >     </init-param>
>> > </filter>
>> > but I learned that in general we can't use system property in web.xml.
>> > I realize that I can use replication of config file to achieve this, but
>> I thought that creates unnecessary dependencies for slaves on master
>> instance.
>> > So here is my proposal:
>> > make SolrDispatchFilter look up another init parameter, say
>> 'solrconfig-filename-property', and its value is a system property name, and
>> if this property is set, we get the file name, otherwise nothing happens (of
>> course, if both exist, 'solrconfig-filename' takes precedence). This will
>> give us maximum flexibility of specifying configuration files for different
>> instances.
>> > Your thoughts?
>> > Thanks,
>> > Jianhan
>>
>> --
>> This message is automatically generated by JIRA.
>> -
>> You can reply to this email to add a comment to the issue online.
>>
>>
>



-- 
-----------------------------------------------------
Noble Paul | Principal Engineer| AOL | http://aol.com

Re: [jira] Commented: (SOLR-1154) allow specifying solr configuration file through system property to simplify deployment procedure in certain cases

Posted by Jian Han Guo <ji...@gmail.com>.

You are right that keeping directories for solr home can solve this problem.
The only issue is that you then have to keep two copies of everything in the
solr home directory, which is a potential maintenance problem. I still yet
to figure out how to extract the common part of solrconfig.xml and
solrconfig-slave.xml, but at least we only duplicate one file instead of the
whole directory if this feature is available.

Thanks,

Jianhan

On Sun, May 10, 2009 at 10:09 PM, Noble Paul (JIRA) <ji...@apache.org> wrote:

>
>    [
> https://issues.apache.org/jira/browse/SOLR-1154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12707885#action_12707885]
>
> Noble Paul commented on SOLR-1154:
> ----------------------------------
>
> replying to the mail http://markmail.org/message/52vj7lughfplxmks
>
> it is still possible to achieve what you want to do by setting a different
> solr.solr.home from the system property. when you package your war keep two
> instance directories one for master and another for slave. pass different
> solr.solr.home as command line param when you start the server
>
> > allow specifying solr configuration file through system property to
> simplify deployment procedure in certain cases
> >
> ------------------------------------------------------------------------------------------------------------------
> >
> >                 Key: SOLR-1154
> >                 URL: https://issues.apache.org/jira/browse/SOLR-1154
> >             Project: Solr
> >          Issue Type: Improvement
> >    Affects Versions: 1.4
> >            Reporter: Jianhan
> >            Priority: Minor
> >             Fix For: 1.4
> >
> >         Attachments: SOLR-1154.patch
> >
> >   Original Estimate: 5h
> >  Remaining Estimate: 5h
> >
> > Hi,
> > I wanted to use this parameter to specify different solr configuration
> files for master and slave to simplify deployment procedure. Unfortunately,
> I can't dynamically replace the value of this parameter. Basically, what I
> want is
> >   <filter>
> >     <filter-name>SolrRequestFilter</filter-name>
> >
> <filter-class>org.apache.solr.servlet.SolrDispatchFilter</filter-class>
> >     <init-param>
> >       <param-name>solrconfig-filename</param-name>
> >       <param-value>solrconfig-master.xml</param-value>
> >     </init-param>
> > </filter>
> > for master instance, and
> >   <filter>
> >     <filter-name>SolrRequestFilter</filter-name>
> >
> <filter-class>org.apache.solr.servlet.SolrDispatchFilter</filter-class>
> >     <init-param>
> >       <param-name>solrconfig-filename</param-name>
> >       <param-value>solrconfig-slave.xml</param-value>
> >     </init-param>
> > </filter>
> > for slave instance.
> > Ideally, if I can use system property for its value like in
> solrconfig.xml. For example,
> >   <filter>
> >     <filter-name>SolrRequestFilter</filter-name>
> >
> <filter-class>org.apache.solr.servlet.SolrDispatchFilter</filter-class>
> >     <init-param>
> >       <param-name>solrconfig-filename</param-name>
> >       <param-value>${solr.config.filename: solrconfig.xml}</param-value>
> >     </init-param>
> > </filter>
> > but I learned that in general we can't use system property in web.xml.
> > I realize that I can use replication of config file to achieve this, but
> I thought that creates unnecessary dependencies for slaves on master
> instance.
> > So here is my proposal:
> > make SolrDispatchFilter look up another init parameter, say
> 'solrconfig-filename-property', and its value is a system property name, and
> if this property is set, we get the file name, otherwise nothing happens (of
> course, if both exist, 'solrconfig-filename' takes precedence). This will
> give us maximum flexibility of specifying configuration files for different
> instances.
> > Your thoughts?
> > Thanks,
> > Jianhan
>
> --
> This message is automatically generated by JIRA.
> -
> You can reply to this email to add a comment to the issue online.
>
>

Re: [jira] Commented: (SOLR-1154) allow specifying solr configuration file through system property to simplify deployment procedure in certain cases

Posted by Jian Han Guo <ji...@gmail.com>.

In my opinion, we don't need do anything, at least for now. It is not ideal
but we have a work around.

Thanks,

Jianhan


On Thu, May 21, 2009 at 12:18 AM, Noble Paul (JIRA) <ji...@apache.org> wrote:

>
>    [
> https://issues.apache.org/jira/browse/SOLR-1154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12711500#action_12711500]
>
> Noble Paul commented on SOLR-1154:
> ----------------------------------
>
> do we still need to pursue this?
>
> > allow specifying solr configuration file through system property to
> simplify deployment procedure in certain cases
> >
> ------------------------------------------------------------------------------------------------------------------
> >
> >                 Key: SOLR-1154
> >                 URL: https://issues.apache.org/jira/browse/SOLR-1154
> >             Project: Solr
> >          Issue Type: Improvement
> >    Affects Versions: 1.4
> >            Reporter: Jianhan
> >            Priority: Minor
> >             Fix For: 1.4
> >
> >         Attachments: SOLR-1154.patch, SOLR-1154.patch
> >
> >   Original Estimate: 5h
> >  Remaining Estimate: 5h
> >
> > Hi,
> > I wanted to use this parameter to specify different solr configuration
> files for master and slave to simplify deployment procedure. Unfortunately,
> I can't dynamically replace the value of this parameter. Basically, what I
> want is
> >   <filter>
> >     <filter-name>SolrRequestFilter</filter-name>
> >
> <filter-class>org.apache.solr.servlet.SolrDispatchFilter</filter-class>
> >     <init-param>
> >       <param-name>solrconfig-filename</param-name>
> >       <param-value>solrconfig-master.xml</param-value>
> >     </init-param>
> > </filter>
> > for master instance, and
> >   <filter>
> >     <filter-name>SolrRequestFilter</filter-name>
> >
> <filter-class>org.apache.solr.servlet.SolrDispatchFilter</filter-class>
> >     <init-param>
> >       <param-name>solrconfig-filename</param-name>
> >       <param-value>solrconfig-slave.xml</param-value>
> >     </init-param>
> > </filter>
> > for slave instance.
> > Ideally, if I can use system property for its value like in
> solrconfig.xml. For example,
> >   <filter>
> >     <filter-name>SolrRequestFilter</filter-name>
> >
> <filter-class>org.apache.solr.servlet.SolrDispatchFilter</filter-class>
> >     <init-param>
> >       <param-name>solrconfig-filename</param-name>
> >       <param-value>${solr.config.filename: solrconfig.xml}</param-value>
> >     </init-param>
> > </filter>
> > but I learned that in general we can't use system property in web.xml.
> > I realize that I can use replication of config file to achieve this, but
> I thought that creates unnecessary dependencies for slaves on master
> instance.
> > So here is my proposal:
> > make SolrDispatchFilter look up another init parameter, say
> 'solrconfig-filename-property', and its value is a system property name, and
> if this property is set, we get the file name, otherwise nothing happens (of
> course, if both exist, 'solrconfig-filename' takes precedence). This will
> give us maximum flexibility of specifying configuration files for different
> instances.
> > Your thoughts?
> > Thanks,
> > Jianhan
>
> --
> This message is automatically generated by JIRA.
> -
> You can reply to this email to add a comment to the issue online.
>
>