You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Vicky_Dev <vi...@yahoo.co.in> on 2009/07/02 17:02:40 UTC
Re: Master Slave data distribution | rsync fail issue
Yes. Permissions are same across cores
~Vikrant
Bill Au wrote:
>
> Are the user/group/permissions on the snapshot files the same for both
> cases
> (manual vs postCommit/postOptimize events)?
>
> Bill
>
> On Tue, May 5, 2009 at 12:54 PM, tushar kapoor <
> tushar_kapoor_47@rediffmail.com> wrote:
>
>>
>> Hi,
>>
>> I am facing an issue while performing snapshot pulling thru Snappuller
>> script from slave server :
>> We have the setup of multicores on Master Solr and Slave Solr servers.
>> Scenario , 2 cores are set :
>> i) CORE_WWW.ABCD.COM
>> ii) CORE_WWW.XYZ.COM
>>
>> rsync-enable and rsync-start script run from CORE_WWW.ABCD.COM on master
>> server. Thus rsyncd.commf file got generated on CORE_WWW.ABCD.COM only ,
>> but not on CORE_WWW.XYZ.COM.
>> Rsyncd.conf of CORE_WWW.ABCD.COM :
>> #### rsyncd.conf file ####
>> uid = webuser
>> gid = webuser
>> use chroot = no
>> list = no
>> pid file =
>> /opt/apache-tomcat-6.0.18/apache-solr-1.3.0/example/solr/multicore/
>> CORE_WWW.ABCD.COM/logs/rsyncd.pid
>> log file =
>> /opt/apache-tomcat-6.0.18/apache-solr-1.3.0/example/solr/multicore/
>> CORE_WWW.ABCD.COM/logs/rsyncd.log
>> [solr]
>> path =
>> /opt/apache-tomcat-6.0.18/apache-solr-1.3.0/example/solr/multicore/
>> CORE_WWW.ABCD.COM/data
>> comment = Solr
>>
>> rsync error used to get generated while doing the pulling of master
>> server
>> snapshot of a particular core "CORE_WWW.XYZ.COM" from slave end, for core
>> "CORE_WWW.ABCD.COM" snappuller occured without any error.
>>
>> Also, this issue is coming only when snapshot are generated at master end
>> thru the way given below:
>> A) Snapshot are generated automatically by
>> editing “${SOLR_HOME}/solr/conf/solrconfig.xml” to let either commit
>> index
>> or optimize index trigger the snapshooter (search “postCommit” and
>> “postOptimize” to find the configuration section).
>>
>> Sample of solrconfig.xml entry on Master server End:
>> I)
>> <listener event="postCommit" class="solr.RunExecutableListener">
>> <str
>>
>> name="exe">/opt/apache-tomcat-6.0.18/apache-solr-1.3.0/example/solr/multicore/
>> CORE_WWW.ABCD.COM/bin/snapshooter</str>
>> <str
>>
>> name="dir">/opt/apache-tomcat-6.0.18/apache-solr-1.3.0/example/solr/multicore/
>> CORE_WWW.ABCD.COM/bin</str>
>> <bool name="wait">true</bool>
>> <arr name="args"> <str>arg1</str> <str>arg2</str> </arr>
>> <arr name="env"> <str>MYVAR=val1</str> </arr>
>> </listener>
>>
>> same way done for core "CORE_WWW.XYZ.COM" solrConfig.xml.
>> II) The <dataDir> tag remains commented on both the cores .XML on master
>> server.
>>
>> Log sample for more clearity :
>> rsyncd.log of the core "CORE_WWW.XYZ.COM":
>> 2009/05/01 15:48:40 command: ./rsyncd-start
>> 2009/05/01 15:48:40 [15064] rsyncd version 2.6.3 starting, listening on
>> port
>> 18983
>> 2009/05/01 15:48:40 rsyncd started with
>>
>> data_dir=/opt/apache-tomcat-6.0.18/apache-solr-1.3.0/example/solr/multicore/
>> CORE_WWW.XYZ.COm/data
>> and accepting requests
>> 2009/05/01 15:50:36 [15195] rsync on solr/snapshot.20090501153311/ from
>> deltrialmac.mac1.com (10.210.7.191)
>> 2009/05/01 15:50:36 [15195] rsync: link_stat "snapshot.20090501153311/."
>> (in
>> solr) failed: No such file or directory (2)
>> 2009/05/01 15:50:36 [15195] rsync error: some files could not be
>> transferred
>> (code 23) at main.c(442)
>> 2009/05/01 15:52:23 [15301] rsync on solr/snapshot.20090501155030/ from
>> delpearsondm.sapient.com (10.210.7.191)
>> 2009/05/01 15:52:23 [15301] wrote 3438 bytes read 290 bytes total size
>> 2779
>> 2009/05/01 16:03:31 [15553] rsync on solr/snapshot.20090501160112/ from
>> deltrialmac.mac1.com (10.210.7.191)
>> 2009/05/01 16:03:31 [15553] rsync: link_stat "snapshot.20090501160112/."
>> (in
>> solr) failed: No such file or directory (2)
>> 2009/05/01 16:03:31 [15553] rsync error: some files could not be
>> transferred
>> (code 23) at main.c(442)
>> 2009/05/01 16:04:27 [15674] rsync on solr/snapshot.20090501160054/ from
>> deltrialmac.mac1.com (10.210.7.191)
>> 2009/05/01 16:04:27 [15674] wrote 4173214 bytes read 290 bytes total
>> size
>> 4174633
>>
>> I m unable to figure out that from where "/." gets appeneded at the end
>> "snapshot.20090501153311/."
>> "Snappuller.log"
>> 2009/05/04 16:55:43 started by solrUser
>> 2009/05/04 16:55:43 command:
>> /opt/apache-solr-1.3.0/example/solr/multicore/
>> CORE_WWW.PUFFINBOOKS.CA/bin/snappuller
>> -u <http://CORE_WWW.PUFFINBOOKS.CA/bin/snappuller%0A-u> webuser
>> 2009/05/04 16:55:52 pulling snapshot snapshot.20090504164935
>> 2009/05/04 16:56:09 rsync failed
>> 2009/05/04 16:56:24 failed (elapsed time: 41 sec)
>>
>> Error shown on console :
>> rsync: link_stat "snapshot.20090504164935/." (in solr) failed: No such
>> file
>> or directory (2)
>> client: nothing to do: perhaps you need to specify some filenames or the
>> --recursive option?
>> rsync error: some files could not be transferred (code 23) at main.c(723)
>>
>> B) The same issue is not coming while manually running the Snapshot
>> script
>> after reguler interval of time at Master server and then running
>> Snappuller
>> script at slave end for multiple cores. The postCommit/postOptimize part
>> of
>> solrConfig.xml has been commented.
>> Here also rsync script run thru the core "CORE_WWW.ABCD.COM". Snappuller
>> and
>> snapinstaller occurred successfully.
>>
>> Thanks in advance.
>>
>> --
>> View this message in context:
>> http://www.nabble.com/Master-Slave-data-distribution-%7C-rsync-fail-issue-tp23391580p23391580.html
>> Sent from the Solr - User mailing list archive at Nabble.com.
>>
>>
>
>
--
View this message in context: http://www.nabble.com/Master-Slave-data-distribution-%7C-rsync-fail-issue-tp23391580p24308389.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Master Slave data distribution | rsync fail issue
Posted by Bill Au <bi...@gmail.com>.
You can add the "-V" option to both your automatic and manual invocation of
snappuller and snapinstaller tor both core and compare the debug info.
Bill
On Thu, Jul 2, 2009 at 11:02 AM, Vicky_Dev
<vi...@yahoo.co.in>wrote:
>
> Yes. Permissions are same across cores
>
> ~Vikrant
>
>
> Bill Au wrote:
> >
> > Are the user/group/permissions on the snapshot files the same for both
> > cases
> > (manual vs postCommit/postOptimize events)?
> >
> > Bill
> >
> > On Tue, May 5, 2009 at 12:54 PM, tushar kapoor <
> > tushar_kapoor_47@rediffmail.com> wrote:
> >
> >>
> >> Hi,
> >>
> >> I am facing an issue while performing snapshot pulling thru Snappuller
> >> script from slave server :
> >> We have the setup of multicores on Master Solr and Slave Solr servers.
> >> Scenario , 2 cores are set :
> >> i) CORE_WWW.ABCD.COM
> >> ii) CORE_WWW.XYZ.COM
> >>
> >> rsync-enable and rsync-start script run from CORE_WWW.ABCD.COM on
> master
> >> server. Thus rsyncd.commf file got generated on CORE_WWW.ABCD.COM only
> ,
> >> but not on CORE_WWW.XYZ.COM.
> >> Rsyncd.conf of CORE_WWW.ABCD.COM :
> >> #### rsyncd.conf file ####
> >> uid = webuser
> >> gid = webuser
> >> use chroot = no
> >> list = no
> >> pid file =
> >> /opt/apache-tomcat-6.0.18/apache-solr-1.3.0/example/solr/multicore/
> >> CORE_WWW.ABCD.COM/logs/rsyncd.pid
> >> log file =
> >> /opt/apache-tomcat-6.0.18/apache-solr-1.3.0/example/solr/multicore/
> >> CORE_WWW.ABCD.COM/logs/rsyncd.log
> >> [solr]
> >> path =
> >> /opt/apache-tomcat-6.0.18/apache-solr-1.3.0/example/solr/multicore/
> >> CORE_WWW.ABCD.COM/data
> >> comment = Solr
> >>
> >> rsync error used to get generated while doing the pulling of master
> >> server
> >> snapshot of a particular core "CORE_WWW.XYZ.COM" from slave end, for
> core
> >> "CORE_WWW.ABCD.COM" snappuller occured without any error.
> >>
> >> Also, this issue is coming only when snapshot are generated at master
> end
> >> thru the way given below:
> >> A) Snapshot are generated automatically by
> >> editing “${SOLR_HOME}/solr/conf/solrconfig.xml” to let either commit
> >> index
> >> or optimize index trigger the snapshooter (search “postCommit” and
> >> “postOptimize” to find the configuration section).
> >>
> >> Sample of solrconfig.xml entry on Master server End:
> >> I)
> >> <listener event="postCommit" class="solr.RunExecutableListener">
> >> <str
> >>
> >>
> name="exe">/opt/apache-tomcat-6.0.18/apache-solr-1.3.0/example/solr/multicore/
> >> CORE_WWW.ABCD.COM/bin/snapshooter</str>
> >> <str
> >>
> >>
> name="dir">/opt/apache-tomcat-6.0.18/apache-solr-1.3.0/example/solr/multicore/
> >> CORE_WWW.ABCD.COM/bin</str>
> >> <bool name="wait">true</bool>
> >> <arr name="args"> <str>arg1</str> <str>arg2</str> </arr>
> >> <arr name="env"> <str>MYVAR=val1</str> </arr>
> >> </listener>
> >>
> >> same way done for core "CORE_WWW.XYZ.COM" solrConfig.xml.
> >> II) The <dataDir> tag remains commented on both the cores .XML on
> master
> >> server.
> >>
> >> Log sample for more clearity :
> >> rsyncd.log of the core "CORE_WWW.XYZ.COM":
> >> 2009/05/01 15:48:40 command: ./rsyncd-start
> >> 2009/05/01 15:48:40 [15064] rsyncd version 2.6.3 starting, listening on
> >> port
> >> 18983
> >> 2009/05/01 15:48:40 rsyncd started with
> >>
> >>
> data_dir=/opt/apache-tomcat-6.0.18/apache-solr-1.3.0/example/solr/multicore/
> >> CORE_WWW.XYZ.COm/data
> >> and accepting requests
> >> 2009/05/01 15:50:36 [15195] rsync on solr/snapshot.20090501153311/ from
> >> deltrialmac.mac1.com (10.210.7.191)
> >> 2009/05/01 15:50:36 [15195] rsync: link_stat "snapshot.20090501153311/."
> >> (in
> >> solr) failed: No such file or directory (2)
> >> 2009/05/01 15:50:36 [15195] rsync error: some files could not be
> >> transferred
> >> (code 23) at main.c(442)
> >> 2009/05/01 15:52:23 [15301] rsync on solr/snapshot.20090501155030/ from
> >> delpearsondm.sapient.com (10.210.7.191)
> >> 2009/05/01 15:52:23 [15301] wrote 3438 bytes read 290 bytes total size
> >> 2779
> >> 2009/05/01 16:03:31 [15553] rsync on solr/snapshot.20090501160112/ from
> >> deltrialmac.mac1.com (10.210.7.191)
> >> 2009/05/01 16:03:31 [15553] rsync: link_stat "snapshot.20090501160112/."
> >> (in
> >> solr) failed: No such file or directory (2)
> >> 2009/05/01 16:03:31 [15553] rsync error: some files could not be
> >> transferred
> >> (code 23) at main.c(442)
> >> 2009/05/01 16:04:27 [15674] rsync on solr/snapshot.20090501160054/ from
> >> deltrialmac.mac1.com (10.210.7.191)
> >> 2009/05/01 16:04:27 [15674] wrote 4173214 bytes read 290 bytes total
> >> size
> >> 4174633
> >>
> >> I m unable to figure out that from where "/." gets appeneded at the end
> >> "snapshot.20090501153311/."
> >> "Snappuller.log"
> >> 2009/05/04 16:55:43 started by solrUser
> >> 2009/05/04 16:55:43 command:
> >> /opt/apache-solr-1.3.0/example/solr/multicore/
> >> CORE_WWW.PUFFINBOOKS.CA/bin/snappuller
> >> -u <http://CORE_WWW.PUFFINBOOKS.CA/bin/snappuller%0A-u> webuser
> >> 2009/05/04 16:55:52 pulling snapshot snapshot.20090504164935
> >> 2009/05/04 16:56:09 rsync failed
> >> 2009/05/04 16:56:24 failed (elapsed time: 41 sec)
> >>
> >> Error shown on console :
> >> rsync: link_stat "snapshot.20090504164935/." (in solr) failed: No such
> >> file
> >> or directory (2)
> >> client: nothing to do: perhaps you need to specify some filenames or the
> >> --recursive option?
> >> rsync error: some files could not be transferred (code 23) at
> main.c(723)
> >>
> >> B) The same issue is not coming while manually running the Snapshot
> >> script
> >> after reguler interval of time at Master server and then running
> >> Snappuller
> >> script at slave end for multiple cores. The postCommit/postOptimize part
> >> of
> >> solrConfig.xml has been commented.
> >> Here also rsync script run thru the core "CORE_WWW.ABCD.COM".
> Snappuller
> >> and
> >> snapinstaller occurred successfully.
> >>
> >> Thanks in advance.
> >>
> >> --
> >> View this message in context:
> >>
> http://www.nabble.com/Master-Slave-data-distribution-%7C-rsync-fail-issue-tp23391580p23391580.html
> >> Sent from the Solr - User mailing list archive at Nabble.com.
> >>
> >>
> >
> >
>
> --
> View this message in context:
> http://www.nabble.com/Master-Slave-data-distribution-%7C-rsync-fail-issue-tp23391580p24308389.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
>