wrong path in snappuller

[galo <ga...@last.fm>]

I have downloaded all the scripts from the current version in the trunk 
and I'm finding the same issues as 
https://issues.apache.org/jira/browse/SOLR-188 in the snappuller, I 
haven't looked in other scripts yet.

rsync -Wa${verbose}${compress} --delete ${sizeonly} \
${stats} rsync://${master_host}:${rsyncd_port}/solr/${name}/ 
${data_dir}/${name}-wip

that command fails in non-default installations due to that /solr/

Is this known or should I log it in JIRA?

thanks,

galo

Re: wrong path in snappuller

[Chris Hostetter <ho...@fucit.org>]

: The "solr" on the rsync command line is just a label which is defined in
: rsyncd.conf on the master.  rsyncd.conf is created on the fly by the script
: rsyncd-start:
	...
: This label is then mapped to the path defined in $data_dir.

Ah... right, i forgot about that.

: > Why does it need to start an rsyncd in the master in a different port
: > for each ap, is it not enough to call rsync on master:path?

one of the reasons for this appraoch is to make it easier to run solr in a
somewhat self contained setup .. you don't have to rely on an "external"
(to the Solr install) instance of rsyncd running rooted at base of the
filesystem.  the other nice thing with having seperate rsyncd for each
solr instance is that you can shutoff all replication with a single
command on a master solr port (without disabling other solr masters
running on the same machine, or breaking other non-solr uses of rsync on
that machine)

this can be handy when you want to do a upgrade to a solr tier without any
down time:
  1) turn of the master's rsync port,
  2) disable snappuller on all of the slaves
  3) shutdown and upgrade the master solr port
  4) rebuild the index on the master as needed
  5) run queries against the master to test things are working well.
  6) start the master's rsyncd port
  7) take half of your slaves out of rotation from your load balancer
  8) shutdown and upgrade the slaves that are out of rotation
  9) enable snappulling on the slaves that are out of rotation
 10) swap which slaves are in/out of rotation on your load balancer
 11) repeat steps 8 and 9
 12) add all slaves back into rotation on your load balancer.

...if you had a sincel rsync port for the entire machine, then this
wouldn't work very cleanly if the machine you were using as the "master"
was hosting more then solr index (or any other apps using rsync)


-Hoss

Re: wrong path in snappuller

[Bill Au <bi...@gmail.com>]

What OS are you running on?
The "solr" on the rsync command line is just a label which is defined in
rsyncd.conf on the master.  rsyncd.conf is created on the fly by the script
rsyncd-start:
cat <<EOF > ${solr_root}/conf/rsyncd.conf
#### rsyncd.conf file ####

uid = $(whoami)
gid = $(whoami)
use chroot = no
list = no
pid file = ${solr_root}/logs/rsyncd.pid
log file = ${solr_root}/logs/rsyncd.log
[solr]
    path = ${data_dir}
    comment = Solr
EOF

This label is then mapped to the path defined in $data_dir.

Bill


On 4/25/07, galo <ga...@last.fm> wrote:
>
> Ok, i will create an issue.
>
> I got round it changing this
>
> > : rsync -Wa${verbose}${compress} --delete ${sizeonly} \
> > : ${stats} rsync://${master_host}:${rsyncd_port}/solr/${name}/
> > : ${data_dir}/${name}-wip
>
> for
>
> > : rsync -Wa${verbose}${compress} --delete ${sizeonly} \
> > : ${stats} ${master_host}:${master_data_dir}/${name}/
> > : ${data_dir}/${name}-wip
>
> I had to remove the rsync:// as it was causing some problems finding the
> path and I didn't have much time to investigate. It works with absolute
> or relative paths set in the slave's master data folder param.
>
> Why does it need to start an rsyncd in the master in a different port
> for each ap, is it not enough to call rsync on master:path?
>
> Thanks for answering,
>
> Galo
>
>
> Chris Hostetter wrote:
> > : and I'm finding the same issues as
> > : https://issues.apache.org/jira/browse/SOLR-188 in the snappuller, I
> > : haven't looked in other scripts yet.
> > :
> > : rsync -Wa${verbose}${compress} --delete ${sizeonly} \
> > : ${stats} rsync://${master_host}:${rsyncd_port}/solr/${name}/
> > : ${data_dir}/${name}-wip
> >
> > that would be a seperate issue from SOLR-188 ... 188 has to do with non
> > standard URLs, this seems to be an issue with snappuller assuming a
> > specific rsync path (which if i understand correctly, is relative the
> > working directory of rsyncd?)
> >
> > : Is this known or should I log it in JIRA?
> >
> > please open a new Jira issue ... i'm guessing a new optional param will
> be
> > needed for the master's solr_home relative the rsync server.
> >
> >
> > -Hoss
> >
> >
>
>
>

Re: wrong path in snappuller

[galo <ga...@last.fm>]

Ok, i will create an issue.

I got round it changing this

 > : rsync -Wa${verbose}${compress} --delete ${sizeonly} \
 > : ${stats} rsync://${master_host}:${rsyncd_port}/solr/${name}/
 > : ${data_dir}/${name}-wip

for

 > : rsync -Wa${verbose}${compress} --delete ${sizeonly} \
 > : ${stats} ${master_host}:${master_data_dir}/${name}/
 > : ${data_dir}/${name}-wip

I had to remove the rsync:// as it was causing some problems finding the 
path and I didn't have much time to investigate. It works with absolute 
or relative paths set in the slave's master data folder param.

Why does it need to start an rsyncd in the master in a different port 
for each ap, is it not enough to call rsync on master:path?

Thanks for answering,

Galo


Chris Hostetter wrote:
> : and I'm finding the same issues as
> : https://issues.apache.org/jira/browse/SOLR-188 in the snappuller, I
> : haven't looked in other scripts yet.
> :
> : rsync -Wa${verbose}${compress} --delete ${sizeonly} \
> : ${stats} rsync://${master_host}:${rsyncd_port}/solr/${name}/
> : ${data_dir}/${name}-wip
> 
> that would be a seperate issue from SOLR-188 ... 188 has to do with non
> standard URLs, this seems to be an issue with snappuller assuming a
> specific rsync path (which if i understand correctly, is relative the
> working directory of rsyncd?)
> 
> : Is this known or should I log it in JIRA?
> 
> please open a new Jira issue ... i'm guessing a new optional param will be
> needed for the master's solr_home relative the rsync server.
> 
> 
> -Hoss
> 
> 

Re: wrong path in snappuller

[Chris Hostetter <ho...@fucit.org>]

: and I'm finding the same issues as
: https://issues.apache.org/jira/browse/SOLR-188 in the snappuller, I
: haven't looked in other scripts yet.
:
: rsync -Wa${verbose}${compress} --delete ${sizeonly} \
: ${stats} rsync://${master_host}:${rsyncd_port}/solr/${name}/
: ${data_dir}/${name}-wip

that would be a seperate issue from SOLR-188 ... 188 has to do with non
standard URLs, this seems to be an issue with snappuller assuming a
specific rsync path (which if i understand correctly, is relative the
working directory of rsyncd?)

: Is this known or should I log it in JIRA?

please open a new Jira issue ... i'm guessing a new optional param will be
needed for the master's solr_home relative the rsync server.


-Hoss