You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Matthew Runo <mr...@zappos.com> on 2007/08/31 22:08:57 UTC
Replication broken.. no helpful errors?
Hello!
On a somewhat related note, our replication seems very much broken.
I've added -v to all my cron jobs, and I think I've seen the error
(below).
As you can see, it's rsyncing an updated index, but then doesn't seem
to know to install it. I'm not sure why though.. no errors are
reported anywhere via the -v. Any help would be most appreciated, I'm
sure I'm just missing something. You can see the cronjob command in
the subject of the forwarded message.
+--------------------------------------------------------+
| Matthew Runo
| Zappos Development
| mruno@zappos.com
| 702-943-7833
+--------------------------------------------------------+
Begin forwarded message:
> From: root@web03.zappos.com (Cron Daemon)
> Date: August 31, 2007 1:02:36 PM PDT
> To: mruno@zappos.com
> Subject: Cron <to...@search2> /opt/solr/bin/snappuller -M search1
> -P 18080 -D /opt/solr/data -S /opt/solr/logs -d /opt/solr/data -v;/
> opt/solr/bin/snapinstaller -M search1 -S /opt/solr/logs -d /opt/
> solr/data -v
>
> started by tomcat5
> command: /opt/solr/bin/snappuller -M search1 -P 18080 -D /opt/solr/
> data -S /opt/solr/logs -d /opt/solr/data -v
> pulling snapshot snapshot.20070831130005
> receiving file list ... done
> deleting segments_19zm
> deleting _10bg.tis
> deleting _10bg.tii
> deleting _10bg.prx
> deleting _10bg.nrm
> deleting _10bg.frq
> deleting _10bg.fnm
> deleting _10bg.fdx
> deleting _10bg.fdt
> ./
> _14ff.fdt
> _14ff.fdx
> _14ff.fnm
> _14ff.frq
> _14ff.nrm
> _14ff.prx
> _14ff.tii
> _14ff.tis
> _14ff_9.del
> _14fq.fdt
> _14fq.fdx
> _14fq.fnm
> _14fq.frq
> _14fq.nrm
> _14fq.prx
> _14fq.tii
> _14fq.tis
> _14fq_3.del
> _14fr.fdt
> _14fr.fdx
> _14fr.fnm
> _14fr.frq
> _14fr.nrm
> _14fr.prx
> _14fr.tii
> _14fr.tis
> _14fr_2.del
> _14fs.fdt
> _14fs.fdx
> _14fs.fnm
> _14fs.frq
> _14fs.nrm
> _14fs.prx
> _14fs.tii
> _14fs.tis
> _14fs_1.del
> segments.gen
> segments_1fza
> write.lock
>
> sent 871 bytes received 843185604 bytes 24440187.68 bytes/sec
> total size is 843080453 speedup is 1.00
> started by tomcat5
> command: /opt/solr/bin/snapinstaller -M search1 -S /opt/solr/logs -
> d /opt/solr/data -v
> latest snapshot /opt/solr/data/temp-snapshot.20070816120113 already
> installed
>
Re: Replication broken.. no helpful errors?
Posted by Bill Au <bi...@gmail.com>.
As I had pointed out in my first reply to this thread, you had a directory
named temp-snapshot.20070816120113
in your data directory on the slave. Snapinstaller was mistakenly treating
that as the lastest snapshot and was installing that every time it was
called. Snapinstaller didn't trigger a commit since the same snapshot had
already been installed.
> latest snapshot /opt/solr/data/temp-snapshot.20070816120113 already
> installed
I have open a bug to improve snapinstaller:
http://issues.apache.org/jira/browse/SOLR-346
Bill
On 9/6/07, Matthew Runo <mr...@zappos.com> wrote:
>
> Well, I've been playing around with it (removed all the snapshots,
> restarted tomcat) and it seems like it works now.. maybe.
>
> I was noticing that search2 and search3, the slaves, had searchers
> that had been opened several days ago - when we do several 100
> commits and 2 optimizes on search1, the master, every day.
>
> +--------------------------------------------------------+
> | Matthew Runo
> | Zappos Development
> | mruno@zappos.com
> | 702-943-7833
> +--------------------------------------------------------+
>
>
> On Sep 6, 2007, at 12:37 PM, Yonik Seeley wrote:
>
> > On 9/6/07, Matthew Runo <mr...@zappos.com> wrote:
> >> The thing is that a new searcher is not opened if I look in the
> >> stats.jsp page. The index version never changes.
> >
> > The index version is read from the index... hence if the lucene index
> > doesn't change (even if a ew snapshot was taken), the version won't
> > change even if a new searcher was opened.
> >
> > Is the problem on the master side now since it looks like the slave is
> > pulling a temp-snapshot?
> >
> > -Yonik
> >
>
>
Re: Replication broken.. no helpful errors?
Posted by Matthew Runo <mr...@zappos.com>.
Well, I've been playing around with it (removed all the snapshots,
restarted tomcat) and it seems like it works now.. maybe.
I was noticing that search2 and search3, the slaves, had searchers
that had been opened several days ago - when we do several 100
commits and 2 optimizes on search1, the master, every day.
+--------------------------------------------------------+
| Matthew Runo
| Zappos Development
| mruno@zappos.com
| 702-943-7833
+--------------------------------------------------------+
On Sep 6, 2007, at 12:37 PM, Yonik Seeley wrote:
> On 9/6/07, Matthew Runo <mr...@zappos.com> wrote:
>> The thing is that a new searcher is not opened if I look in the
>> stats.jsp page. The index version never changes.
>
> The index version is read from the index... hence if the lucene index
> doesn't change (even if a ew snapshot was taken), the version won't
> change even if a new searcher was opened.
>
> Is the problem on the master side now since it looks like the slave is
> pulling a temp-snapshot?
>
> -Yonik
>
Re: Replication broken.. no helpful errors?
Posted by Yonik Seeley <yo...@apache.org>.
On 9/6/07, Matthew Runo <mr...@zappos.com> wrote:
> The thing is that a new searcher is not opened if I look in the
> stats.jsp page. The index version never changes.
The index version is read from the index... hence if the lucene index
doesn't change (even if a ew snapshot was taken), the version won't
change even if a new searcher was opened.
Is the problem on the master side now since it looks like the slave is
pulling a temp-snapshot?
-Yonik
Re: Replication broken.. no helpful errors?
Posted by Matthew Runo <mr...@zappos.com>.
The thing is that a new searcher is not opened if I look in the
stats.jsp page. The index version never changes.
When I run..
sudo /opt/solr/bin/commit -V -u tomcat5
..I get a new searcher opened, but even though it (in theory)
installed the new index, I see no docs in there. During the
snapinstaller...
+ echo 2007/09/06 11:43:49 command: /opt/solr/bin/snapinstaller -M
search1 -S /opt/solr/logs -d /opt/solr/data -V -u tomcat5
+ [[ -n '' ]]
++ ls /opt/solr/data
++ grep 'snapshot\.'
++ grep -v wip
++ sort -r
++ head -1
+ name=temp-snapshot.20070905150504
+ trap 'echo "caught INT/TERM, exiting now but partial installation
may have already occured";/bin/rm -rf ${data_dir"/index.tmp$$;logExit
aborted 13' INT TERM
+ [[ temp-snapshot.20070905150504 == '' ]]
+ name=/opt/solr/data/temp-snapshot.20070905150504
++ cat /opt/solr/logs/snapshot.current
...it would seem that snappuller might not be properly setting the
directory name - or should it be temp-*?
I had replication working for a few weeks, and then it broke, and has
been down since. We're going live with this project in about a week,
and I really need to get this going before then =p
+--------------------------------------------------------+
| Matthew Runo
| Zappos Development
| mruno@zappos.com
| 702-943-7833
+--------------------------------------------------------+
On Sep 6, 2007, at 6:01 AM, Bill Au wrote:
> The snapinstaller script opens a new searcher by calling commit.
> From the
> attached debug output it looks like that actually worked:
>
> + /opt/solr/bin/commit
> + [[ 0 != 0 ]]
> + logExit ended 0
>
> Try running the /opt/solr/bin/commit directly with the -V option.
>
> Bill
>
> On 9/5/07, Matthew Runo <mr...@zappos.com> wrote:
>>
>> If it helps anyone, this index is around a gig in size.
>>
>> +--------------------------------------------------------+
>> | Matthew Runo
>> | Zappos Development
>> | mruno@zappos.com
>> | 702-943-7833
>> +--------------------------------------------------------+
>>
>>
>> On Sep 5, 2007, at 3:14 PM, Matthew Runo wrote:
>>
>>> It seems that the scripts cannot open new searchers at the end of
>>> the process, for some reason. Here's a message from cron, but I'm
>>> not sure what to make of it... It looks like the files properly
>>> copied over, but failed the install. I removed the temp* directory,
>>> but still SOLR could not launch a new searcher. I don't see any
>>> activity in catalina.out though...
>>>
>>>
>>> started by tomcat5
>>> command: /opt/solr/bin/snappuller -M search1 -P 18080 -D /opt/solr/
>>> data -S /opt/solr/logs -d /opt/solr/data -v
>>> pulling snapshot temp-snapshot.20070905150504
>>> receiving file list ... done
>>> deleting segments_1ine
>>> deleting _164h_1.del
>>> deleting _164h.tis
>>> deleting _164h.tii
>>> deleting _164h.prx
>>> deleting _164h.nrm
>>> deleting _164h.frq
>>> deleting _164h.fnm
>>> deleting _164h.fdx
>>> deleting _164h.fdt
>>> deleting _164g_1.del
>>> deleting _164g.tis
>>> deleting _164g.tii
>>> deleting _164g.prx
>>> deleting _164g.nrm
>>> deleting _164g.frq
>>> deleting _164g.fnm
>>> deleting _164g.fdx
>>> deleting _164g.fdt
>>> deleting _164f_1.del
>>> deleting _164f.tis
>>> deleting _164f.tii
>>> deleting _164f.prx
>>> deleting _164f.nrm
>>> deleting _164f.frq
>>> deleting _164f.fnm
>>> deleting _164f.fdx
>>> deleting _164f.fdt
>>> deleting _164e_1.del
>>> deleting _164e.tis
>>> deleting _164e.tii
>>> deleting _164e.prx
>>> deleting _164e.nrm
>>> deleting _164e.frq
>>> deleting _164e.fnm
>>> deleting _164e.fdx
>>> deleting _164e.fdt
>>> deleting _164d_1.del
>>> deleting _164d.tis
>>> deleting _164d.tii
>>> deleting _164d.prx
>>> deleting _164d.nrm
>>> deleting _164d.frq
>>> deleting _164d.fnm
>>> deleting _164d.fdx
>>> deleting _164d.fdt
>>> deleting _164c_1.del
>>> deleting _164c.tis
>>> deleting _164c.tii
>>> deleting _164c.prx
>>> deleting _164c.nrm
>>> deleting _164c.frq
>>> deleting _164c.fnm
>>> deleting _164c.fdx
>>> deleting _164c.fdt
>>> deleting _164b_1.del
>>> deleting _164b.tis
>>> deleting _164b.tii
>>> deleting _164b.prx
>>> deleting _164b.nrm
>>> deleting _164b.frq
>>> deleting _164b.fnm
>>> deleting _164b.fdx
>>> deleting _164b.fdt
>>> deleting _164a_1.del
>>> deleting _164a.tis
>>> deleting _164a.tii
>>> deleting _164a.prx
>>> deleting _164a.nrm
>>> deleting _164a.frq
>>> deleting _164a.fnm
>>> deleting _164a.fdx
>>> deleting _164a.fdt
>>> deleting _163z_3.del
>>> deleting _163z.tis
>>> deleting _163z.tii
>>> deleting _163z.prx
>>> deleting _163z.nrm
>>> deleting _163z.frq
>>> deleting _163z.fnm
>>> deleting _163z.fdx
>>> deleting _163z.fdt
>>> deleting _163o_3.del
>>> deleting _163o.tis
>>> deleting _163o.tii
>>> deleting _163o.prx
>>> deleting _163o.nrm
>>> deleting _163o.frq
>>> deleting _163o.fnm
>>> deleting _163o.fdx
>>> deleting _163o.fdt
>>> deleting _163d_4.del
>>> deleting _163d.tis
>>> deleting _163d.tii
>>> deleting _163d.prx
>>> deleting _163d.nrm
>>> deleting _163d.frq
>>> deleting _163d.fnm
>>> deleting _163d.fdx
>>> deleting _163d.fdt
>>> deleting _1632_6.del
>>> deleting _1632.tis
>>> deleting _1632.tii
>>> deleting _1632.prx
>>> deleting _1632.nrm
>>> deleting _1632.frq
>>> deleting _1632.fnm
>>> deleting _1632.fdx
>>> deleting _1632.fdt
>>> deleting _162r_7.del
>>> deleting _162r.tis
>>> deleting _162r.tii
>>> deleting _162r.prx
>>> deleting _162r.nrm
>>> deleting _162r.frq
>>> deleting _162r.fnm
>>> deleting _162r.fdx
>>> deleting _162r.fdt
>>> deleting _162g_d.del
>>> deleting _162g.tis
>>> deleting _162g.tii
>>> deleting _162g.prx
>>> deleting _162g.nrm
>>> deleting _162g.frq
>>> deleting _162g.fnm
>>> deleting _162g.fdx
>>> deleting _162g.fdt
>>> deleting _1625_m.del
>>> deleting _1625.tis
>>> deleting _1625.tii
>>> deleting _1625.prx
>>> deleting _1625.nrm
>>> deleting _1625.frq
>>> deleting _1625.fnm
>>> deleting _1625.fdx
>>> deleting _1625.fdt
>>> deleting _161u_w.del
>>> deleting _161u.tis
>>> deleting _161u.tii
>>> deleting _161u.prx
>>> deleting _161u.nrm
>>> deleting _161u.frq
>>> deleting _161u.fnm
>>> deleting _161u.fdx
>>> deleting _161u.fdt
>>> deleting _161j_16.del
>>> ./
>>> _161j_17.del
>>> _164m.fdt
>>> _164m.fdx
>>> _164m.fnm
>>> _164m.frq
>>> _164m.nrm
>>> _164m.prx
>>> _164m.tii
>>> _164m.tis
>>> _164m_1.del
>>> _164x.fdt
>>> _164x.fdx
>>> _164x.fnm
>>> _164x.frq
>>> _164x.nrm
>>> _164x.prx
>>> _164x.tii
>>> _164x.tis
>>> _164x_1.del
>>> segments.gen
>>> segments_1inv
>>>
>>> sent 516 bytes received 105864302 bytes 30247090.86 bytes/sec
>>> total size is 966107226 speedup is 9.13
>>> + [[ -z search1 ]]
>>> + [[ -z /opt/solr/logs ]]
>>> + fixUser -M search1 -S /opt/solr/logs -d /opt/solr/data -V
>>> + [[ -z tomcat5 ]]
>>> ++ whoami
>>> + [[ tomcat5 != tomcat5 ]]
>>> ++ who -m
>>> ++ cut '-d ' -f1
>>> ++ sed '-es/^.*!//'
>>> + oldwhoami=
>>> + [[ '' == '' ]]
>>> +++ pgrep -g0 snapinstaller
>>> ++ tail -1
>>> ++ cut -f1 '-d '
>>> ++ ps h -Hfp 3621 3629 3630 3631
>>> + oldwhoami=tomcat5
>>> + [[ -z /opt/solr/data ]]
>>> ++ echo /opt/solr/data
>>> ++ cut -c1
>>> + [[ / != \/ ]]
>>> ++ echo /opt/solr/logs
>>> ++ cut -c1
>>> + [[ / != \/ ]]
>>> ++ date +%s
>>> + start=1189030205
>>> + logMessage started by tomcat5
>>> ++ timeStamp
>>> ++ date '+%Y/%m/%d %H:%M:%S'
>>> + echo 2007/09/05 15:10:05 started by tomcat5
>>> + [[ -n '' ]]
>>> + logMessage command: /opt/solr/bin/snapinstaller -M search1 -S /
>>> opt/solr/logs -d /opt/solr/data -V
>>> ++ timeStamp
>>> ++ date '+%Y/%m/%d %H:%M:%S'
>>> + echo 2007/09/05 15:10:05 command: /opt/solr/bin/snapinstaller -M
>>> search1 -S /opt/solr/logs -d /opt/solr/data -V
>>> + [[ -n '' ]]
>>> ++ ls /opt/solr/data
>>> ++ grep 'snapshot\.'
>>> ++ grep -v wip
>>> ++ sort -r
>>> ++ head -1
>>> + name=temp-snapshot.20070905150504
>>> + trap 'echo "caught INT/TERM, exiting now but partial installation
>>> may have already occured";/bin/rm -rf ${data_dir"/index.tmp$
>>> $;logExit aborted 13' INT TERM
>>> + [[ temp-snapshot.20070905150504 == '' ]]
>>> + name=/opt/solr/data/temp-snapshot.20070905150504
>>> ++ cat /opt/solr/logs/snapshot.current
>>> + [[ /opt/solr/data/temp-snapshot.20070905150504 == /opt/solr/data/
>>> snapshot.20070905150005 ]]
>>> + ssh -o StrictHostKeyChecking=no search1 mkdir -p /opt/solr/logs
>>> + logMessage installing snapshot /opt/solr/data/temp-snapshot.
>>> 20070905150504
>>> ++ timeStamp
>>> ++ date '+%Y/%m/%d %H:%M:%S'
>>> + echo 2007/09/05 15:10:05 installing snapshot /opt/solr/data/temp-
>>> snapshot.20070905150504
>>> + [[ -n '' ]]
>>> + cp -lr /opt/solr/data/temp-snapshot.20070905150504/ /opt/solr/
>>> data/index.tmp3621
>>> + /bin/rm -rf /opt/solr/data/index
>>> + mv -f /opt/solr/data/index.tmp3621 /opt/solr/data/index
>>> + echo /opt/solr/data/temp-snapshot.20070905150504
>>> ++ uname -n
>>> + scp -q -o StrictHostKeyChecking=no /opt/solr/logs/
>>> snapshot.current search1:/opt/solr/logs/snapshot.current.search2
>>> + logMessage notifing Solr to open a new Searcher
>>> ++ timeStamp
>>> ++ date '+%Y/%m/%d %H:%M:%S'
>>> + echo 2007/09/05 15:10:06 notifing Solr to open a new Searcher
>>> + [[ -n '' ]]
>>> + /opt/solr/bin/commit
>>> + [[ 0 != 0 ]]
>>> + logExit ended 0
>>> ++ date +%s
>>> + end=1189030210
>>> ++ expr 1189030210 - 1189030205
>>> + diff=5
>>> ++ timeStamp
>>> ++ date '+%Y/%m/%d %H:%M:%S'
>>> + echo '2007/09/05 15:10:10 ended (elapsed time: 5 sec)'
>>> + exit 0
>>>
>>>
>>>
>>> +--------------------------------------------------------+
>>> | Matthew Runo
>>> | Zappos Development
>>> | mruno@zappos.com
>>> | 702-943-7833
>>> +--------------------------------------------------------+
>>>
>>>
>>> On Aug 31, 2007, at 6:11 PM, Bill Au wrote:
>>>
>>>>> latest snapshot /opt/solr/data/temp-snapshot.20070816120113
>>>>> already
>>>>> installed
>>>>
>>>> It looks like you have a directory named temp-snapshot.
>>>> 20070816120113
>>>> in your data directory. You should remove it. One of the other
>>>> script might have left that behind somehow.
>>>>
>>>> I will update the snapinstaller script to ignore non-snapshot when
>>>> looking for the latest snapshot to install.
>>>>
>>>> Bill
>>>>
>>>> On 8/31/07, Matthew Runo <mr...@zappos.com> wrote:
>>>>> Hello!
>>>>>
>>>>> On a somewhat related note, our replication seems very much
>>>>> broken.
>>>>> I've added -v to all my cron jobs, and I think I've seen the error
>>>>> (below).
>>>>>
>>>>> As you can see, it's rsyncing an updated index, but then doesn't
>>>>> seem
>>>>> to know to install it. I'm not sure why though.. no errors are
>>>>> reported anywhere via the -v. Any help would be most appreciated,
>>>>> I'm
>>>>> sure I'm just missing something. You can see the cronjob
>>>>> command in
>>>>> the subject of the forwarded message.
>>>>>
>>>>> +--------------------------------------------------------+
>>>>> | Matthew Runo
>>>>> | Zappos Development
>>>>> | mruno@zappos.com
>>>>> | 702-943-7833
>>>>> +--------------------------------------------------------+
>>>>>
>>>>>
>>>>> Begin forwarded message:
>>>>>
>>>>>> From: root@web03.zappos.com (Cron Daemon)
>>>>>> Date: August 31, 2007 1:02:36 PM PDT
>>>>>> To: mruno@zappos.com
>>>>>> Subject: Cron <to...@search2> /opt/solr/bin/snappuller -M
>>>>>> search1
>>>>>> -P 18080 -D /opt/solr/data -S /opt/solr/logs -d /opt/solr/data
>>>>>> -v;/
>>>>>> opt/solr/bin/snapinstaller -M search1 -S /opt/solr/logs -d /opt/
>>>>>> solr/data -v
>>>>>>
>>>>>> started by tomcat5
>>>>>> command: /opt/solr/bin/snappuller -M search1 -P 18080 -D /opt/
>>>>>> solr/
>>>>>> data -S /opt/solr/logs -d /opt/solr/data -v
>>>>>> pulling snapshot snapshot.20070831130005
>>>>>> receiving file list ... done
>>>>>> deleting segments_19zm
>>>>>> deleting _10bg.tis
>>>>>> deleting _10bg.tii
>>>>>> deleting _10bg.prx
>>>>>> deleting _10bg.nrm
>>>>>> deleting _10bg.frq
>>>>>> deleting _10bg.fnm
>>>>>> deleting _10bg.fdx
>>>>>> deleting _10bg.fdt
>>>>>> ./
>>>>>> _14ff.fdt
>>>>>> _14ff.fdx
>>>>>> _14ff.fnm
>>>>>> _14ff.frq
>>>>>> _14ff.nrm
>>>>>> _14ff.prx
>>>>>> _14ff.tii
>>>>>> _14ff.tis
>>>>>> _14ff_9.del
>>>>>> _14fq.fdt
>>>>>> _14fq.fdx
>>>>>> _14fq.fnm
>>>>>> _14fq.frq
>>>>>> _14fq.nrm
>>>>>> _14fq.prx
>>>>>> _14fq.tii
>>>>>> _14fq.tis
>>>>>> _14fq_3.del
>>>>>> _14fr.fdt
>>>>>> _14fr.fdx
>>>>>> _14fr.fnm
>>>>>> _14fr.frq
>>>>>> _14fr.nrm
>>>>>> _14fr.prx
>>>>>> _14fr.tii
>>>>>> _14fr.tis
>>>>>> _14fr_2.del
>>>>>> _14fs.fdt
>>>>>> _14fs.fdx
>>>>>> _14fs.fnm
>>>>>> _14fs.frq
>>>>>> _14fs.nrm
>>>>>> _14fs.prx
>>>>>> _14fs.tii
>>>>>> _14fs.tis
>>>>>> _14fs_1.del
>>>>>> segments.gen
>>>>>> segments_1fza
>>>>>> write.lock
>>>>>>
>>>>>> sent 871 bytes received 843185604 bytes 24440187.68 bytes/sec
>>>>>> total size is 843080453 speedup is 1.00
>>>>>> started by tomcat5
>>>>>> command: /opt/solr/bin/snapinstaller -M search1 -S /opt/solr/
>>>>>> logs -
>>>>>> d /opt/solr/data -v
>>>>>> latest snapshot /opt/solr/data/temp-snapshot.20070816120113
>>>>>> already
>>>>>> installed
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>>
Re: Replication broken.. no helpful errors?
Posted by Bill Au <bi...@gmail.com>.
The snapinstaller script opens a new searcher by calling commit. From the
attached debug output it looks like that actually worked:
+ /opt/solr/bin/commit
+ [[ 0 != 0 ]]
+ logExit ended 0
Try running the /opt/solr/bin/commit directly with the -V option.
Bill
On 9/5/07, Matthew Runo <mr...@zappos.com> wrote:
>
> If it helps anyone, this index is around a gig in size.
>
> +--------------------------------------------------------+
> | Matthew Runo
> | Zappos Development
> | mruno@zappos.com
> | 702-943-7833
> +--------------------------------------------------------+
>
>
> On Sep 5, 2007, at 3:14 PM, Matthew Runo wrote:
>
> > It seems that the scripts cannot open new searchers at the end of
> > the process, for some reason. Here's a message from cron, but I'm
> > not sure what to make of it... It looks like the files properly
> > copied over, but failed the install. I removed the temp* directory,
> > but still SOLR could not launch a new searcher. I don't see any
> > activity in catalina.out though...
> >
> >
> > started by tomcat5
> > command: /opt/solr/bin/snappuller -M search1 -P 18080 -D /opt/solr/
> > data -S /opt/solr/logs -d /opt/solr/data -v
> > pulling snapshot temp-snapshot.20070905150504
> > receiving file list ... done
> > deleting segments_1ine
> > deleting _164h_1.del
> > deleting _164h.tis
> > deleting _164h.tii
> > deleting _164h.prx
> > deleting _164h.nrm
> > deleting _164h.frq
> > deleting _164h.fnm
> > deleting _164h.fdx
> > deleting _164h.fdt
> > deleting _164g_1.del
> > deleting _164g.tis
> > deleting _164g.tii
> > deleting _164g.prx
> > deleting _164g.nrm
> > deleting _164g.frq
> > deleting _164g.fnm
> > deleting _164g.fdx
> > deleting _164g.fdt
> > deleting _164f_1.del
> > deleting _164f.tis
> > deleting _164f.tii
> > deleting _164f.prx
> > deleting _164f.nrm
> > deleting _164f.frq
> > deleting _164f.fnm
> > deleting _164f.fdx
> > deleting _164f.fdt
> > deleting _164e_1.del
> > deleting _164e.tis
> > deleting _164e.tii
> > deleting _164e.prx
> > deleting _164e.nrm
> > deleting _164e.frq
> > deleting _164e.fnm
> > deleting _164e.fdx
> > deleting _164e.fdt
> > deleting _164d_1.del
> > deleting _164d.tis
> > deleting _164d.tii
> > deleting _164d.prx
> > deleting _164d.nrm
> > deleting _164d.frq
> > deleting _164d.fnm
> > deleting _164d.fdx
> > deleting _164d.fdt
> > deleting _164c_1.del
> > deleting _164c.tis
> > deleting _164c.tii
> > deleting _164c.prx
> > deleting _164c.nrm
> > deleting _164c.frq
> > deleting _164c.fnm
> > deleting _164c.fdx
> > deleting _164c.fdt
> > deleting _164b_1.del
> > deleting _164b.tis
> > deleting _164b.tii
> > deleting _164b.prx
> > deleting _164b.nrm
> > deleting _164b.frq
> > deleting _164b.fnm
> > deleting _164b.fdx
> > deleting _164b.fdt
> > deleting _164a_1.del
> > deleting _164a.tis
> > deleting _164a.tii
> > deleting _164a.prx
> > deleting _164a.nrm
> > deleting _164a.frq
> > deleting _164a.fnm
> > deleting _164a.fdx
> > deleting _164a.fdt
> > deleting _163z_3.del
> > deleting _163z.tis
> > deleting _163z.tii
> > deleting _163z.prx
> > deleting _163z.nrm
> > deleting _163z.frq
> > deleting _163z.fnm
> > deleting _163z.fdx
> > deleting _163z.fdt
> > deleting _163o_3.del
> > deleting _163o.tis
> > deleting _163o.tii
> > deleting _163o.prx
> > deleting _163o.nrm
> > deleting _163o.frq
> > deleting _163o.fnm
> > deleting _163o.fdx
> > deleting _163o.fdt
> > deleting _163d_4.del
> > deleting _163d.tis
> > deleting _163d.tii
> > deleting _163d.prx
> > deleting _163d.nrm
> > deleting _163d.frq
> > deleting _163d.fnm
> > deleting _163d.fdx
> > deleting _163d.fdt
> > deleting _1632_6.del
> > deleting _1632.tis
> > deleting _1632.tii
> > deleting _1632.prx
> > deleting _1632.nrm
> > deleting _1632.frq
> > deleting _1632.fnm
> > deleting _1632.fdx
> > deleting _1632.fdt
> > deleting _162r_7.del
> > deleting _162r.tis
> > deleting _162r.tii
> > deleting _162r.prx
> > deleting _162r.nrm
> > deleting _162r.frq
> > deleting _162r.fnm
> > deleting _162r.fdx
> > deleting _162r.fdt
> > deleting _162g_d.del
> > deleting _162g.tis
> > deleting _162g.tii
> > deleting _162g.prx
> > deleting _162g.nrm
> > deleting _162g.frq
> > deleting _162g.fnm
> > deleting _162g.fdx
> > deleting _162g.fdt
> > deleting _1625_m.del
> > deleting _1625.tis
> > deleting _1625.tii
> > deleting _1625.prx
> > deleting _1625.nrm
> > deleting _1625.frq
> > deleting _1625.fnm
> > deleting _1625.fdx
> > deleting _1625.fdt
> > deleting _161u_w.del
> > deleting _161u.tis
> > deleting _161u.tii
> > deleting _161u.prx
> > deleting _161u.nrm
> > deleting _161u.frq
> > deleting _161u.fnm
> > deleting _161u.fdx
> > deleting _161u.fdt
> > deleting _161j_16.del
> > ./
> > _161j_17.del
> > _164m.fdt
> > _164m.fdx
> > _164m.fnm
> > _164m.frq
> > _164m.nrm
> > _164m.prx
> > _164m.tii
> > _164m.tis
> > _164m_1.del
> > _164x.fdt
> > _164x.fdx
> > _164x.fnm
> > _164x.frq
> > _164x.nrm
> > _164x.prx
> > _164x.tii
> > _164x.tis
> > _164x_1.del
> > segments.gen
> > segments_1inv
> >
> > sent 516 bytes received 105864302 bytes 30247090.86 bytes/sec
> > total size is 966107226 speedup is 9.13
> > + [[ -z search1 ]]
> > + [[ -z /opt/solr/logs ]]
> > + fixUser -M search1 -S /opt/solr/logs -d /opt/solr/data -V
> > + [[ -z tomcat5 ]]
> > ++ whoami
> > + [[ tomcat5 != tomcat5 ]]
> > ++ who -m
> > ++ cut '-d ' -f1
> > ++ sed '-es/^.*!//'
> > + oldwhoami=
> > + [[ '' == '' ]]
> > +++ pgrep -g0 snapinstaller
> > ++ tail -1
> > ++ cut -f1 '-d '
> > ++ ps h -Hfp 3621 3629 3630 3631
> > + oldwhoami=tomcat5
> > + [[ -z /opt/solr/data ]]
> > ++ echo /opt/solr/data
> > ++ cut -c1
> > + [[ / != \/ ]]
> > ++ echo /opt/solr/logs
> > ++ cut -c1
> > + [[ / != \/ ]]
> > ++ date +%s
> > + start=1189030205
> > + logMessage started by tomcat5
> > ++ timeStamp
> > ++ date '+%Y/%m/%d %H:%M:%S'
> > + echo 2007/09/05 15:10:05 started by tomcat5
> > + [[ -n '' ]]
> > + logMessage command: /opt/solr/bin/snapinstaller -M search1 -S /
> > opt/solr/logs -d /opt/solr/data -V
> > ++ timeStamp
> > ++ date '+%Y/%m/%d %H:%M:%S'
> > + echo 2007/09/05 15:10:05 command: /opt/solr/bin/snapinstaller -M
> > search1 -S /opt/solr/logs -d /opt/solr/data -V
> > + [[ -n '' ]]
> > ++ ls /opt/solr/data
> > ++ grep 'snapshot\.'
> > ++ grep -v wip
> > ++ sort -r
> > ++ head -1
> > + name=temp-snapshot.20070905150504
> > + trap 'echo "caught INT/TERM, exiting now but partial installation
> > may have already occured";/bin/rm -rf ${data_dir"/index.tmp$
> > $;logExit aborted 13' INT TERM
> > + [[ temp-snapshot.20070905150504 == '' ]]
> > + name=/opt/solr/data/temp-snapshot.20070905150504
> > ++ cat /opt/solr/logs/snapshot.current
> > + [[ /opt/solr/data/temp-snapshot.20070905150504 == /opt/solr/data/
> > snapshot.20070905150005 ]]
> > + ssh -o StrictHostKeyChecking=no search1 mkdir -p /opt/solr/logs
> > + logMessage installing snapshot /opt/solr/data/temp-snapshot.
> > 20070905150504
> > ++ timeStamp
> > ++ date '+%Y/%m/%d %H:%M:%S'
> > + echo 2007/09/05 15:10:05 installing snapshot /opt/solr/data/temp-
> > snapshot.20070905150504
> > + [[ -n '' ]]
> > + cp -lr /opt/solr/data/temp-snapshot.20070905150504/ /opt/solr/
> > data/index.tmp3621
> > + /bin/rm -rf /opt/solr/data/index
> > + mv -f /opt/solr/data/index.tmp3621 /opt/solr/data/index
> > + echo /opt/solr/data/temp-snapshot.20070905150504
> > ++ uname -n
> > + scp -q -o StrictHostKeyChecking=no /opt/solr/logs/
> > snapshot.current search1:/opt/solr/logs/snapshot.current.search2
> > + logMessage notifing Solr to open a new Searcher
> > ++ timeStamp
> > ++ date '+%Y/%m/%d %H:%M:%S'
> > + echo 2007/09/05 15:10:06 notifing Solr to open a new Searcher
> > + [[ -n '' ]]
> > + /opt/solr/bin/commit
> > + [[ 0 != 0 ]]
> > + logExit ended 0
> > ++ date +%s
> > + end=1189030210
> > ++ expr 1189030210 - 1189030205
> > + diff=5
> > ++ timeStamp
> > ++ date '+%Y/%m/%d %H:%M:%S'
> > + echo '2007/09/05 15:10:10 ended (elapsed time: 5 sec)'
> > + exit 0
> >
> >
> >
> > +--------------------------------------------------------+
> > | Matthew Runo
> > | Zappos Development
> > | mruno@zappos.com
> > | 702-943-7833
> > +--------------------------------------------------------+
> >
> >
> > On Aug 31, 2007, at 6:11 PM, Bill Au wrote:
> >
> >>> latest snapshot /opt/solr/data/temp-snapshot.20070816120113 already
> >>> installed
> >>
> >> It looks like you have a directory named temp-snapshot.20070816120113
> >> in your data directory. You should remove it. One of the other
> >> script might have left that behind somehow.
> >>
> >> I will update the snapinstaller script to ignore non-snapshot when
> >> looking for the latest snapshot to install.
> >>
> >> Bill
> >>
> >> On 8/31/07, Matthew Runo <mr...@zappos.com> wrote:
> >>> Hello!
> >>>
> >>> On a somewhat related note, our replication seems very much broken.
> >>> I've added -v to all my cron jobs, and I think I've seen the error
> >>> (below).
> >>>
> >>> As you can see, it's rsyncing an updated index, but then doesn't
> >>> seem
> >>> to know to install it. I'm not sure why though.. no errors are
> >>> reported anywhere via the -v. Any help would be most appreciated,
> >>> I'm
> >>> sure I'm just missing something. You can see the cronjob
> >>> command in
> >>> the subject of the forwarded message.
> >>>
> >>> +--------------------------------------------------------+
> >>> | Matthew Runo
> >>> | Zappos Development
> >>> | mruno@zappos.com
> >>> | 702-943-7833
> >>> +--------------------------------------------------------+
> >>>
> >>>
> >>> Begin forwarded message:
> >>>
> >>>> From: root@web03.zappos.com (Cron Daemon)
> >>>> Date: August 31, 2007 1:02:36 PM PDT
> >>>> To: mruno@zappos.com
> >>>> Subject: Cron <to...@search2> /opt/solr/bin/snappuller -M search1
> >>>> -P 18080 -D /opt/solr/data -S /opt/solr/logs -d /opt/solr/data -v;/
> >>>> opt/solr/bin/snapinstaller -M search1 -S /opt/solr/logs -d /opt/
> >>>> solr/data -v
> >>>>
> >>>> started by tomcat5
> >>>> command: /opt/solr/bin/snappuller -M search1 -P 18080 -D /opt/solr/
> >>>> data -S /opt/solr/logs -d /opt/solr/data -v
> >>>> pulling snapshot snapshot.20070831130005
> >>>> receiving file list ... done
> >>>> deleting segments_19zm
> >>>> deleting _10bg.tis
> >>>> deleting _10bg.tii
> >>>> deleting _10bg.prx
> >>>> deleting _10bg.nrm
> >>>> deleting _10bg.frq
> >>>> deleting _10bg.fnm
> >>>> deleting _10bg.fdx
> >>>> deleting _10bg.fdt
> >>>> ./
> >>>> _14ff.fdt
> >>>> _14ff.fdx
> >>>> _14ff.fnm
> >>>> _14ff.frq
> >>>> _14ff.nrm
> >>>> _14ff.prx
> >>>> _14ff.tii
> >>>> _14ff.tis
> >>>> _14ff_9.del
> >>>> _14fq.fdt
> >>>> _14fq.fdx
> >>>> _14fq.fnm
> >>>> _14fq.frq
> >>>> _14fq.nrm
> >>>> _14fq.prx
> >>>> _14fq.tii
> >>>> _14fq.tis
> >>>> _14fq_3.del
> >>>> _14fr.fdt
> >>>> _14fr.fdx
> >>>> _14fr.fnm
> >>>> _14fr.frq
> >>>> _14fr.nrm
> >>>> _14fr.prx
> >>>> _14fr.tii
> >>>> _14fr.tis
> >>>> _14fr_2.del
> >>>> _14fs.fdt
> >>>> _14fs.fdx
> >>>> _14fs.fnm
> >>>> _14fs.frq
> >>>> _14fs.nrm
> >>>> _14fs.prx
> >>>> _14fs.tii
> >>>> _14fs.tis
> >>>> _14fs_1.del
> >>>> segments.gen
> >>>> segments_1fza
> >>>> write.lock
> >>>>
> >>>> sent 871 bytes received 843185604 bytes 24440187.68 bytes/sec
> >>>> total size is 843080453 speedup is 1.00
> >>>> started by tomcat5
> >>>> command: /opt/solr/bin/snapinstaller -M search1 -S /opt/solr/logs -
> >>>> d /opt/solr/data -v
> >>>> latest snapshot /opt/solr/data/temp-snapshot.20070816120113 already
> >>>> installed
> >>>>
> >>>
> >>>
> >>
> >
>
>
Re: Replication broken.. no helpful errors?
Posted by Matthew Runo <mr...@zappos.com>.
If it helps anyone, this index is around a gig in size.
+--------------------------------------------------------+
| Matthew Runo
| Zappos Development
| mruno@zappos.com
| 702-943-7833
+--------------------------------------------------------+
On Sep 5, 2007, at 3:14 PM, Matthew Runo wrote:
> It seems that the scripts cannot open new searchers at the end of
> the process, for some reason. Here's a message from cron, but I'm
> not sure what to make of it... It looks like the files properly
> copied over, but failed the install. I removed the temp* directory,
> but still SOLR could not launch a new searcher. I don't see any
> activity in catalina.out though...
>
>
> started by tomcat5
> command: /opt/solr/bin/snappuller -M search1 -P 18080 -D /opt/solr/
> data -S /opt/solr/logs -d /opt/solr/data -v
> pulling snapshot temp-snapshot.20070905150504
> receiving file list ... done
> deleting segments_1ine
> deleting _164h_1.del
> deleting _164h.tis
> deleting _164h.tii
> deleting _164h.prx
> deleting _164h.nrm
> deleting _164h.frq
> deleting _164h.fnm
> deleting _164h.fdx
> deleting _164h.fdt
> deleting _164g_1.del
> deleting _164g.tis
> deleting _164g.tii
> deleting _164g.prx
> deleting _164g.nrm
> deleting _164g.frq
> deleting _164g.fnm
> deleting _164g.fdx
> deleting _164g.fdt
> deleting _164f_1.del
> deleting _164f.tis
> deleting _164f.tii
> deleting _164f.prx
> deleting _164f.nrm
> deleting _164f.frq
> deleting _164f.fnm
> deleting _164f.fdx
> deleting _164f.fdt
> deleting _164e_1.del
> deleting _164e.tis
> deleting _164e.tii
> deleting _164e.prx
> deleting _164e.nrm
> deleting _164e.frq
> deleting _164e.fnm
> deleting _164e.fdx
> deleting _164e.fdt
> deleting _164d_1.del
> deleting _164d.tis
> deleting _164d.tii
> deleting _164d.prx
> deleting _164d.nrm
> deleting _164d.frq
> deleting _164d.fnm
> deleting _164d.fdx
> deleting _164d.fdt
> deleting _164c_1.del
> deleting _164c.tis
> deleting _164c.tii
> deleting _164c.prx
> deleting _164c.nrm
> deleting _164c.frq
> deleting _164c.fnm
> deleting _164c.fdx
> deleting _164c.fdt
> deleting _164b_1.del
> deleting _164b.tis
> deleting _164b.tii
> deleting _164b.prx
> deleting _164b.nrm
> deleting _164b.frq
> deleting _164b.fnm
> deleting _164b.fdx
> deleting _164b.fdt
> deleting _164a_1.del
> deleting _164a.tis
> deleting _164a.tii
> deleting _164a.prx
> deleting _164a.nrm
> deleting _164a.frq
> deleting _164a.fnm
> deleting _164a.fdx
> deleting _164a.fdt
> deleting _163z_3.del
> deleting _163z.tis
> deleting _163z.tii
> deleting _163z.prx
> deleting _163z.nrm
> deleting _163z.frq
> deleting _163z.fnm
> deleting _163z.fdx
> deleting _163z.fdt
> deleting _163o_3.del
> deleting _163o.tis
> deleting _163o.tii
> deleting _163o.prx
> deleting _163o.nrm
> deleting _163o.frq
> deleting _163o.fnm
> deleting _163o.fdx
> deleting _163o.fdt
> deleting _163d_4.del
> deleting _163d.tis
> deleting _163d.tii
> deleting _163d.prx
> deleting _163d.nrm
> deleting _163d.frq
> deleting _163d.fnm
> deleting _163d.fdx
> deleting _163d.fdt
> deleting _1632_6.del
> deleting _1632.tis
> deleting _1632.tii
> deleting _1632.prx
> deleting _1632.nrm
> deleting _1632.frq
> deleting _1632.fnm
> deleting _1632.fdx
> deleting _1632.fdt
> deleting _162r_7.del
> deleting _162r.tis
> deleting _162r.tii
> deleting _162r.prx
> deleting _162r.nrm
> deleting _162r.frq
> deleting _162r.fnm
> deleting _162r.fdx
> deleting _162r.fdt
> deleting _162g_d.del
> deleting _162g.tis
> deleting _162g.tii
> deleting _162g.prx
> deleting _162g.nrm
> deleting _162g.frq
> deleting _162g.fnm
> deleting _162g.fdx
> deleting _162g.fdt
> deleting _1625_m.del
> deleting _1625.tis
> deleting _1625.tii
> deleting _1625.prx
> deleting _1625.nrm
> deleting _1625.frq
> deleting _1625.fnm
> deleting _1625.fdx
> deleting _1625.fdt
> deleting _161u_w.del
> deleting _161u.tis
> deleting _161u.tii
> deleting _161u.prx
> deleting _161u.nrm
> deleting _161u.frq
> deleting _161u.fnm
> deleting _161u.fdx
> deleting _161u.fdt
> deleting _161j_16.del
> ./
> _161j_17.del
> _164m.fdt
> _164m.fdx
> _164m.fnm
> _164m.frq
> _164m.nrm
> _164m.prx
> _164m.tii
> _164m.tis
> _164m_1.del
> _164x.fdt
> _164x.fdx
> _164x.fnm
> _164x.frq
> _164x.nrm
> _164x.prx
> _164x.tii
> _164x.tis
> _164x_1.del
> segments.gen
> segments_1inv
>
> sent 516 bytes received 105864302 bytes 30247090.86 bytes/sec
> total size is 966107226 speedup is 9.13
> + [[ -z search1 ]]
> + [[ -z /opt/solr/logs ]]
> + fixUser -M search1 -S /opt/solr/logs -d /opt/solr/data -V
> + [[ -z tomcat5 ]]
> ++ whoami
> + [[ tomcat5 != tomcat5 ]]
> ++ who -m
> ++ cut '-d ' -f1
> ++ sed '-es/^.*!//'
> + oldwhoami=
> + [[ '' == '' ]]
> +++ pgrep -g0 snapinstaller
> ++ tail -1
> ++ cut -f1 '-d '
> ++ ps h -Hfp 3621 3629 3630 3631
> + oldwhoami=tomcat5
> + [[ -z /opt/solr/data ]]
> ++ echo /opt/solr/data
> ++ cut -c1
> + [[ / != \/ ]]
> ++ echo /opt/solr/logs
> ++ cut -c1
> + [[ / != \/ ]]
> ++ date +%s
> + start=1189030205
> + logMessage started by tomcat5
> ++ timeStamp
> ++ date '+%Y/%m/%d %H:%M:%S'
> + echo 2007/09/05 15:10:05 started by tomcat5
> + [[ -n '' ]]
> + logMessage command: /opt/solr/bin/snapinstaller -M search1 -S /
> opt/solr/logs -d /opt/solr/data -V
> ++ timeStamp
> ++ date '+%Y/%m/%d %H:%M:%S'
> + echo 2007/09/05 15:10:05 command: /opt/solr/bin/snapinstaller -M
> search1 -S /opt/solr/logs -d /opt/solr/data -V
> + [[ -n '' ]]
> ++ ls /opt/solr/data
> ++ grep 'snapshot\.'
> ++ grep -v wip
> ++ sort -r
> ++ head -1
> + name=temp-snapshot.20070905150504
> + trap 'echo "caught INT/TERM, exiting now but partial installation
> may have already occured";/bin/rm -rf ${data_dir"/index.tmp$
> $;logExit aborted 13' INT TERM
> + [[ temp-snapshot.20070905150504 == '' ]]
> + name=/opt/solr/data/temp-snapshot.20070905150504
> ++ cat /opt/solr/logs/snapshot.current
> + [[ /opt/solr/data/temp-snapshot.20070905150504 == /opt/solr/data/
> snapshot.20070905150005 ]]
> + ssh -o StrictHostKeyChecking=no search1 mkdir -p /opt/solr/logs
> + logMessage installing snapshot /opt/solr/data/temp-snapshot.
> 20070905150504
> ++ timeStamp
> ++ date '+%Y/%m/%d %H:%M:%S'
> + echo 2007/09/05 15:10:05 installing snapshot /opt/solr/data/temp-
> snapshot.20070905150504
> + [[ -n '' ]]
> + cp -lr /opt/solr/data/temp-snapshot.20070905150504/ /opt/solr/
> data/index.tmp3621
> + /bin/rm -rf /opt/solr/data/index
> + mv -f /opt/solr/data/index.tmp3621 /opt/solr/data/index
> + echo /opt/solr/data/temp-snapshot.20070905150504
> ++ uname -n
> + scp -q -o StrictHostKeyChecking=no /opt/solr/logs/
> snapshot.current search1:/opt/solr/logs/snapshot.current.search2
> + logMessage notifing Solr to open a new Searcher
> ++ timeStamp
> ++ date '+%Y/%m/%d %H:%M:%S'
> + echo 2007/09/05 15:10:06 notifing Solr to open a new Searcher
> + [[ -n '' ]]
> + /opt/solr/bin/commit
> + [[ 0 != 0 ]]
> + logExit ended 0
> ++ date +%s
> + end=1189030210
> ++ expr 1189030210 - 1189030205
> + diff=5
> ++ timeStamp
> ++ date '+%Y/%m/%d %H:%M:%S'
> + echo '2007/09/05 15:10:10 ended (elapsed time: 5 sec)'
> + exit 0
>
>
>
> +--------------------------------------------------------+
> | Matthew Runo
> | Zappos Development
> | mruno@zappos.com
> | 702-943-7833
> +--------------------------------------------------------+
>
>
> On Aug 31, 2007, at 6:11 PM, Bill Au wrote:
>
>>> latest snapshot /opt/solr/data/temp-snapshot.20070816120113 already
>>> installed
>>
>> It looks like you have a directory named temp-snapshot.20070816120113
>> in your data directory. You should remove it. One of the other
>> script might have left that behind somehow.
>>
>> I will update the snapinstaller script to ignore non-snapshot when
>> looking for the latest snapshot to install.
>>
>> Bill
>>
>> On 8/31/07, Matthew Runo <mr...@zappos.com> wrote:
>>> Hello!
>>>
>>> On a somewhat related note, our replication seems very much broken.
>>> I've added -v to all my cron jobs, and I think I've seen the error
>>> (below).
>>>
>>> As you can see, it's rsyncing an updated index, but then doesn't
>>> seem
>>> to know to install it. I'm not sure why though.. no errors are
>>> reported anywhere via the -v. Any help would be most appreciated,
>>> I'm
>>> sure I'm just missing something. You can see the cronjob
>>> command in
>>> the subject of the forwarded message.
>>>
>>> +--------------------------------------------------------+
>>> | Matthew Runo
>>> | Zappos Development
>>> | mruno@zappos.com
>>> | 702-943-7833
>>> +--------------------------------------------------------+
>>>
>>>
>>> Begin forwarded message:
>>>
>>>> From: root@web03.zappos.com (Cron Daemon)
>>>> Date: August 31, 2007 1:02:36 PM PDT
>>>> To: mruno@zappos.com
>>>> Subject: Cron <to...@search2> /opt/solr/bin/snappuller -M search1
>>>> -P 18080 -D /opt/solr/data -S /opt/solr/logs -d /opt/solr/data -v;/
>>>> opt/solr/bin/snapinstaller -M search1 -S /opt/solr/logs -d /opt/
>>>> solr/data -v
>>>>
>>>> started by tomcat5
>>>> command: /opt/solr/bin/snappuller -M search1 -P 18080 -D /opt/solr/
>>>> data -S /opt/solr/logs -d /opt/solr/data -v
>>>> pulling snapshot snapshot.20070831130005
>>>> receiving file list ... done
>>>> deleting segments_19zm
>>>> deleting _10bg.tis
>>>> deleting _10bg.tii
>>>> deleting _10bg.prx
>>>> deleting _10bg.nrm
>>>> deleting _10bg.frq
>>>> deleting _10bg.fnm
>>>> deleting _10bg.fdx
>>>> deleting _10bg.fdt
>>>> ./
>>>> _14ff.fdt
>>>> _14ff.fdx
>>>> _14ff.fnm
>>>> _14ff.frq
>>>> _14ff.nrm
>>>> _14ff.prx
>>>> _14ff.tii
>>>> _14ff.tis
>>>> _14ff_9.del
>>>> _14fq.fdt
>>>> _14fq.fdx
>>>> _14fq.fnm
>>>> _14fq.frq
>>>> _14fq.nrm
>>>> _14fq.prx
>>>> _14fq.tii
>>>> _14fq.tis
>>>> _14fq_3.del
>>>> _14fr.fdt
>>>> _14fr.fdx
>>>> _14fr.fnm
>>>> _14fr.frq
>>>> _14fr.nrm
>>>> _14fr.prx
>>>> _14fr.tii
>>>> _14fr.tis
>>>> _14fr_2.del
>>>> _14fs.fdt
>>>> _14fs.fdx
>>>> _14fs.fnm
>>>> _14fs.frq
>>>> _14fs.nrm
>>>> _14fs.prx
>>>> _14fs.tii
>>>> _14fs.tis
>>>> _14fs_1.del
>>>> segments.gen
>>>> segments_1fza
>>>> write.lock
>>>>
>>>> sent 871 bytes received 843185604 bytes 24440187.68 bytes/sec
>>>> total size is 843080453 speedup is 1.00
>>>> started by tomcat5
>>>> command: /opt/solr/bin/snapinstaller -M search1 -S /opt/solr/logs -
>>>> d /opt/solr/data -v
>>>> latest snapshot /opt/solr/data/temp-snapshot.20070816120113 already
>>>> installed
>>>>
>>>
>>>
>>
>
Re: Replication broken.. no helpful errors?
Posted by Matthew Runo <mr...@zappos.com>.
It seems that the scripts cannot open new searchers at the end of the
process, for some reason. Here's a message from cron, but I'm not
sure what to make of it... It looks like the files properly copied
over, but failed the install. I removed the temp* directory, but
still SOLR could not launch a new searcher. I don't see any activity
in catalina.out though...
started by tomcat5
command: /opt/solr/bin/snappuller -M search1 -P 18080 -D /opt/solr/
data -S /opt/solr/logs -d /opt/solr/data -v
pulling snapshot temp-snapshot.20070905150504
receiving file list ... done
deleting segments_1ine
deleting _164h_1.del
deleting _164h.tis
deleting _164h.tii
deleting _164h.prx
deleting _164h.nrm
deleting _164h.frq
deleting _164h.fnm
deleting _164h.fdx
deleting _164h.fdt
deleting _164g_1.del
deleting _164g.tis
deleting _164g.tii
deleting _164g.prx
deleting _164g.nrm
deleting _164g.frq
deleting _164g.fnm
deleting _164g.fdx
deleting _164g.fdt
deleting _164f_1.del
deleting _164f.tis
deleting _164f.tii
deleting _164f.prx
deleting _164f.nrm
deleting _164f.frq
deleting _164f.fnm
deleting _164f.fdx
deleting _164f.fdt
deleting _164e_1.del
deleting _164e.tis
deleting _164e.tii
deleting _164e.prx
deleting _164e.nrm
deleting _164e.frq
deleting _164e.fnm
deleting _164e.fdx
deleting _164e.fdt
deleting _164d_1.del
deleting _164d.tis
deleting _164d.tii
deleting _164d.prx
deleting _164d.nrm
deleting _164d.frq
deleting _164d.fnm
deleting _164d.fdx
deleting _164d.fdt
deleting _164c_1.del
deleting _164c.tis
deleting _164c.tii
deleting _164c.prx
deleting _164c.nrm
deleting _164c.frq
deleting _164c.fnm
deleting _164c.fdx
deleting _164c.fdt
deleting _164b_1.del
deleting _164b.tis
deleting _164b.tii
deleting _164b.prx
deleting _164b.nrm
deleting _164b.frq
deleting _164b.fnm
deleting _164b.fdx
deleting _164b.fdt
deleting _164a_1.del
deleting _164a.tis
deleting _164a.tii
deleting _164a.prx
deleting _164a.nrm
deleting _164a.frq
deleting _164a.fnm
deleting _164a.fdx
deleting _164a.fdt
deleting _163z_3.del
deleting _163z.tis
deleting _163z.tii
deleting _163z.prx
deleting _163z.nrm
deleting _163z.frq
deleting _163z.fnm
deleting _163z.fdx
deleting _163z.fdt
deleting _163o_3.del
deleting _163o.tis
deleting _163o.tii
deleting _163o.prx
deleting _163o.nrm
deleting _163o.frq
deleting _163o.fnm
deleting _163o.fdx
deleting _163o.fdt
deleting _163d_4.del
deleting _163d.tis
deleting _163d.tii
deleting _163d.prx
deleting _163d.nrm
deleting _163d.frq
deleting _163d.fnm
deleting _163d.fdx
deleting _163d.fdt
deleting _1632_6.del
deleting _1632.tis
deleting _1632.tii
deleting _1632.prx
deleting _1632.nrm
deleting _1632.frq
deleting _1632.fnm
deleting _1632.fdx
deleting _1632.fdt
deleting _162r_7.del
deleting _162r.tis
deleting _162r.tii
deleting _162r.prx
deleting _162r.nrm
deleting _162r.frq
deleting _162r.fnm
deleting _162r.fdx
deleting _162r.fdt
deleting _162g_d.del
deleting _162g.tis
deleting _162g.tii
deleting _162g.prx
deleting _162g.nrm
deleting _162g.frq
deleting _162g.fnm
deleting _162g.fdx
deleting _162g.fdt
deleting _1625_m.del
deleting _1625.tis
deleting _1625.tii
deleting _1625.prx
deleting _1625.nrm
deleting _1625.frq
deleting _1625.fnm
deleting _1625.fdx
deleting _1625.fdt
deleting _161u_w.del
deleting _161u.tis
deleting _161u.tii
deleting _161u.prx
deleting _161u.nrm
deleting _161u.frq
deleting _161u.fnm
deleting _161u.fdx
deleting _161u.fdt
deleting _161j_16.del
./
_161j_17.del
_164m.fdt
_164m.fdx
_164m.fnm
_164m.frq
_164m.nrm
_164m.prx
_164m.tii
_164m.tis
_164m_1.del
_164x.fdt
_164x.fdx
_164x.fnm
_164x.frq
_164x.nrm
_164x.prx
_164x.tii
_164x.tis
_164x_1.del
segments.gen
segments_1inv
sent 516 bytes received 105864302 bytes 30247090.86 bytes/sec
total size is 966107226 speedup is 9.13
+ [[ -z search1 ]]
+ [[ -z /opt/solr/logs ]]
+ fixUser -M search1 -S /opt/solr/logs -d /opt/solr/data -V
+ [[ -z tomcat5 ]]
++ whoami
+ [[ tomcat5 != tomcat5 ]]
++ who -m
++ cut '-d ' -f1
++ sed '-es/^.*!//'
+ oldwhoami=
+ [[ '' == '' ]]
+++ pgrep -g0 snapinstaller
++ tail -1
++ cut -f1 '-d '
++ ps h -Hfp 3621 3629 3630 3631
+ oldwhoami=tomcat5
+ [[ -z /opt/solr/data ]]
++ echo /opt/solr/data
++ cut -c1
+ [[ / != \/ ]]
++ echo /opt/solr/logs
++ cut -c1
+ [[ / != \/ ]]
++ date +%s
+ start=1189030205
+ logMessage started by tomcat5
++ timeStamp
++ date '+%Y/%m/%d %H:%M:%S'
+ echo 2007/09/05 15:10:05 started by tomcat5
+ [[ -n '' ]]
+ logMessage command: /opt/solr/bin/snapinstaller -M search1 -S /opt/
solr/logs -d /opt/solr/data -V
++ timeStamp
++ date '+%Y/%m/%d %H:%M:%S'
+ echo 2007/09/05 15:10:05 command: /opt/solr/bin/snapinstaller -M
search1 -S /opt/solr/logs -d /opt/solr/data -V
+ [[ -n '' ]]
++ ls /opt/solr/data
++ grep 'snapshot\.'
++ grep -v wip
++ sort -r
++ head -1
+ name=temp-snapshot.20070905150504
+ trap 'echo "caught INT/TERM, exiting now but partial installation
may have already occured";/bin/rm -rf ${data_dir"/index.tmp$$;logExit
aborted 13' INT TERM
+ [[ temp-snapshot.20070905150504 == '' ]]
+ name=/opt/solr/data/temp-snapshot.20070905150504
++ cat /opt/solr/logs/snapshot.current
+ [[ /opt/solr/data/temp-snapshot.20070905150504 == /opt/solr/data/
snapshot.20070905150005 ]]
+ ssh -o StrictHostKeyChecking=no search1 mkdir -p /opt/solr/logs
+ logMessage installing snapshot /opt/solr/data/temp-snapshot.
20070905150504
++ timeStamp
++ date '+%Y/%m/%d %H:%M:%S'
+ echo 2007/09/05 15:10:05 installing snapshot /opt/solr/data/temp-
snapshot.20070905150504
+ [[ -n '' ]]
+ cp -lr /opt/solr/data/temp-snapshot.20070905150504/ /opt/solr/data/
index.tmp3621
+ /bin/rm -rf /opt/solr/data/index
+ mv -f /opt/solr/data/index.tmp3621 /opt/solr/data/index
+ echo /opt/solr/data/temp-snapshot.20070905150504
++ uname -n
+ scp -q -o StrictHostKeyChecking=no /opt/solr/logs/snapshot.current
search1:/opt/solr/logs/snapshot.current.search2
+ logMessage notifing Solr to open a new Searcher
++ timeStamp
++ date '+%Y/%m/%d %H:%M:%S'
+ echo 2007/09/05 15:10:06 notifing Solr to open a new Searcher
+ [[ -n '' ]]
+ /opt/solr/bin/commit
+ [[ 0 != 0 ]]
+ logExit ended 0
++ date +%s
+ end=1189030210
++ expr 1189030210 - 1189030205
+ diff=5
++ timeStamp
++ date '+%Y/%m/%d %H:%M:%S'
+ echo '2007/09/05 15:10:10 ended (elapsed time: 5 sec)'
+ exit 0
+--------------------------------------------------------+
| Matthew Runo
| Zappos Development
| mruno@zappos.com
| 702-943-7833
+--------------------------------------------------------+
On Aug 31, 2007, at 6:11 PM, Bill Au wrote:
>> latest snapshot /opt/solr/data/temp-snapshot.20070816120113 already
>> installed
>
> It looks like you have a directory named temp-snapshot.20070816120113
> in your data directory. You should remove it. One of the other
> script might have left that behind somehow.
>
> I will update the snapinstaller script to ignore non-snapshot when
> looking for the latest snapshot to install.
>
> Bill
>
> On 8/31/07, Matthew Runo <mr...@zappos.com> wrote:
>> Hello!
>>
>> On a somewhat related note, our replication seems very much broken.
>> I've added -v to all my cron jobs, and I think I've seen the error
>> (below).
>>
>> As you can see, it's rsyncing an updated index, but then doesn't seem
>> to know to install it. I'm not sure why though.. no errors are
>> reported anywhere via the -v. Any help would be most appreciated, I'm
>> sure I'm just missing something. You can see the cronjob command in
>> the subject of the forwarded message.
>>
>> +--------------------------------------------------------+
>> | Matthew Runo
>> | Zappos Development
>> | mruno@zappos.com
>> | 702-943-7833
>> +--------------------------------------------------------+
>>
>>
>> Begin forwarded message:
>>
>>> From: root@web03.zappos.com (Cron Daemon)
>>> Date: August 31, 2007 1:02:36 PM PDT
>>> To: mruno@zappos.com
>>> Subject: Cron <to...@search2> /opt/solr/bin/snappuller -M search1
>>> -P 18080 -D /opt/solr/data -S /opt/solr/logs -d /opt/solr/data -v;/
>>> opt/solr/bin/snapinstaller -M search1 -S /opt/solr/logs -d /opt/
>>> solr/data -v
>>>
>>> started by tomcat5
>>> command: /opt/solr/bin/snappuller -M search1 -P 18080 -D /opt/solr/
>>> data -S /opt/solr/logs -d /opt/solr/data -v
>>> pulling snapshot snapshot.20070831130005
>>> receiving file list ... done
>>> deleting segments_19zm
>>> deleting _10bg.tis
>>> deleting _10bg.tii
>>> deleting _10bg.prx
>>> deleting _10bg.nrm
>>> deleting _10bg.frq
>>> deleting _10bg.fnm
>>> deleting _10bg.fdx
>>> deleting _10bg.fdt
>>> ./
>>> _14ff.fdt
>>> _14ff.fdx
>>> _14ff.fnm
>>> _14ff.frq
>>> _14ff.nrm
>>> _14ff.prx
>>> _14ff.tii
>>> _14ff.tis
>>> _14ff_9.del
>>> _14fq.fdt
>>> _14fq.fdx
>>> _14fq.fnm
>>> _14fq.frq
>>> _14fq.nrm
>>> _14fq.prx
>>> _14fq.tii
>>> _14fq.tis
>>> _14fq_3.del
>>> _14fr.fdt
>>> _14fr.fdx
>>> _14fr.fnm
>>> _14fr.frq
>>> _14fr.nrm
>>> _14fr.prx
>>> _14fr.tii
>>> _14fr.tis
>>> _14fr_2.del
>>> _14fs.fdt
>>> _14fs.fdx
>>> _14fs.fnm
>>> _14fs.frq
>>> _14fs.nrm
>>> _14fs.prx
>>> _14fs.tii
>>> _14fs.tis
>>> _14fs_1.del
>>> segments.gen
>>> segments_1fza
>>> write.lock
>>>
>>> sent 871 bytes received 843185604 bytes 24440187.68 bytes/sec
>>> total size is 843080453 speedup is 1.00
>>> started by tomcat5
>>> command: /opt/solr/bin/snapinstaller -M search1 -S /opt/solr/logs -
>>> d /opt/solr/data -v
>>> latest snapshot /opt/solr/data/temp-snapshot.20070816120113 already
>>> installed
>>>
>>
>>
>
Re: Replication broken.. no helpful errors?
Posted by Bill Au <bi...@gmail.com>.
> latest snapshot /opt/solr/data/temp-snapshot.20070816120113 already
> installed
It looks like you have a directory named temp-snapshot.20070816120113
in your data directory. You should remove it. One of the other
script might have left that behind somehow.
I will update the snapinstaller script to ignore non-snapshot when
looking for the latest snapshot to install.
Bill
On 8/31/07, Matthew Runo <mr...@zappos.com> wrote:
> Hello!
>
> On a somewhat related note, our replication seems very much broken.
> I've added -v to all my cron jobs, and I think I've seen the error
> (below).
>
> As you can see, it's rsyncing an updated index, but then doesn't seem
> to know to install it. I'm not sure why though.. no errors are
> reported anywhere via the -v. Any help would be most appreciated, I'm
> sure I'm just missing something. You can see the cronjob command in
> the subject of the forwarded message.
>
> +--------------------------------------------------------+
> | Matthew Runo
> | Zappos Development
> | mruno@zappos.com
> | 702-943-7833
> +--------------------------------------------------------+
>
>
> Begin forwarded message:
>
> > From: root@web03.zappos.com (Cron Daemon)
> > Date: August 31, 2007 1:02:36 PM PDT
> > To: mruno@zappos.com
> > Subject: Cron <to...@search2> /opt/solr/bin/snappuller -M search1
> > -P 18080 -D /opt/solr/data -S /opt/solr/logs -d /opt/solr/data -v;/
> > opt/solr/bin/snapinstaller -M search1 -S /opt/solr/logs -d /opt/
> > solr/data -v
> >
> > started by tomcat5
> > command: /opt/solr/bin/snappuller -M search1 -P 18080 -D /opt/solr/
> > data -S /opt/solr/logs -d /opt/solr/data -v
> > pulling snapshot snapshot.20070831130005
> > receiving file list ... done
> > deleting segments_19zm
> > deleting _10bg.tis
> > deleting _10bg.tii
> > deleting _10bg.prx
> > deleting _10bg.nrm
> > deleting _10bg.frq
> > deleting _10bg.fnm
> > deleting _10bg.fdx
> > deleting _10bg.fdt
> > ./
> > _14ff.fdt
> > _14ff.fdx
> > _14ff.fnm
> > _14ff.frq
> > _14ff.nrm
> > _14ff.prx
> > _14ff.tii
> > _14ff.tis
> > _14ff_9.del
> > _14fq.fdt
> > _14fq.fdx
> > _14fq.fnm
> > _14fq.frq
> > _14fq.nrm
> > _14fq.prx
> > _14fq.tii
> > _14fq.tis
> > _14fq_3.del
> > _14fr.fdt
> > _14fr.fdx
> > _14fr.fnm
> > _14fr.frq
> > _14fr.nrm
> > _14fr.prx
> > _14fr.tii
> > _14fr.tis
> > _14fr_2.del
> > _14fs.fdt
> > _14fs.fdx
> > _14fs.fnm
> > _14fs.frq
> > _14fs.nrm
> > _14fs.prx
> > _14fs.tii
> > _14fs.tis
> > _14fs_1.del
> > segments.gen
> > segments_1fza
> > write.lock
> >
> > sent 871 bytes received 843185604 bytes 24440187.68 bytes/sec
> > total size is 843080453 speedup is 1.00
> > started by tomcat5
> > command: /opt/solr/bin/snapinstaller -M search1 -S /opt/solr/logs -
> > d /opt/solr/data -v
> > latest snapshot /opt/solr/data/temp-snapshot.20070816120113 already
> > installed
> >
>
>