Re: [Solr Wiki] Update of "TaskList" by YonikSeeley

[Chris Hostetter <ho...@fucit.org>]

: + * NTFS w/ WinXP or later does support hard links for files (and cygwin
: "ln" works for files).  The current replication scripts could be ported
: to cygwin.

Interesting.  THe one other potential hitch that comes to mind is the file
locking/deletion issue that forces Lucene to use a "deletable" file on
Windows -- I wonder if there would be any problematic interactions between
that code path and hardlinks?


-Hoss

Re: [Solr Wiki] Update of "TaskList" by YonikSeeley

[Yonik Seeley <yo...@apache.org>]

On 8/9/06, Yonik Seeley <yo...@apache.org> wrote:
> On 8/9/06, Chris Hostetter <ho...@fucit.org> wrote:
> > : + * NTFS w/ WinXP or later does support hard links for files (and cygwin
> > : "ln" works for files).  The current replication scripts could be ported
> > : to cygwin.
> >
> > Interesting.  THe one other potential hitch that comes to mind is the file
> > locking/deletion issue that forces Lucene to use a "deletable" file on
> > Windows -- I wonder if there would be any problematic interactions between
> > that code path and hardlinks?
>
> Yes, the behavior seems different (under cygwin at least).  I can't
> get an open file to delete, even if it's a hard link to the original
> file opened.
>
> The rsync dist method relies on being able to delete open files and
> replace them with new versions.  Bummer.

Although all is not lost.  AFAIK, the files that change are read
completely when the IndexReader is opened and then closed (segments,
.del, deletable).  So one would be left with two problems:

1) switching to the new index... you can't remove or mv a directory in
use AFAIK.  Solr would need to be changed to open a new IndexReader on
the newest snapshot directory (or something to that effect).

2) cleaning up old segment files... same problem that Lucene has now.
 Lucene tries to delete files in deletable during a segment merge I
think... since a "slave" doesn't do merges, a little piece of separate
code would need to try and delete the deletable files periodically.
That could be completely outside Solr though.

If someone want's to tackle using Windows with this index replication
method, it does seem doable though.

-Yonik

Re: [Solr Wiki] Update of "TaskList" by YonikSeeley

[Yonik Seeley <yo...@apache.org>]

On 8/9/06, Chris Hostetter <ho...@fucit.org> wrote:
> : + * NTFS w/ WinXP or later does support hard links for files (and cygwin
> : "ln" works for files).  The current replication scripts could be ported
> : to cygwin.
>
> Interesting.  THe one other potential hitch that comes to mind is the file
> locking/deletion issue that forces Lucene to use a "deletable" file on
> Windows -- I wonder if there would be any problematic interactions between
> that code path and hardlinks?

Yes, the behavior seems different (under cygwin at least).  I can't
get an open file to delete, even if it's a hard link to the original
file opened.

The rsync dist method relies on being able to delete open files and
replace them with new versions.  Bummer.

-Yonik