You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@lucy.apache.org by goran kent <go...@gmail.com> on 2011/11/02 16:23:13 UTC
[lucy-user] Index state during merges
Hi,
If the following is true, then it will simplify my efforts:
during a merge of indexes to a $target, and an error occurs -- for
whatever reason (typically because of a broken source index) -- is
the $target buggered or still in a safe state?
I'm jumping through hoops preserving the sanity of target indexes
(which involves a cp of $target to a temp, merge subindexes to temp,
if success then mv temp to target). I'm doing all this cp/mv'ing
because of past experiences when trying to handle errors during
merging.
If I could perform merging on a single target (a target which is also
actively being searched on, btw, which is another reason I need to
simplify this process and get rid of the intermediate step of a $temp
target), and know that if an error occurs the target will always be
sane, it would help tremendously.
Put another way:
foreach $subindex { $bigindex->add_index($subindex); }
$bigindex->commit
if one of those subindexes is broken, will it break the entire
bigindex? or does Lucy work on temp files and if an error occurs the
bigindex is not harmed (ie, any changes are rolled back)? I know I
could test this theory, I just haven't had the time and would like to
draw on MH's knowledge.
Thanks
Re: [lucy-user] Index state during merges
Posted by Marvin Humphrey <ma...@rectangular.com>.
On Wed, Nov 02, 2011 at 11:59:34AM -0700, Nathan Kurz wrote:
> On Wed, Nov 2, 2011 at 11:29 AM, Marvin Humphrey <ma...@rectangular.com> wrote:
> > What do you mean by "broken source index"? Corrupt because bad UTF-8 snuck
> > in, and now it refuses to be read?
> >
> > Maybe we should consider scanning incoming fields for UTF-8 sanity after all.
> > I don't like making everybody pay this penalty -- small though it is --
> > because you'll only get bad UTF-8 if your indexing setup is broken somehow.
> > On the other hand, I don't like that once a single bad UTF-8 sequence makes it
> > through a commit, the index is irretrievably corrupt -- and you only discover
> > that after the damage is done.
>
> This seems like good practice. I don't know the exact routine, but
> the performance impact has to be minimal.
It turns out that the UTF-8 validity checking has been enabled after all -- for
several years now. :P
For the record, I benchmarked disabling it, and got a speedup on the indexing
benchmark by about half a percent. That's pretty dang small, especially since
the indexing benchmarker uses an unrealistically simple Analyzer.
> ps. I came across this possibly relevant discussion of a Perl
> 'feature' I wasn't aware of:
> http://jeremy.zawodny.com/blog/archives/010546.html
The patch to disable the sanity checking, pasted below my sig, involves
changing a method call from "Assign_Str" (which performs a validity check) to
"Assign_Trusted_Str" (which trusts that the string is valid and skips the
check). I deliberately gave the unsafe method a more cumbersome and
unambiguous name so that, so that anybody invoking the "wrong" method would
make their error in the "safe" direction -- think of it as "fail-safe"
interface design applied to method naming.
The primary influence on this design was the negative example set by Perl's
lousy UTF-8 input interface, as detailed in that Jeremy Zawodny blog post
(which I've read before). I wanted to do the opposite of this:
# Short, obvious name is unsafe -- no sanity checking.
open( my $fh, '<:utf8', $path ) or die $!;
# Long, obscure incantation is safe -- sanity checking is enabled.
open( my $fh, '<:encoding(UTF-8)', $path ) or die $!;
Marvin Humphrey
Index: ../perl/xs/Lucy/Index/Inverter.c
===================================================================
--- ../perl/xs/Lucy/Index/Inverter.c (revision 1196798)
+++ ../perl/xs/Lucy/Index/Inverter.c (working copy)
@@ -102,7 +102,7 @@
char *val_ptr = SvPVutf8(value_sv, val_len);
lucy_ViewCharBuf *value
= (lucy_ViewCharBuf*)inv_entry->value;
- Lucy_ViewCB_Assign_Str(value, val_ptr, val_len);
+ Lucy_ViewCB_Assign_Trusted_Str(value, val_ptr, val_len);
break;
}
case lucy_FType_BLOB: {
Re: [lucy-user] Index state during merges
Posted by Nathan Kurz <na...@verse.com>.
On Wed, Nov 2, 2011 at 11:29 AM, Marvin Humphrey <ma...@rectangular.com> wrote:
> What do you mean by "broken source index"? Corrupt because bad UTF-8 snuck
> in, and now it refuses to be read?
>
> Maybe we should consider scanning incoming fields for UTF-8 sanity after all.
> I don't like making everybody pay this penalty -- small though it is --
> because you'll only get bad UTF-8 if your indexing setup is broken somehow.
> On the other hand, I don't like that once a single bad UTF-8 sequence makes it
> through a commit, the index is irretrievably corrupt -- and you only discover
> that after the damage is done.
This seems like good practice. I don't know the exact routine, but
the performance impact has to be minimal. If it's already in
processor cache, any single pass through the string will be almost
free: it's already in cache, and I can't believe this step is CPU
limited. If you want, you could make it be Safe by default and Risky
by explicit option, but you might test first to be sure you even need
the option.
--nate
ps. I came across this possibly relevant discussion of a Perl
'feature' I wasn't aware of:
http://jeremy.zawodny.com/blog/archives/010546.html
Re: [lucy-user] Index state during merges
Posted by Grant McLean <gr...@catalyst.net.nz>.
On Wed, 2011-11-02 at 11:29 -0700, Marvin Humphrey wrote:
> Maybe we should consider scanning incoming fields for UTF-8 sanity after all.
> I don't like making everybody pay this penalty -- small though it is --
> because you'll only get bad UTF-8 if your indexing setup is broken somehow.
> On the other hand, I don't like that once a single bad UTF-8 sequence makes it
> through a commit, the index is irretrievably corrupt -- and you only discover
> that after the damage is done.
Perhaps the sanity checking could be controlled by an option that
defaults to 'on'. Then people who *know* their setup is UTF-8 clean can
call something like $indexer->no_validate_utf8() to avoid the
performance penalty.
Cheers
Grant
Re: [lucy-user] Index state during merges
Posted by Peter Karman <pe...@peknet.com>.
Grant McLean wrote on 11/2/11 2:20 PM:
> My https://metacpan.org/module/Encoding::FixLatin module is designed to
> help in dealing with that sort of situation and especially the case
> where a single document contains bytes from more than one encoding.
+1 to that.
Encoding::FixLatin is a great module. I depend on it in
Search::Tools::Transliterate, and in conjunction with Search::Tools::UTF8 I find
I can handle (or refuse to handle) just about any encoding mess that comes my way.
--
Peter Karman . http://peknet.com/ . peter@peknet.com
Re: [lucy-user] Index state during merges
Posted by Grant McLean <gr...@catalyst.net.nz>.
On Wed, 2011-11-02 at 21:03 +0200, goran kent wrote:
> WIth precisely this in mind, my code does some gymnastics to try and
> make sure bad utf8 doesn't make it in. But,... you never know when
> dealing with the vagaries of the 'tubes.
It's not uncommon for web sites to lie about the encoding of the content
they serve up. In particular, ASCII, UTF-8, ISO8859-1 and CP1252 are
all completely interchangeable - up to the point where they're not.
My https://metacpan.org/module/Encoding::FixLatin module is designed to
help in dealing with that sort of situation and especially the case
where a single document contains bytes from more than one encoding.
Cheers
Grant
Re: [lucy-user] Index state during merges
Posted by Marvin Humphrey <ma...@rectangular.com>.
On Thu, Nov 03, 2011 at 09:16:57AM +0200, goran kent wrote:
> On Wed, Nov 2, 2011 at 9:03 PM, goran kent <go...@gmail.com> wrote:
> > Is it ok to remove the seg_2 folder and lockfile and rename
> > snapshot_2.json to snapshot_1.json and likewise with schema_2.json ->
> > schema_1.json?
Here's the party line: Lucy's file format is not an official public API, and
thus hand editing of files is not supported.
Nevertheless, the human readable JSON format was deliberately selected to
facilitate troubleshooting and curious hacking, and it's gratifying to see
that you were able to able to intuit your way through. :)
> Removed:
> locks/write.lock
> seg_2/
FWIW, that write.lock file (which is also in JSON format) would have been
zapped the next time you opened an Indexer. The relevant method is
Lucy::Store::Lock#clear_stale.
http://incubator.apache.org/lucy/docs/perl/Lucy/Store/Lock.html#clear_stale-
clear_stale()
Release all locks that meet the following three conditions: the lock name
matches, the host id matches, and the process id that the lock was created
under no longer identifies an active process.
With regards to removing seg_2 (the most recent segment), that worked just as
you surmised. However, seg_2's data presumably existed at one time; it was
almost certainly wiped out when an Indexer session from one machine did not
realize that a write.lock file was owned by an active Indexer session running
on another machine.
> Renamed:
> schema_2.json -> schema_1.json
> snapshot_2.json -> snapshot_1.json
The number in the schema file name is not sync'd with the number in the
snapshot file name. Changing the number in the schema file name achieved no
purpose, but it was harmless since you also edited the name in the snapshot
file which referenced it.
Changing the number embedded in the snapshot file name had no effect since
there was only one snapshot file present in the index directory.
> At least, a search on the index returns results, that being the only way to
> check if the index is sane... (lisc - lucy-index-system-check anyone?)
An "lisc" tool would certainly be useful. I don't know of any other Lucy
community member who is working on something like that, and we don't have a
pressing need for it at my office (because we have me :P). If you want to
work on it yourself, though, I will help -- my top priority, in the near-term
and medium-term, is to help other developers help themselves.
> btw, lucy designers, thank you for using human-readable-editable
> files, it is the UNIX way.
We are on the same page, and I'm sure I can speak for Peter and the others who
were involved in that design when I say: it's great that you were able to make
use of them.
Cheers,
Marvin Humphrey
Re: [lucy-user] Index state during merges
Posted by goran kent <go...@gmail.com>.
On Wed, Nov 2, 2011 at 9:03 PM, goran kent <go...@gmail.com> wrote:
> Lucy::Index::Indexer->new failed (Failed to read seg_2
> S_try_open_elements at
> /home/projects/lucy/lucy/perl/../core/Lucy/Index/PolyReader.c line 251
>
> If I inspect the index it looks like the seg_2 folder is empty.
>
> /.../snapshot_2.json contains
> {
> "entries": [
> "schema_2.json",
> "seg_1",
> "seg_2"
> ],
> "format": "2",
> "subformat": "1"
> }
>
> and there's a schema_2.json file. Also, the locks/write.lock is
> present indicating things ended badly.
>
> I'll investigate what caused the in situ index update failure later
> (probably a bad source index :P), but what interests me know is a
> recovery procedure for these kinds of failures.
>
> Is it ok to remove the seg_2 folder and lockfile and rename
> snapshot_2.json to snapshot_1.json and likewise with schema_2.json ->
> schema_1.json?
Since USAians are unreasonably sleeping at the mo', I decided to go
ahead and edit the files and see what happens:
Removed:
locks/write.lock
seg_2/
Renamed:
schema_2.json -> schema_1.json
snapshot_2.json -> snapshot_1.json
Edited:
/master_indexstore/21/796/snapshot_1.json from:
{
"entries": [
"schema_2.json",
"seg_1",
"seg_2"
],
"format": "2",
"subformat": "1"
}
to
{
"entries": [
"schema_1.json",
"seg_1"
],
"format": "2",
"subformat": "1"
}
and all seems to be ok again. At least, a search on the index returns
results, that being the only way to check if the index is sane...
(lisc - lucy-index-system-check anyone?)
btw, lucy designers, thank you for using human-readable-editable
files, it is the UNIX way.
bye-bye
Re: [lucy-user] Index state during merges
Posted by goran kent <go...@gmail.com>.
On Thu, Nov 3, 2011 at 2:02 PM, Marvin Humphrey <ma...@rectangular.com> wrote:
> Is this index on a shared volume?
Yes it is. However, I take care of concurrency/locking outside Lucy.
The issue I had with this index was not related to lucy at all, so not
to worry (it was a script bug).
I just needed to know that I could edit the files, etc, to recover,
and it seems I can.
cheers
Re: [lucy-user] Index state during merges
Posted by Marvin Humphrey <ma...@rectangular.com>.
On Wed, Nov 02, 2011 at 09:03:40PM +0200, goran kent wrote:
> I just had a look at the logs after going ahead and making some
> changes to see how things would fair: some of my large index (in
> place - ie, no move to temp, blah blah) merge sessions are croaking
> with:
>
> Lucy::Index::Indexer->new failed (Failed to read seg_2
> S_try_open_elements at
> /home/projects/lucy/lucy/perl/../core/Lucy/Index/PolyReader.c line 251
>
> If I inspect the index it looks like the seg_2 folder is empty.
>
> /.../snapshot_2.json contains
> {
> "entries": [
> "schema_2.json",
> "seg_1",
> "seg_2"
> ],
> "format": "2",
> "subformat": "1"
> }
>
> and there's a schema_2.json file. Also, the locks/write.lock is
> present indicating things ended badly.
Is this index on a shared volume?
http://incubator.apache.org/lucy/docs/perl/Lucy/Index/Indexer.html
If an index is located on a shared volume, each writer application must
identify itself by supplying an IndexManager with a unique host id to
Indexer's constructor or index corruption will occur. See
Lucy::Docs::FileLocking for a detailed discussion.
Marvin Humphrey
Re: [lucy-user] Index state during merges
Posted by goran kent <go...@gmail.com>.
On Wed, Nov 2, 2011 at 8:29 PM, Marvin Humphrey <ma...@rectangular.com> wrote:
> Hi Goran,
>
> I think the general answer to your question is one you will be pleased to
> hear:
>
> Any indexing session which is aborted prior to the completion of
> Indexer#commit leaves the index completely unchanged.
>
> On Wed, Nov 02, 2011 at 05:23:13PM +0200, goran kent wrote:
>> during a merge of indexes to a $target, and an error occurs -- for
>> whatever reason (typically because of a broken source index) -- is
>> the $target buggered or still in a safe state?
>
> What do you mean by "broken source index"? Corrupt because bad UTF-8 snuck
> in, and now it refuses to be read?
>
> Maybe we should consider scanning incoming fields for UTF-8 sanity after all.
> I don't like making everybody pay this penalty -- small though it is --
> because you'll only get bad UTF-8 if your indexing setup is broken somehow.
> On the other hand, I don't like that once a single bad UTF-8 sequence makes it
> through a commit, the index is irretrievably corrupt -- and you only discover
> that after the damage is done.
WIth precisely this in mind, my code does some gymnastics to try and
make sure bad utf8 doesn't make it in. But,... you never know when
dealing with the vagaries of the 'tubes.
>> foreach $subindex { $bigindex->add_index($subindex); }
>> $bigindex->commit
>>
>> if one of those subindexes is broken, will it break the entire
>> bigindex?
>
> If the commit succeeds, then "bigindex" will probably be busted. If the
> commit fails, then "bigindex" will be unharmed.
ok. Nothing else to do but see what happens then -- this is where a
util to rapidly scan (not correct anything) the index to verify it's
integrity would be incredibly useful.
>
>> or does Lucy work on temp files and if an error occurs the
>> bigindex is not harmed (ie, any changes are rolled back)?
>
> Staged changes aren't rolled back so much as they are never committed.
>
> These passages from Lucy::Docs::FileFormat provide useful background:
>
> http://incubator.apache.org/lucy/docs/perl/Lucy/Docs/FileFormat.html
>
> Write-once philosophy
>
> All segment directory names consist of the string "seg_" followed by a
> number in base 36: seg_1, seg_5m, seg_p9s2 and so on, with higher numbers
> indicating more recent segments. Once a segment is finished and committed,
> its name is never re-used and its files are never modified.
>
> ...
>
> snapshot_XXX.json
>
> A "snapshot" file, e.g. snapshot_m7p.json, is list of index files and
> directories. Because index files, once written, are never modified, the
> list of entries in a snapshot defines a point-in-time view of the data in
> an index.
>
> Lucy creates some temporary files during indexing, but that doesn't get to the
> heart of the matter. The main thing to understand is that Lucy creates a new
> segment during indexing, but that segment is not considered part of the index
> until a new snapshot file which references it gets moved into place.
>
> The actual commit point is an atomic action: we add a new hard link to a
> snapshot file which has been written out under a temporary file name. At that
> moment, the new view of the index goes live, and any new IndexSearcher will
> see the new segment.
>
> If the indexing session is aborted prior to the addition of that hard link,
> then the new segment data remains orphaned and unreferenced. It will sit
> there taking up space until the next time you fire up a new Indexer (or
> BackgroundMerger), at which point it will be wiped to make room for new data
> about to be written.
Thanks for that detailed response, much appreciated.
I just had a look at the logs after going ahead and making some
changes to see how things would fair: some of my large index (in
place - ie, no move to temp, blah blah) merge sessions are croaking
with:
Lucy::Index::Indexer->new failed (Failed to read seg_2
S_try_open_elements at
/home/projects/lucy/lucy/perl/../core/Lucy/Index/PolyReader.c line 251
If I inspect the index it looks like the seg_2 folder is empty.
/.../snapshot_2.json contains
{
"entries": [
"schema_2.json",
"seg_1",
"seg_2"
],
"format": "2",
"subformat": "1"
}
and there's a schema_2.json file. Also, the locks/write.lock is
present indicating things ended badly.
I'll investigate what caused the in situ index update failure later
(probably a bad source index :P), but what interests me know is a
recovery procedure for these kinds of failures.
Is it ok to remove the seg_2 folder and lockfile and rename
snapshot_2.json to snapshot_1.json and likewise with schema_2.json ->
schema_1.json?
Thanks
Re: [lucy-user] Index state during merges
Posted by Marvin Humphrey <ma...@rectangular.com>.
Hi Goran,
I think the general answer to your question is one you will be pleased to
hear:
Any indexing session which is aborted prior to the completion of
Indexer#commit leaves the index completely unchanged.
On Wed, Nov 02, 2011 at 05:23:13PM +0200, goran kent wrote:
> during a merge of indexes to a $target, and an error occurs -- for
> whatever reason (typically because of a broken source index) -- is
> the $target buggered or still in a safe state?
What do you mean by "broken source index"? Corrupt because bad UTF-8 snuck
in, and now it refuses to be read?
Maybe we should consider scanning incoming fields for UTF-8 sanity after all.
I don't like making everybody pay this penalty -- small though it is --
because you'll only get bad UTF-8 if your indexing setup is broken somehow.
On the other hand, I don't like that once a single bad UTF-8 sequence makes it
through a commit, the index is irretrievably corrupt -- and you only discover
that after the damage is done.
> foreach $subindex { $bigindex->add_index($subindex); }
> $bigindex->commit
>
> if one of those subindexes is broken, will it break the entire
> bigindex?
If the commit succeeds, then "bigindex" will probably be busted. If the
commit fails, then "bigindex" will be unharmed.
> or does Lucy work on temp files and if an error occurs the
> bigindex is not harmed (ie, any changes are rolled back)?
Staged changes aren't rolled back so much as they are never committed.
These passages from Lucy::Docs::FileFormat provide useful background:
http://incubator.apache.org/lucy/docs/perl/Lucy/Docs/FileFormat.html
Write-once philosophy
All segment directory names consist of the string "seg_" followed by a
number in base 36: seg_1, seg_5m, seg_p9s2 and so on, with higher numbers
indicating more recent segments. Once a segment is finished and committed,
its name is never re-used and its files are never modified.
...
snapshot_XXX.json
A "snapshot" file, e.g. snapshot_m7p.json, is list of index files and
directories. Because index files, once written, are never modified, the
list of entries in a snapshot defines a point-in-time view of the data in
an index.
Lucy creates some temporary files during indexing, but that doesn't get to the
heart of the matter. The main thing to understand is that Lucy creates a new
segment during indexing, but that segment is not considered part of the index
until a new snapshot file which references it gets moved into place.
The actual commit point is an atomic action: we add a new hard link to a
snapshot file which has been written out under a temporary file name. At that
moment, the new view of the index goes live, and any new IndexSearcher will
see the new segment.
If the indexing session is aborted prior to the addition of that hard link,
then the new segment data remains orphaned and unreferenced. It will sit
there taking up space until the next time you fire up a new Indexer (or
BackgroundMerger), at which point it will be wiped to make room for new data
about to be written.
Marvin Humphrey