You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Nadav Har'El <ny...@math.technion.ac.il> on 2007/02/15 16:01:43 UTC

Re: NO_NORMS and TOKENIZED?

On Fri, Jan 26, 2007, Otis Gospodnetic wrote about "Re: NO_NORMS and TOKENIZED?":
> Funny, I was looking to do the same thing the other day and gave up thinking it wasn't possible, not being aware of setOmitNorms().  Yeah, a javadoc patch would be welcome.
> 
> Otis

Before I go ahead and post a javadoc patch, I want to question again the
wisdom of this whole situation:

Currently, most of a Field's parameters must be defined during its
construction. There is no method to change whether this Field object is to
be stored, to be compressed, to be indexed, to be tokenized - all these
things MUST be defined during the field's construction. So it is very
strange, and completely unexpected (at least to me and to Otis), that just
the "omitNorms" parameter can be changed after after construction, with a
"setOmitNorms" method - and not only can it be set after construction, in
some cases it must be set after construction, because the constructor doesn't
allow you to set it if you want an analyzer...

So perhaps changing the code, not just the javadoc, would be better?
One way to do it while keeping backward compatibility is to add something
like TOKENIZED_NO_NORMS to Field.Index.

> >...
> > I hadn't added a Field.Index option at all, and Doug suggested
> > NO_NORMS, probably because it's mostly harmless to new users who might
> > disable length normalization without realizing the implications.

If we had also a "TOKENIZED_NO_NORMS", why would new users accidentally
use it? I guess the javadoc of this parameter could also warn against its
use (something like "not recommended for general use", or whatever)?

-- 
Nadav Har'El                        |    Thursday, Feb 15 2007, 27 Shevat 5767
IBM Haifa Research Lab              |-----------------------------------------
                                    |How long a minute depends on what side of
http://nadav.harel.org.il           |the bathroom door you're on.

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Re: NO_NORMS and TOKENIZED?

Posted by Chris Hostetter <ho...@fucit.org>.
: Sorry for the rant... I guess my short answer is that I don't have an
: opinion on adding another type-safe constant TOKENIZED_NO_NORMS
: because I don't like the whole scheme.

agreed ... if people feel there is enough of a disconnect between the
options available as constructor args and the options available via
setters i would much rather see more setters then more constructor
args/constants.

i think having constructor arg constants for the "common" cases makes a
lot of sense ... but at this point i think we have way more then just the
common cases.

-Hoss


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Re: NO_NORMS and TOKENIZED?

Posted by Marvin Humphrey <ma...@rectangular.com>.
On Feb 16, 2007, at 5:42 PM, Erik Hatcher wrote:

> I think we could simplify (wow, even at this early stage) the solrb  
> code a bit by simply representing a document as a Hash.

FWIW, "documents as hashes" was something that Dave Balmain and I  
discussed at length on the Ferret list and found ourselves in  
complete agreement about.

> Marvin makes some other great points about fixed schemas, which  
> maps to the schema.xml facility of Solr I believe.

You are kind.  The funny thing is, I know very little about Solr.   
(I'm an accidental legacy subscriber to this list.)  I didn't even  
realize it had a "schema" package until I was preparing to post and  
went researching.

The KinoSearch::Schema class was actually inspired by the raft of  
object-relational mappers on CPAN: Class::DBI (by Tony Bowden, a  
primary developer of Plucene), DBIx::Class, etc.  I thought of it as  
and ORM without the underlying SQL table definition.

If there is convergence between KS and Solr on this issue, though, I  
wouldn't be surprised.

Marvin Humphrey
Rectangular Research
http://www.rectangular.com/



RE: NO_NORMS and TOKENIZED?

Posted by Steven Parkes <st...@esseff.org>.
I guess I'm in the minority here. I tend to use custom classes because I
can tweak the API to make it easier for people writing to the API. I'll
duck type all over the place to make things act like what someone might
expect, but I don't like forcing people to create raw data structures
when there are semantics that can make things easier.

In this case at hand, I was going to say that making a doc a hash makes
adding fields sequentially difficult. But ruby's nice enough that it
simply makes it a little ugly:
	doc[:contents].to_a.push data

I like being able to write
	doc[:contents] = data
where I know there's no other data for that field but I like being able
to write
      doc << { :content => data }
A doc can certainly act like a hash in some cases, but it's more than a
hash.

One of the first tests I wrote (a long time ago, with less ruby under my
belt) for the jruby interface for adding docs was:

  def test_add_lists

    @index << [ :contents, "the quick brown fox jumped over the lazy
dog" ] \
           << [ :contents, "Alas poor Yorick,", \
                :contents, "I knew him Horatio" ] \
           << [ :contents, [ "To be,", "or not ", "to be" ] ]

  end

Was never too happy with the middle case. Maybe an extra list level?

This doesn't use an explicit doc object but it makes one internally.
That nice little cool (pronounced "scary") ability in ruby to say

class Array
  def to_lucene_doc
	...
  end
end

class Hash
  def to_lucene_doc
	...
  end
end

module Lucene
  class Document
    def to_lucene_doc
      self
    end
  end
  class Index
    def << *docs
       docs.map! { |doc| doc.to_lucene_doc }
       ...
    end
end

I haven't had as much time as I would like, either, to follow solrb (and
I guess I'm also in the minority thinking that solrb was a clever name).

-----Original Message-----
From: Erik Hatcher [mailto:erik@ehatchersolutions.com] 
Sent: Friday, February 16, 2007 5:43 PM
To: ruby-dev@lucene.apache.org
Subject: Fwd: NO_NORMS and TOKENIZED?

A recent e-mail from Mr. KinoSearch to java-user has a quote that I  
wanted to point out here:

Begin forwarded message:
> KS 0.20 doesn't even have Document or Field classes.  :)  They've  
> been eliminated, and native Perl hashes are now used to transport  
> document data.

I think we could simplify (wow, even at this early stage) the solrb  
code a bit by simply representing a document as a Hash.  For  
multiValued data, the values would be arrays.  Do we really need any  
other semantics at the solrb level, or does a Hash convey it all?   
Just thinking out loud here, so feel free to ignore me.

Marvin makes some other great points about fixed schemas, which maps  
to the schema.xml facility of Solr I believe.  I am interested in  
exploring how field names get mapped, along with client knowledge of  
Solr's schema.xml structure, can make an elegant API.

	Erik

p.s. I've queued up several ruby-dev e-mails to respond to over the  
next several days.  I'm happily swamped and eager to keep the  
momentum of solrb and Flare going strong.

Re: NO_NORMS and TOKENIZED?

Posted by Marvin Humphrey <ma...@rectangular.com>.
On Feb 16, 2007, at 5:57 PM, Yonik Seeley wrote:

>> Do we really need any
>> other semantics at the solrb level, or does a Hash convey it all?
>
> boosts?

    $index_writer->add_doc( \%doc, boost => $boost );

In KS, the add_doc method takes a hashref as its first argument, then  
additional labeled parameters, of which there is currently only one:  
boost.

Field boosts are set as part of the FieldSpec subclass.

    package MySchema::title
    use base qw( KinoSearch::Schema::FieldSpec );
    sub boost { 2.5 }

It's not possible to boost individual fields within documents.

(: But it is possible to boost individual Tokens, since KS has  
partially implemented the "flexible indexing" postings format. :)

Marvin Humphrey
Rectangular Research
http://www.rectangular.com/



Re: NO_NORMS and TOKENIZED?

Posted by Yonik Seeley <yo...@apache.org>.
On 2/16/07, Erik Hatcher <er...@ehatchersolutions.com> wrote:
> A recent e-mail from Mr. KinoSearch to java-user has a quote that I
> wanted to point out here:
>
> Begin forwarded message:
> > KS 0.20 doesn't even have Document or Field classes.  :)  They've
> > been eliminated, and native Perl hashes are now used to transport

> > document data.
>
> I think we could simplify (wow, even at this early stage) the solrb
> code a bit by simply representing a document as a Hash.  For
> multiValued data, the values would be arrays.

That's how it works with the simple python client... I like to use
natives when possible.

Example usage from solr.py:
from solr import *

c = SolrConnection(host='localhost:8983', persistent=True)
c.add(id='500',name='python test doc')
c.delete('123')
c.commit()
print c.search(q='id:[* TO *]', wt='python', rows='10',indent='on')

The separate params to add is just syntactic sugar for a map (I
believe ruby has the same thing).  Adding multiple documents is done
as an array-of-map.

> Do we really need any
> other semantics at the solrb level, or does a Hash convey it all?

boosts?
Perhaps you could still use a hash, but the value could optionally be
a boosted value... is it possible to annotate any value in Ruby, or is
a separate BoostedValue class needed?

Might want to keep in mind updateable documents for the future.  Not
sure how you would want them represented, but it looks like there will
be a separate param (or parameters) telling solr how to update
different fields (append, overwrite, increment, remove, etc)

-Yonik

Fwd: NO_NORMS and TOKENIZED?

Posted by Erik Hatcher <er...@ehatchersolutions.com>.
A recent e-mail from Mr. KinoSearch to java-user has a quote that I  
wanted to point out here:

Begin forwarded message:
> KS 0.20 doesn't even have Document or Field classes.  :)  They've  
> been eliminated, and native Perl hashes are now used to transport  
> document data.

I think we could simplify (wow, even at this early stage) the solrb  
code a bit by simply representing a document as a Hash.  For  
multiValued data, the values would be arrays.  Do we really need any  
other semantics at the solrb level, or does a Hash convey it all?   
Just thinking out loud here, so feel free to ignore me.

Marvin makes some other great points about fixed schemas, which maps  
to the schema.xml facility of Solr I believe.  I am interested in  
exploring how field names get mapped, along with client knowledge of  
Solr's schema.xml structure, can make an elegant API.

	Erik

p.s. I've queued up several ruby-dev e-mails to respond to over the  
next several days.  I'm happily swamped and eager to keep the  
momentum of solrb and Flare going strong.

Re: NO_NORMS and TOKENIZED?

Posted by Grant Ingersoll <gs...@apache.org>.
On Feb 21, 2007, at 2:35 AM, Chris Hostetter wrote:

>
> the other situation i brought up in that thread from way back was
> something Solr doesn't currently have a good solution for: one
> field value for display, but other values (in the same field) for
> searching ... likewise indexing one field value for sorting, but  
> storing
> several field values.
>
> the workarround for this of course is to use multiple field names:
> title, title_sort, title_search, etc...
>

I wonder if payloads could address this?  You could store your  
display info as a payload for the thing you are searching, etc.


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Re: NO_NORMS and TOKENIZED?

Posted by Marvin Humphrey <ma...@rectangular.com>.
On Feb 20, 2007, at 11:35 PM, Chris Hostetter wrote:
> the biggest differnece is that the field infos aren't globals, so as
> segments merge and old segments get deleted old data (and field info)
> vanishes into hte ether ... i take advantage of that a lot when  
> planning
> upgrades ... many types of field changes can be made in Solr by  
> deploying
> the schema.xml reindexing slowly over as long as it takes (which  
> the index
> is live and serving queries) then deploying the application changes  
> that
> take advantage of the new field options.

Well, there's no fundamental technical limitation that makes it so  
that global field defs can't be purged -- especially after  
optimization.  I just don't have a system in place that makes it  
possible.  There'd probably be a few bugs if you tried shrinking a  
Schema now, but a few tests should find them and they're killable.

> : The multi-dimensional data problem is the one I'm most interested in
>
> i believe yonik already elaborated on that and gave you some good  
> ideas.
>
> the other situation i brought up in that thread from way back was
> something Solr doesn't currently have a good solution for: one
> field value for display, but other values (in the same field) for
> searching ... likewise indexing one field value for sorting, but  
> storing
> several field values.

In KS, the DocWriter class writes stored fields.  Since the field  
defs have global semantics, you don't need to know anything about the  
contents of the serialized document -- no conflict resolutions.  It's  
just a blob with a certain number of bytes.

My plan is to enable subclassing of DocWriter/DocReader.  That should  
make it possible to scratch a number of itches:

   * multiple values per field
   * different field values for display and searching
   * lazy loading
   * arbitrary data
   * arbitrary compression algo choice
   * complete document recovery
   * ...

Marvin Humphrey
Rectangular Research
http://www.rectangular.com/



---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Re: NO_NORMS and TOKENIZED?

Posted by Chris Hostetter <ho...@fucit.org>.
: > en if i wanted to be able to use
: > an option on field foo for some docs and not on others i'd have to
: > have
: > foo_optOFF and foo_optON ... then anytime i wanted to search on
: > "foo" i'd
: > have to use a booleanquery without a coord factor across both.
:
: I'm trying to think of an example where that would actually come into
: play.  What are some of the options you'd turn on and off?  Norms?
: Tokenization?

it could be anything ... maybe i only care about termvectors for
editorials in a collection that contains editorials, news items, etc...

: IIUC, that's a third objection in addition to your two from the
: previous discussion.  The other two were "evolving" indexes, and what
: you described as "dynamically typed fields" but what I would call
: "multi-dimensional data".

evolving indexes is a big issue i hadn't thought of my last message,
but definitely important...

: KinoSearch's Schema scheme allows you to add new fields, but you
: can't take any away or change any existing defs  -- so you're able to
: evolve, but only within fairly tight constraints.  I can't think of

...yeah, i would find that... frustrating.  it would require downtime when
making a lot of upgrades that i currently can do on the fly.

: aspect "good enough" and we're moving on.  I don't think it's really
: any worse than what we have now -- where field defs persist, stored
: in field infos files, etc, and the resolution of conflicts can bite

the biggest differnece is that the field infos aren't globals, so as
segments merge and old segments get deleted old data (and field info)
vanishes into hte ether ... i take advantage of that a lot when planning
upgrades ... many types of field changes can be made in Solr by deploying
the schema.xml reindexing slowly over as long as it takes (which the index
is live and serving queries) then deploying the application changes that
take advantage of the new field options.

: The multi-dimensional data problem is the one I'm most interested in

i believe yonik already elaborated on that and gave you some good ideas.

the other situation i brought up in that thread from way back was
something Solr doesn't currently have a good solution for: one
field value for display, but other values (in the same field) for
searching ... likewise indexing one field value for sorting, but storing
several field values.

the workarround for this of course is to use multiple field names:
title, title_sort, title_search, etc...





-Hoss


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Re: NO_NORMS and TOKENIZED?

Posted by Marvin Humphrey <ma...@rectangular.com>.
On Feb 19, 2007, at 8:45 AM, Yonik Seeley wrote:

> If I had to do it over again, I'd be tempted to further restrict the
> patterns so that they could be looked up from a Map rather than
> linearly.

Awesome.  I know exactly how I'm going to implement this now.

> This hasn't proved to be a problem so far though, as the
> number of field-types for dynamic fields normally remains small.

For KS, there will be only one abstract class dedicated to multi- 
dimensional data.  Users will subclass to provide their own arbitrary  
field definitions.  The field definition itself won't be dynamic --  
only the suffix on the field name will be.

For a hashmap lookup, a prefix pattern could be restricted one of two  
ways: fixed length, or terminal character.  I'm inclined to go with a  
terminating underscore in the field name -- that allows the users to  
choose their own prefix for maximum readability, at the cost of an  
additional scan.

Here's how the schema for your CNET index might look.

    # ./CNETSchema.pm

    package CNETSchema::name;
    use base 'KinoSearch::Schema::FieldSpec';

    package CNETSchema::description;
    use base 'KinoSearch::Schema::FieldSpec';
    sub similarity {
        return KinoSearch::Contrib::LongFieldSim->new;
    }

    package CNETSchema::product_id;
    use base 'KinoSearch::Schema::FieldSpec';
    sub analyzed { 0 }

    package CNETSchema::attr;
    use base 'KinoSearch::Schema::DeepFieldSpec';
    sub analyzed { 0 }
    sub stored   { 0 }

    package CNETSchema;
    use base 'KinoSearch::Schema';
    use KinoSearch::Analyzer::PolyAnalyzer;
    sub analyzer {
        return KinoSearch::Analysis::PolyAnalyzer->new( language =>  
'en' );
    }
    __PACKAGE__->load_fields(qw( name description product_id attr ));

    1;

Then, at index time, you'll be able to do this:

    $index_writer->add_doc({
        name                         => 'Acme LT-1 Laptop',
        description                  => 'blah blah blah...',
        product_id                   => 'acme-lt-1',
        attr_weight                  => 6.3,
        attr_heat_dissipation_factor => 20,
    });

I'll need to make a few backend tweaks, but this API pretty much  
solves the multi-dimensional data problem. :)

Thoughts?

Marvin Humphrey
Rectangular Research
http://www.rectangular.com/



---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Re: NO_NORMS and TOKENIZED?

Posted by Yonik Seeley <yo...@apache.org>.
On 2/19/07, Marvin Humphrey <ma...@rectangular.com> wrote:
> At present, KS doesn't let you do something like that -- you have to
> define all your fields up front.  What I'd like to do is come up with
> a FieldDef subclass that handles multi-dimensional data.  I seem to
> recall that Solr had something along those lines, using prefixed
> field names or something.  Do I recall correctly?

Yes, Solr does a simple prefix or suffix match.
Solr looks up field type for a field in a Map<String,SchemaField>, and
if it doesn't find it, it checks it's "dynamic fields".  If a dynamic
field is found, the SchemaField for it (a template) is quickly cloned,
with only the name being appropriately changed.

Normal field definition (the type attribute refers to another part of
schema.xml):
   <field name="text" type="text" indexed="true" stored="false"
multiValued="true"/>

Dynamic field definitions:
   <!-- Dynamic field definitions.  If a field name is not found, dynamicFields
        will be used if the name matches any of the patterns.
        RESTRICTION: the glob-like pattern in the name attribute must have
        a "*" only at the start or the end.
        EXAMPLE:  name="*_i" will match any field ending in _i (like
myid_i, z_i)
        Longer patterns will be matched first.  if equal size patterns
        both match, the first appearing in the schema will be used.  -->
   <dynamicField name="*_i"  type="sint"    indexed="true"  stored="true"/>
   <dynamicField name="*_s"  type="string"  indexed="true"  stored="true"/>
   <dynamicField name="*_l"  type="slong"   indexed="true"  stored="true"/>
   <dynamicField name="*_t"  type="text"    indexed="true"  stored="true"/>
   <dynamicField name="*_b"  type="boolean" indexed="true"  stored="true"/>
   <dynamicField name="*_f"  type="sfloat"  indexed="true"  stored="true"/>
   <dynamicField name="*_d"  type="sdouble" indexed="true"  stored="true"/>
   <dynamicField name="*_dt" type="date"    indexed="true"  stored="true"/>

If I had to do it over again, I'd be tempted to further restrict the
patterns so that they could be looked up from a Map rather than
linearly.  This hasn't proved to be a problem so far though, as the
number of field-types for dynamic fields normally remains small.

-Yonik

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Re: NO_NORMS and TOKENIZED?

Posted by Grant Ingersoll <gs...@apache.org>.
On Feb 19, 2007, at 3:07 PM, Marvin Humphrey wrote:

>
> On Feb 19, 2007, at 11:32 AM, Grant Ingersoll wrote:
>
>> FWIW, we support, in our in-house system and in addition to fixed  
>> field semantics,  completely dynamic field names for some  
>> applications, but they have a fixed field type.  So, the field  
>> name can be anything, but the attributes of the field are fixed  
>> (i.e. it will always be tokenized with norms). This is useful for  
>> us, in some cases, when indexing XML files where the tag name  
>> becomes the field name and the set of tag names are not known  
>> ahead of time.  I suppose there are ways around this (by  
>> preprocessing all the files), but having the ability to add  
>> arbitrary fields is a good thing for us and some of the  
>> applications we do.
>
> The thing I don't like about this is that it prevents validation of  
> field names, which is something I use a lot  in KS (e.g. try to  
> delete a term from a field that's not indexed, get an error, as the  
> field name was probably misspelled).  I can see the use, it just  
> means sacrificing a lot of type safety for the more common cases.   
> The user base at large has to suffer with more frequent, hard-to- 
> detect bugs for a feature only needed by a few users.
>

Since all our dynamically named fields are of the same type, it isn't  
an issue for us at the moment.  Then again, though, we only have in- 
house users and don't have the same issue that you have.


> About your app in particular -- how do you handle identical XML tag  
> names that mean totally different things when nested inside  
> different elements?


It doesn't happen.  The tags are based on the output of some other  
processes and are unique and the tag/field name has semantics  
attached to it that is meaningful to the application.  I suppose,  
technically, they are known ahead of time, but there are potentially  
hundreds of them such that it doesn't make sense to populate them  
into our Field schema ahead of time as maintenance would be a nightmare.


>    <company>
>      <name>Acme</name>
>    </company>
>    <product>
>      <name>Widget</name>
>    </product>
>
> Marvin Humphrey
> Rectangular Research
> http://www.rectangular.com/
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>

--------------------------
Grant Ingersoll
Center for Natural Language Processing
http://www.cnlp.org

Read the Lucene Java FAQ at http://wiki.apache.org/jakarta-lucene/ 
LuceneFAQ



---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Re: NO_NORMS and TOKENIZED?

Posted by Marvin Humphrey <ma...@rectangular.com>.
On Feb 19, 2007, at 11:32 AM, Grant Ingersoll wrote:

> FWIW, we support, in our in-house system and in addition to fixed  
> field semantics,  completely dynamic field names for some  
> applications, but they have a fixed field type.  So, the field name  
> can be anything, but the attributes of the field are fixed (i.e. it  
> will always be tokenized with norms). This is useful for us, in  
> some cases, when indexing XML files where the tag name becomes the  
> field name and the set of tag names are not known ahead of time.  I  
> suppose there are ways around this (by preprocessing all the  
> files), but having the ability to add arbitrary fields is a good  
> thing for us and some of the applications we do.

The thing I don't like about this is that it prevents validation of  
field names, which is something I use a lot  in KS (e.g. try to  
delete a term from a field that's not indexed, get an error, as the  
field name was probably misspelled).  I can see the use, it just  
means sacrificing a lot of type safety for the more common cases.   
The user base at large has to suffer with more frequent, hard-to- 
detect bugs for a feature only needed by a few users.

About your app in particular -- how do you handle identical XML tag  
names that mean totally different things when nested inside different  
elements?

    <company>
      <name>Acme</name>
    </company>
    <product>
      <name>Widget</name>
    </product>

Marvin Humphrey
Rectangular Research
http://www.rectangular.com/



---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Re: NO_NORMS and TOKENIZED?

Posted by Grant Ingersoll <gr...@gmail.com>.
FWIW, we support, in our in-house system and in addition to fixed  
field semantics,  completely dynamic field names for some  
applications, but they have a fixed field type.  So, the field name  
can be anything, but the attributes of the field are fixed (i.e. it  
will always be tokenized with norms). This is useful for us, in some  
cases, when indexing XML files where the tag name becomes the field  
name and the set of tag names are not known ahead of time.  I suppose  
there are ways around this (by preprocessing all the files), but  
having the ability to add arbitrary fields is a good thing for us and  
some of the applications we do.

-Grant

On Feb 19, 2007, at 2:14 AM, Marvin Humphrey wrote:

>
> On Feb 18, 2007, at 10:33 PM, Chris Hostetter wrote:
>
>> I don't suppose you have a mailing pointer to my old comments do you
>> Marvin?
>
> http://tinyurl.com/394apl (mail-archives.apache.org)
>
> You're in good company.  The other party with strong objections was  
> Doug.
>
> http://tinyurl.com/36ucj2
>
>> en if i wanted to be able to use
>> an option on field foo for some docs and not on others i'd have to  
>> have
>> foo_optOFF and foo_optON ... then anytime i wanted to search on  
>> "foo" i'd
>> have to use a booleanquery without a coord factor across both.
>
> I'm trying to think of an example where that would actually come  
> into play.  What are some of the options you'd turn on and off?   
> Norms?  Tokenization?
>
> IIUC, that's a third objection in addition to your two from the  
> previous discussion.  The other two were "evolving" indexes, and  
> what you described as "dynamically typed fields" but what I would  
> call "multi-dimensional data".
>
> KinoSearch's Schema scheme allows you to add new fields, but you  
> can't take any away or change any existing defs  -- so you're able  
> to evolve, but only within fairly tight constraints.  I can't think  
> of an elegant way to improve that situation, so I've declared that  
> aspect "good enough" and we're moving on.  I don't think it's  
> really any worse than what we have now -- where field defs persist,  
> stored in field infos files, etc, and the resolution of conflicts  
> can bite you in the butt (as I recall hearing you discuss when no- 
> norms was being hashed out wrt suddenly having lots and lots of  
> memory-sucking norms).
>
> The multi-dimensional data problem is the one I'm most interested  
> in solving.  Lucene/KS indexes don't handle one-to-many  
> relationships well.  In your example, you had an index where the  
> products had "attributes", and the attributes might have taken many  
> different names -- so it wasn't possible to know all the field  
> names in advance.  It sounds like, effectively, you were faking a  
> second table.  So long as the names of your attributes don't crash  
> into the names of your primary fields, that'll work.
>
> At present, KS doesn't let you do something like that -- you have  
> to define all your fields up front.  What I'd like to do is come up  
> with a FieldDef subclass that handles multi-dimensional data.  I  
> seem to recall that Solr had something along those lines, using  
> prefixed field names or something.  Do I recall correctly?
>
> Marvin Humphrey
> Rectangular Research
> http://www.rectangular.com/
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>

------------------------------------------------------
Grant Ingersoll
http://www.grantingersoll.com/
http://lucene.grantingersoll.com
http://www.paperoftheweek.com/



---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Re: NO_NORMS and TOKENIZED?

Posted by Marvin Humphrey <ma...@rectangular.com>.
On Feb 18, 2007, at 10:33 PM, Chris Hostetter wrote:

> I don't suppose you have a mailing pointer to my old comments do you
> Marvin?

http://tinyurl.com/394apl (mail-archives.apache.org)

You're in good company.  The other party with strong objections was  
Doug.

http://tinyurl.com/36ucj2

> en if i wanted to be able to use
> an option on field foo for some docs and not on others i'd have to  
> have
> foo_optOFF and foo_optON ... then anytime i wanted to search on  
> "foo" i'd
> have to use a booleanquery without a coord factor across both.

I'm trying to think of an example where that would actually come into  
play.  What are some of the options you'd turn on and off?  Norms?   
Tokenization?

IIUC, that's a third objection in addition to your two from the  
previous discussion.  The other two were "evolving" indexes, and what  
you described as "dynamically typed fields" but what I would call  
"multi-dimensional data".

KinoSearch's Schema scheme allows you to add new fields, but you  
can't take any away or change any existing defs  -- so you're able to  
evolve, but only within fairly tight constraints.  I can't think of  
an elegant way to improve that situation, so I've declared that  
aspect "good enough" and we're moving on.  I don't think it's really  
any worse than what we have now -- where field defs persist, stored  
in field infos files, etc, and the resolution of conflicts can bite  
you in the butt (as I recall hearing you discuss when no-norms was  
being hashed out wrt suddenly having lots and lots of memory-sucking  
norms).

The multi-dimensional data problem is the one I'm most interested in  
solving.  Lucene/KS indexes don't handle one-to-many relationships  
well.  In your example, you had an index where the products had  
"attributes", and the attributes might have taken many different  
names -- so it wasn't possible to know all the field names in  
advance.  It sounds like, effectively, you were faking a second  
table.  So long as the names of your attributes don't crash into the  
names of your primary fields, that'll work.

At present, KS doesn't let you do something like that -- you have to  
define all your fields up front.  What I'd like to do is come up with  
a FieldDef subclass that handles multi-dimensional data.  I seem to  
recall that Solr had something along those lines, using prefixed  
field names or something.  Do I recall correctly?

Marvin Humphrey
Rectangular Research
http://www.rectangular.com/



---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Re: NO_NORMS and TOKENIZED?

Posted by Chris Hostetter <ho...@fucit.org>.
: Heh.  Your compadre Hoss is one of the most ardent opponents of fixed
: field defs.  :)
:
: I've been planning to ask him at some point how he might deal with
: multidimensional data if he were forced to use KinoSearch's Schema/
: FieldSpec system.  What say you, Hoss, can you think of something?

am i really that bad? ... i remember having a discussion about this (which
i can't find now) and i remember thinking up some examples of why i
thought it was useful to be able to set field properties on a per document
basis -- but i can't for the life of me rememebr what they were now ...
I don't suppose you have a mailing pointer to my old comments do you
Marvin?

assuming for hte sake of argument i had good reasons (i think that's a
safe bet: i always have good reasons) then if i wanted to be able to use
an option on field foo for some docs and not on others i'd have to have
foo_optOFF and foo_optON ... then anytime i wanted to search on "foo" i'd
have to use a booleanquery without a coord factor across both.




-Hoss


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Re: NO_NORMS and TOKENIZED?

Posted by Mike Klaas <mi...@gmail.com>.
On 2/16/07, Marvin Humphrey <ma...@rectangular.com> wrote:

> Solved... with fixed field definitions.

<>

> Imagine a world with no search-time/index-time analyzer mismatches...

I'm sure Yonik can imagine such a world... that's what Solr provide
<g>.  Configure an analyzer for a field (or even separate index-time
and query-time analyzers) once, and never worry about it again.

Note that you don't need a fixed _number_ of fields to achieve these
benefits, just a fixed definition per unique field name.

-Mike

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Re: NO_NORMS and TOKENIZED?

Posted by Marvin Humphrey <ma...@rectangular.com>.
On Feb 15, 2007, at 1:20 PM, Yonik Seeley wrote:

> I originally added it without an Index param at all.
> I can't say I'm a fan of the way Field currently does things, and I
> didn't want everyone to pay the price for yet more options.

Solved... with fixed field definitions.

http://www.rectangular.com/kinosearch/docs/devel/KinoSearch/Schema.html
http://www.rectangular.com/kinosearch/docs/devel/KinoSearch/Schema/ 
FieldSpec.html
http://www.rectangular.com/kinosearch/docs/devel/KinoSearch/ 
InvIndex.html

> I'm probably in the minority, hence I never said
> anything about it before.

Heh.  Your compadre Hoss is one of the most ardent opponents of fixed  
field defs.  :)

I've been planning to ask him at some point how he might deal with  
multidimensional data if he were forced to use KinoSearch's Schema/ 
FieldSpec system.  What say you, Hoss, can you think of something?

For KS, a solution to that problem would be nice to have, but not  
essential.  KS has never allowed dynamic field defs, so there's no  
constituency wedded to them.  And fixed field defs solve so many more  
problems than they cause.

Imagine a world with no search-time/index-time analyzer mismatches...

> How would I have handled it?
>  With a single Field class, I would probably have used old-fashion
> c-style flags (a bit field).  Nice an extensible (you can add new
> flags without adding/changing any APIs, no performance impact to
> adding new options, you can pass around all these flags as a unit,
> check multiple flags with a single instruction, etc.

I don't think you can convey enough information using only flags.

Looking ahead, it's easy to specify a "flexible indexing" posting  
format by adding that property to a FieldSpec via a method or an  
inner class, but hard to see how you'd put that amount of info in a  
bit -- let alone resolve conflicts.

People want to attach greater and greater semantic meaning to field  
names.  The present system doesn't scale well, as you've pointed out;  
there will be ever-increasing tension until it is modified.

> I guess my short answer is that I don't have an
> opinion on adding another type-safe constant TOKENIZED_NO_NORMS
> because I don't like the whole scheme.

KS 0.20 doesn't even have Document or Field classes.  :)  They've  
been eliminated, and native Perl hashes are now used to transport  
document data.

Marvin Humphrey
Rectangular Research
http://www.rectangular.com/



---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Re: NO_NORMS and TOKENIZED?

Posted by Yonik Seeley <yo...@apache.org>.
I originally added it without an Index param at all.
I can't say I'm a fan of the way Field currently does things, and I
didn't want everyone to pay the price for yet more options.

Look at the code for the Field constructor:

  public Field(String name, String value, Store store, Index index,
TermVector termVector) {
    if (name == null)
      throw new NullPointerException("name cannot be null");
    if (value == null)
      throw new NullPointerException("value cannot be null");
    if (name.length() == 0 && value.length() == 0)
      throw new IllegalArgumentException("name and value cannot both be empty");
    if (index == Index.NO && store == Store.NO)
      throw new IllegalArgumentException("it doesn't make sense to
have a field that "
         + "is neither indexed nor stored");
    if (index == Index.NO && termVector != TermVector.NO)
      throw new IllegalArgumentException("cannot store term vector information "
         + "for a field that is not indexed");

    this.name = name.intern();        // field names are interned
    this.fieldsData = value;

    if (store == Store.YES){
      this.isStored = true;
      this.isCompressed = false;
    }
    else if (store == Store.COMPRESS) {
      this.isStored = true;
      this.isCompressed = true;
    }
    else if (store == Store.NO){
      this.isStored = false;
      this.isCompressed = false;
    }
    else
      throw new IllegalArgumentException("unknown store parameter " + store);

    if (index == Index.NO) {
      this.isIndexed = false;
      this.isTokenized = false;
    } else if (index == Index.TOKENIZED) {
      this.isIndexed = true;
      this.isTokenized = true;
    } else if (index == Index.UN_TOKENIZED) {
      this.isIndexed = true;
      this.isTokenized = false;
    } else if (index == Index.NO_NORMS) {
      this.isIndexed = true;
      this.isTokenized = false;
      this.omitNorms = true;
    } else {
      throw new IllegalArgumentException("unknown index parameter " + index);
    }

    this.isBinary = false;

    setStoreTermVector(termVector);
  }

 protected void setStoreTermVector(Field.TermVector termVector) {
    if (termVector == Field.TermVector.NO) {
      this.storeTermVector = false;
      this.storePositionWithTermVector = false;
      this.storeOffsetWithTermVector = false;
    }
    else if (termVector == Field.TermVector.YES) {
      this.storeTermVector = true;
      this.storePositionWithTermVector = false;
      this.storeOffsetWithTermVector = false;
    }
    else if (termVector == Field.TermVector.WITH_POSITIONS) {
      this.storeTermVector = true;
      this.storePositionWithTermVector = true;
      this.storeOffsetWithTermVector = false;
    }
    else if (termVector == Field.TermVector.WITH_OFFSETS) {
      this.storeTermVector = true;
      this.storePositionWithTermVector = false;
      this.storeOffsetWithTermVector = true;
    }
    else if (termVector == Field.TermVector.WITH_POSITIONS_OFFSETS) {
      this.storeTermVector = true;
      this.storePositionWithTermVector = true;
      this.storeOffsetWithTermVector = true;
    }
    else {
      throw new IllegalArgumentException("unknown termVector parameter
" + termVector);
    }
  }


I simply think this is too high of a price to pay for "type safety".
Everyone shouldn't have to pay a performance penalty for making things
a little "safer".  I'm probably in the minority, hence I never said
anything about it before.

It's also made Solr code worse because it stores these things as
flags, but has to go through the same if-then-else contortions to
construct a Field with the proper parameters (just to have Field go
through the reverse contortions to de-multiplex these options).

Think about what would happen if we added a few more options on
storing term vectors... exponential explosion in those if-then-else
statements.

How would I have handled it?
  With a single Field class, I would probably have used old-fashion
c-style flags (a bit field).  Nice an extensible (you can add new
flags without adding/changing any APIs, no performance impact to
adding new options, you can pass around all these flags as a unit,
check multiple flags with a single instruction, etc.
  If we look at inheritance, I'd be tempted to let people subclass and
allow them to pass data directly to the indexer rather than trying to
store it all and enumerate all the possibilities in the single Field
class.
   Reader getBinaryValue()  or even int writeBinaryValue(Writer or IndexOutput)
   TokenStream getTokens()
   boolean isIndexed()

Sorry for the rant... I guess my short answer is that I don't have an
opinion on adding another type-safe constant TOKENIZED_NO_NORMS
because I don't like the whole scheme.

-Yonik


On 2/15/07, Nadav Har'El <ny...@math.technion.ac.il> wrote:
> On Fri, Jan 26, 2007, Otis Gospodnetic wrote about "Re: NO_NORMS and TOKENIZED?":
> > Funny, I was looking to do the same thing the other day and gave up thinking it wasn't possible, not being aware of setOmitNorms().  Yeah, a javadoc patch would be welcome.
> >
> > Otis
>
> Before I go ahead and post a javadoc patch, I want to question again the
> wisdom of this whole situation:
>
> Currently, most of a Field's parameters must be defined during its
> construction. There is no method to change whether this Field object is to
> be stored, to be compressed, to be indexed, to be tokenized - all these
> things MUST be defined during the field's construction. So it is very
> strange, and completely unexpected (at least to me and to Otis), that just
> the "omitNorms" parameter can be changed after after construction, with a
> "setOmitNorms" method - and not only can it be set after construction, in
> some cases it must be set after construction, because the constructor doesn't
> allow you to set it if you want an analyzer...
>
> So perhaps changing the code, not just the javadoc, would be better?
> One way to do it while keeping backward compatibility is to add something
> like TOKENIZED_NO_NORMS to Field.Index.
>
> > >...
> > > I hadn't added a Field.Index option at all, and Doug suggested
> > > NO_NORMS, probably because it's mostly harmless to new users who might
> > > disable length normalization without realizing the implications.
>
> If we had also a "TOKENIZED_NO_NORMS", why would new users accidentally
> use it? I guess the javadoc of this parameter could also warn against its
> use (something like "not recommended for general use", or whatever)?
>
> --
> Nadav Har'El                        |    Thursday, Feb 15 2007, 27 Shevat 5767
> IBM Haifa Research Lab              |-----------------------------------------
>                                     |How long a minute depends on what side of
> http://nadav.harel.org.il           |the bathroom door you're on.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org