You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@lucenenet.apache.org by Matt Honeycutt <mb...@gmail.com> on 2009/11/12 01:57:10 UTC
FieldLookup for field with multiple values
It seems that the StringIndex returned by
FieldCache.Fields.Default.GetStringIndex() only indexes one value for a
document even when the document has multiple values for the field. Is there
a performant want to get all the values for a particular field in a
ScoreDoc? I'm having to do this across the entire result set of ScoreDocs
(up to 50,000), and retrieving the values through LuceneDocument.GetFields
is not going to cut it.
RE: FieldLookup for field with multiple values
Posted by Michael Garski <mg...@myspace-inc.com>.
It scales well with tens of millions of documents or more with enough
RAM, provided you have a mechanism for expiring the cache when a reader
is no longer in use and pool cached items for re-use so your process
doesn't incur massive GC sweeps when the cache is expired.
Michael
-----Original Message-----
From: Matt Honeycutt [mailto:mbhoneycutt@gmail.com]
Sent: Thursday, November 12, 2009 12:35 PM
To: lucene-net-user@incubator.apache.org
Subject: Re: FieldLookup for field with multiple values
I shudder to think about what the higher-ups will say about the cost of
faster storage. They're very stingy on things that make our lives
easier.
:)
I like the idea of caching the term vectors somehow, though I wonder how
that would scale with millions of documents. I will add that to the
list of
things to prototype for our long-term solution.
Thanks for the feedback,
---Matt
On Thu, Nov 12, 2009 at 12:14 PM, Michael Garski
<mg...@myspace-inc.com>wrote:
> Matt,
>
> Metadata can be collected during the course of a search in a
> HitCollector (note that 2.9 will deprecate the current HitCollector in
> favor of the more flexible Collector). One approach to collecting
> facets is to index the field with the desired facet metadata with
> TermVectors enabled. During collection you can retrieve the
> TermFreqVector for a document with Reader.GetTermFreqVector for each
> document and collect/aggregate metadata however you like.
>
> Making the call to get the TermFreqVector for each hit will be a
> performance hit due to IO and deserialization (even though it's very
> fast, 50K times in one search is a lot). At that point you can add
> faster IO (Fusion IO rules this space) or during an index warmup
create
> an in-memory structure with lookups to the vectors keyed off the
> document id.
>
> Hope that helps.
>
> Michael
>
> -----Original Message-----
> From: Matt Honeycutt [mailto:mbhoneycutt@gmail.com]
> Sent: Thursday, November 12, 2009 7:13 AM
> To: lucene-net-user@incubator.apache.org
> Subject: Re: FieldLookup for field with multiple values
>
> I can elaborate a little on what our *planned* approach for utilizing
> SQL
> Server is. I don't know if this will work, but I've done similar
things
> with SQL CLR and haven't had it explode (yet), so I'm hopeful.
Anyway:
>
> Our system needs two types of output, basically: the full-text reports
> (I'll
> call them documents to reduce ambiguity), and then statistical reports
> built
> on those documents. The documents can easily be retrieved from
Lucene,
> so
> the challenge is building the reports. For that, my plan is to submit
> the
> same query string to SQL Server that I sent to Lucene. Internally,
SQL
> Server would then pass the query back to Lucene and retrieve a list of
> document IDs that matched. The communication may be over WCF or
> something
> similar, and will be compressed during transit to reduce IO overhead.
> Once
> SQL Server has the IDs, they will be loaded into a temporary table
(with
> indexes) or a table variable, which will then be used to filter the
> metadata
> that the statistical reports are built from.
>
> I have no idea how such a system would perform. I do hope to do some
> feasibility tests sometime Real Soon (like in the next few weeks), and
> I'll
> post my results if I manage to get it working.
>
> If anyone has any other suggestions, please do share.
>
> On Thu, Nov 12, 2009 at 8:43 AM, Moray McConnachie <
> mmcconna@oxford-analytica.com> wrote:
>
> > >While we're discussing this, anyone have any advice or suggestions
> for
> > a better solution? We've considered a few things for our long-term
> > solution.
> >
> > I'd be very interested to hear thoughts on intersecting SQL and
Lucene
> > too, as in our case we have very large lists of organisations which
> have
> > different permissions (stored in SQL) for different documents stored
> in
> > Lucene. Showing in the search results only those documents to which
> the
> > organisation has permission is quite expensive for queries with lots
> of
> > results. Storing it in the documents is not manageable because they
> need
> > to be updated frequently across multiple documents. Currently we
> > precompute a list for each organisation, cache that in memory, and
> > recache it every time that organisation is updated. However this too
> is
> > costly.
> >
> > Storing the Lucene document nos in SQL during indexing, and then
> passing
> > the list of Lucene document nos matching a search to SQL for
filtering
> > seems the right way to go. But Matt is right, the problem is with
> > scaling this to searches returning many thousands of documents.
> >
> > Yours,
> > M.
> > -------------------------------------
> > Moray McConnachie
> > Director of IT +44 1865 261 600
> > Oxford Analytica http://www.oxan.com
> >
> > -----Original Message-----
> > From: Matt Honeycutt [mailto:mbhoneycutt@gmail.com]
> > Sent: 12 November 2009 13:53
> > To: lucene-net-user@incubator.apache.org
> > Subject: Re: FieldLookup for field with multiple values
> >
> > Yeah, it is sort of like your standard faceting scenario, except
there
> > are about 20,000 facets (organizations), and there's complex
> > relationships among the facets.
> >
> > The reports we're dealing with only occasionally break the funding
up
> by
> > organization, so we decided (for now) to just store a single funding
> > value, then break it up after-the-fact by dividing it by the number
of
> > organizations. So no, the funding is only stored once.
> >
> > While we're discussing this, anyone have any advice or suggestions
for
> a
> > better solution? We've considered a few things for our long-term
> > solution.
> > One is to put this metadata in a SQL Server instance, and use SQL
CLR
> to
> > build a temporary table based on document IDs from a Lucene index
> > (hosted over WCF or something similar), then do the reporting within
> SQL
> > Server. We plan to compress the list of IDs going back from Lucene
to
> > SQL Server to cut down on IO overhead, but we're still concerned
that
> > approach won't scale as we go from hundreds of thousands to millions
> of
> > reports.
> >
> > Another option we've discussed is to precompute data cubes and use
> these
> > to calculate reporting information. The concern here is the high
> > dimensionality of the data (we have about 20,000 distinct
> organizations
> > now, but fully expect that to increase by an order of magnitude) as
> well
> > as the accuracy of the generated reports, since there's (probably)
not
> a
> > good way to divide the cube based on arbitrary Lucene queries.
> >
> > On Thu, Nov 12, 2009 at 1:03 AM, Michael Garski
> > <mg...@myspace-inc.com>wrote:
> >
> > > Sounds like a full-text search with the results simply being
facets
> on
> >
> > > the organizations sorted by the funding amount?
> > >
> > > You mentioned adding the org ID once for each document. Do you do
> the
> >
> > > same for the funding, with the funding for each corresponding
> > organization?
> > >
> > > Michael
> > >
> > >
> > > -----Original Message-----
> > > From: Matt Honeycutt [mailto:mbhoneycutt@gmail.com]
> > > Sent: Wed 11/11/2009 10:17 PM
> > > To: lucene-net-user@incubator.apache.org
> > > Subject: Re: FieldLookup for field with multiple values
> > >
> > > Well, let me prefix what I'm about to describe by saying that I
know
> > > that I'm doing something with Lucene that it wasn't meant to do.
> This
> >
> > > is for a "proof of concept" system that I'm helping put together
on
> a
> > > tight schedule with very limited resources, and we're trying to
get
> to
> >
> > > a mostly-working state as quickly as possible.
> > >
> > > That said, we are basically storing reports in Lucene. The
reports
> > > are fairly standard documents for the most part: they have a
title,
> > > body, abstract, etc, all of which we index and search with Lucene.
> > > However, they also have a few fields that aren't standard,
including
> a
> >
> > > list of involved organizations as well as a dollar amount for each
> > > report. The organizations are stored as IDs, and we add the org
ID
> > > field multiple times, once for each organization involved in the
> > > report. The funding is also stored as a non-indexed field on the
> > > Lucene document.
> > >
> > > What I'm trying to do is build a quick-and-dirty org-by-dollar
> report
> > > off of the reports that match the user's query. So, a query for
> > > "aerospace" might match 50,000 documents, and I want to show the
> user
> > > the top 5 organizations in terms of dollars. Again, I know
> reporting
> > > like this isn't what Lucene was meant for, and we do have some
ideas
> > > on how to handle it long-term, but for now, I'm trying to get it
> > > working as well as I can using Lucene alone, and Lucene does do a
> > > great job of finding the relevant set of documents to build a
report
> > > from.
> > >
> > > On Wed, Nov 11, 2009 at 8:56 PM, Michael Garski
> > > <mgarski@myspace-inc.com
> > > >wrote:
> > >
> > > > Matt,
> > > >
> > > > StringIndex is for use when a field has only one value in it for
> the
> >
> > > > purposes of sorting results, not for tokenized fields with
> multiple
> > > > values. TermVectors might be a better approach, but for 50K
docs,
> > > > you'll encounter an IO hit on reading them.
> > > >
> > > > I'm curious why you are looking to grab all of the terms for a
> > > > ScoreDoc... can you shed some light on that?
> > > >
> > > > Michael
> > > >
> > > > -----Original Message-----
> > > > From: Matt Honeycutt [mailto:mbhoneycutt@gmail.com]
> > > > Sent: Wednesday, November 11, 2009 4:57 PM
> > > > To: lucene-net-user@incubator.apache.org
> > > > Subject: FieldLookup for field with multiple values
> > > >
> > > > It seems that the StringIndex returned by
> > > > FieldCache.Fields.Default.GetStringIndex() only indexes one
value
> > > > for a document even when the document has multiple values for
the
> > > > field. Is there a performant want to get all the values for a
> > > > particular field in a ScoreDoc? I'm having to do this across
the
> > > > entire result set of ScoreDocs (up to 50,000), and retrieving
the
> > > > values through LuceneDocument.GetFields is not going to cut it.
> > > >
> > > >
> > >
> > >
> > >
> >
> >
>
>
Re: FieldLookup for field with multiple values
Posted by Matt Honeycutt <mb...@gmail.com>.
I shudder to think about what the higher-ups will say about the cost of
faster storage. They're very stingy on things that make our lives easier.
:)
I like the idea of caching the term vectors somehow, though I wonder how
that would scale with millions of documents. I will add that to the list of
things to prototype for our long-term solution.
Thanks for the feedback,
---Matt
On Thu, Nov 12, 2009 at 12:14 PM, Michael Garski <mg...@myspace-inc.com>wrote:
> Matt,
>
> Metadata can be collected during the course of a search in a
> HitCollector (note that 2.9 will deprecate the current HitCollector in
> favor of the more flexible Collector). One approach to collecting
> facets is to index the field with the desired facet metadata with
> TermVectors enabled. During collection you can retrieve the
> TermFreqVector for a document with Reader.GetTermFreqVector for each
> document and collect/aggregate metadata however you like.
>
> Making the call to get the TermFreqVector for each hit will be a
> performance hit due to IO and deserialization (even though it's very
> fast, 50K times in one search is a lot). At that point you can add
> faster IO (Fusion IO rules this space) or during an index warmup create
> an in-memory structure with lookups to the vectors keyed off the
> document id.
>
> Hope that helps.
>
> Michael
>
> -----Original Message-----
> From: Matt Honeycutt [mailto:mbhoneycutt@gmail.com]
> Sent: Thursday, November 12, 2009 7:13 AM
> To: lucene-net-user@incubator.apache.org
> Subject: Re: FieldLookup for field with multiple values
>
> I can elaborate a little on what our *planned* approach for utilizing
> SQL
> Server is. I don't know if this will work, but I've done similar things
> with SQL CLR and haven't had it explode (yet), so I'm hopeful. Anyway:
>
> Our system needs two types of output, basically: the full-text reports
> (I'll
> call them documents to reduce ambiguity), and then statistical reports
> built
> on those documents. The documents can easily be retrieved from Lucene,
> so
> the challenge is building the reports. For that, my plan is to submit
> the
> same query string to SQL Server that I sent to Lucene. Internally, SQL
> Server would then pass the query back to Lucene and retrieve a list of
> document IDs that matched. The communication may be over WCF or
> something
> similar, and will be compressed during transit to reduce IO overhead.
> Once
> SQL Server has the IDs, they will be loaded into a temporary table (with
> indexes) or a table variable, which will then be used to filter the
> metadata
> that the statistical reports are built from.
>
> I have no idea how such a system would perform. I do hope to do some
> feasibility tests sometime Real Soon (like in the next few weeks), and
> I'll
> post my results if I manage to get it working.
>
> If anyone has any other suggestions, please do share.
>
> On Thu, Nov 12, 2009 at 8:43 AM, Moray McConnachie <
> mmcconna@oxford-analytica.com> wrote:
>
> > >While we're discussing this, anyone have any advice or suggestions
> for
> > a better solution? We've considered a few things for our long-term
> > solution.
> >
> > I'd be very interested to hear thoughts on intersecting SQL and Lucene
> > too, as in our case we have very large lists of organisations which
> have
> > different permissions (stored in SQL) for different documents stored
> in
> > Lucene. Showing in the search results only those documents to which
> the
> > organisation has permission is quite expensive for queries with lots
> of
> > results. Storing it in the documents is not manageable because they
> need
> > to be updated frequently across multiple documents. Currently we
> > precompute a list for each organisation, cache that in memory, and
> > recache it every time that organisation is updated. However this too
> is
> > costly.
> >
> > Storing the Lucene document nos in SQL during indexing, and then
> passing
> > the list of Lucene document nos matching a search to SQL for filtering
> > seems the right way to go. But Matt is right, the problem is with
> > scaling this to searches returning many thousands of documents.
> >
> > Yours,
> > M.
> > -------------------------------------
> > Moray McConnachie
> > Director of IT +44 1865 261 600
> > Oxford Analytica http://www.oxan.com
> >
> > -----Original Message-----
> > From: Matt Honeycutt [mailto:mbhoneycutt@gmail.com]
> > Sent: 12 November 2009 13:53
> > To: lucene-net-user@incubator.apache.org
> > Subject: Re: FieldLookup for field with multiple values
> >
> > Yeah, it is sort of like your standard faceting scenario, except there
> > are about 20,000 facets (organizations), and there's complex
> > relationships among the facets.
> >
> > The reports we're dealing with only occasionally break the funding up
> by
> > organization, so we decided (for now) to just store a single funding
> > value, then break it up after-the-fact by dividing it by the number of
> > organizations. So no, the funding is only stored once.
> >
> > While we're discussing this, anyone have any advice or suggestions for
> a
> > better solution? We've considered a few things for our long-term
> > solution.
> > One is to put this metadata in a SQL Server instance, and use SQL CLR
> to
> > build a temporary table based on document IDs from a Lucene index
> > (hosted over WCF or something similar), then do the reporting within
> SQL
> > Server. We plan to compress the list of IDs going back from Lucene to
> > SQL Server to cut down on IO overhead, but we're still concerned that
> > approach won't scale as we go from hundreds of thousands to millions
> of
> > reports.
> >
> > Another option we've discussed is to precompute data cubes and use
> these
> > to calculate reporting information. The concern here is the high
> > dimensionality of the data (we have about 20,000 distinct
> organizations
> > now, but fully expect that to increase by an order of magnitude) as
> well
> > as the accuracy of the generated reports, since there's (probably) not
> a
> > good way to divide the cube based on arbitrary Lucene queries.
> >
> > On Thu, Nov 12, 2009 at 1:03 AM, Michael Garski
> > <mg...@myspace-inc.com>wrote:
> >
> > > Sounds like a full-text search with the results simply being facets
> on
> >
> > > the organizations sorted by the funding amount?
> > >
> > > You mentioned adding the org ID once for each document. Do you do
> the
> >
> > > same for the funding, with the funding for each corresponding
> > organization?
> > >
> > > Michael
> > >
> > >
> > > -----Original Message-----
> > > From: Matt Honeycutt [mailto:mbhoneycutt@gmail.com]
> > > Sent: Wed 11/11/2009 10:17 PM
> > > To: lucene-net-user@incubator.apache.org
> > > Subject: Re: FieldLookup for field with multiple values
> > >
> > > Well, let me prefix what I'm about to describe by saying that I know
> > > that I'm doing something with Lucene that it wasn't meant to do.
> This
> >
> > > is for a "proof of concept" system that I'm helping put together on
> a
> > > tight schedule with very limited resources, and we're trying to get
> to
> >
> > > a mostly-working state as quickly as possible.
> > >
> > > That said, we are basically storing reports in Lucene. The reports
> > > are fairly standard documents for the most part: they have a title,
> > > body, abstract, etc, all of which we index and search with Lucene.
> > > However, they also have a few fields that aren't standard, including
> a
> >
> > > list of involved organizations as well as a dollar amount for each
> > > report. The organizations are stored as IDs, and we add the org ID
> > > field multiple times, once for each organization involved in the
> > > report. The funding is also stored as a non-indexed field on the
> > > Lucene document.
> > >
> > > What I'm trying to do is build a quick-and-dirty org-by-dollar
> report
> > > off of the reports that match the user's query. So, a query for
> > > "aerospace" might match 50,000 documents, and I want to show the
> user
> > > the top 5 organizations in terms of dollars. Again, I know
> reporting
> > > like this isn't what Lucene was meant for, and we do have some ideas
> > > on how to handle it long-term, but for now, I'm trying to get it
> > > working as well as I can using Lucene alone, and Lucene does do a
> > > great job of finding the relevant set of documents to build a report
> > > from.
> > >
> > > On Wed, Nov 11, 2009 at 8:56 PM, Michael Garski
> > > <mgarski@myspace-inc.com
> > > >wrote:
> > >
> > > > Matt,
> > > >
> > > > StringIndex is for use when a field has only one value in it for
> the
> >
> > > > purposes of sorting results, not for tokenized fields with
> multiple
> > > > values. TermVectors might be a better approach, but for 50K docs,
> > > > you'll encounter an IO hit on reading them.
> > > >
> > > > I'm curious why you are looking to grab all of the terms for a
> > > > ScoreDoc... can you shed some light on that?
> > > >
> > > > Michael
> > > >
> > > > -----Original Message-----
> > > > From: Matt Honeycutt [mailto:mbhoneycutt@gmail.com]
> > > > Sent: Wednesday, November 11, 2009 4:57 PM
> > > > To: lucene-net-user@incubator.apache.org
> > > > Subject: FieldLookup for field with multiple values
> > > >
> > > > It seems that the StringIndex returned by
> > > > FieldCache.Fields.Default.GetStringIndex() only indexes one value
> > > > for a document even when the document has multiple values for the
> > > > field. Is there a performant want to get all the values for a
> > > > particular field in a ScoreDoc? I'm having to do this across the
> > > > entire result set of ScoreDocs (up to 50,000), and retrieving the
> > > > values through LuceneDocument.GetFields is not going to cut it.
> > > >
> > > >
> > >
> > >
> > >
> >
> >
>
>
RE: FieldLookup for field with multiple values
Posted by Michael Garski <mg...@myspace-inc.com>.
Matt,
Metadata can be collected during the course of a search in a
HitCollector (note that 2.9 will deprecate the current HitCollector in
favor of the more flexible Collector). One approach to collecting
facets is to index the field with the desired facet metadata with
TermVectors enabled. During collection you can retrieve the
TermFreqVector for a document with Reader.GetTermFreqVector for each
document and collect/aggregate metadata however you like.
Making the call to get the TermFreqVector for each hit will be a
performance hit due to IO and deserialization (even though it's very
fast, 50K times in one search is a lot). At that point you can add
faster IO (Fusion IO rules this space) or during an index warmup create
an in-memory structure with lookups to the vectors keyed off the
document id.
Hope that helps.
Michael
-----Original Message-----
From: Matt Honeycutt [mailto:mbhoneycutt@gmail.com]
Sent: Thursday, November 12, 2009 7:13 AM
To: lucene-net-user@incubator.apache.org
Subject: Re: FieldLookup for field with multiple values
I can elaborate a little on what our *planned* approach for utilizing
SQL
Server is. I don't know if this will work, but I've done similar things
with SQL CLR and haven't had it explode (yet), so I'm hopeful. Anyway:
Our system needs two types of output, basically: the full-text reports
(I'll
call them documents to reduce ambiguity), and then statistical reports
built
on those documents. The documents can easily be retrieved from Lucene,
so
the challenge is building the reports. For that, my plan is to submit
the
same query string to SQL Server that I sent to Lucene. Internally, SQL
Server would then pass the query back to Lucene and retrieve a list of
document IDs that matched. The communication may be over WCF or
something
similar, and will be compressed during transit to reduce IO overhead.
Once
SQL Server has the IDs, they will be loaded into a temporary table (with
indexes) or a table variable, which will then be used to filter the
metadata
that the statistical reports are built from.
I have no idea how such a system would perform. I do hope to do some
feasibility tests sometime Real Soon (like in the next few weeks), and
I'll
post my results if I manage to get it working.
If anyone has any other suggestions, please do share.
On Thu, Nov 12, 2009 at 8:43 AM, Moray McConnachie <
mmcconna@oxford-analytica.com> wrote:
> >While we're discussing this, anyone have any advice or suggestions
for
> a better solution? We've considered a few things for our long-term
> solution.
>
> I'd be very interested to hear thoughts on intersecting SQL and Lucene
> too, as in our case we have very large lists of organisations which
have
> different permissions (stored in SQL) for different documents stored
in
> Lucene. Showing in the search results only those documents to which
the
> organisation has permission is quite expensive for queries with lots
of
> results. Storing it in the documents is not manageable because they
need
> to be updated frequently across multiple documents. Currently we
> precompute a list for each organisation, cache that in memory, and
> recache it every time that organisation is updated. However this too
is
> costly.
>
> Storing the Lucene document nos in SQL during indexing, and then
passing
> the list of Lucene document nos matching a search to SQL for filtering
> seems the right way to go. But Matt is right, the problem is with
> scaling this to searches returning many thousands of documents.
>
> Yours,
> M.
> -------------------------------------
> Moray McConnachie
> Director of IT +44 1865 261 600
> Oxford Analytica http://www.oxan.com
>
> -----Original Message-----
> From: Matt Honeycutt [mailto:mbhoneycutt@gmail.com]
> Sent: 12 November 2009 13:53
> To: lucene-net-user@incubator.apache.org
> Subject: Re: FieldLookup for field with multiple values
>
> Yeah, it is sort of like your standard faceting scenario, except there
> are about 20,000 facets (organizations), and there's complex
> relationships among the facets.
>
> The reports we're dealing with only occasionally break the funding up
by
> organization, so we decided (for now) to just store a single funding
> value, then break it up after-the-fact by dividing it by the number of
> organizations. So no, the funding is only stored once.
>
> While we're discussing this, anyone have any advice or suggestions for
a
> better solution? We've considered a few things for our long-term
> solution.
> One is to put this metadata in a SQL Server instance, and use SQL CLR
to
> build a temporary table based on document IDs from a Lucene index
> (hosted over WCF or something similar), then do the reporting within
SQL
> Server. We plan to compress the list of IDs going back from Lucene to
> SQL Server to cut down on IO overhead, but we're still concerned that
> approach won't scale as we go from hundreds of thousands to millions
of
> reports.
>
> Another option we've discussed is to precompute data cubes and use
these
> to calculate reporting information. The concern here is the high
> dimensionality of the data (we have about 20,000 distinct
organizations
> now, but fully expect that to increase by an order of magnitude) as
well
> as the accuracy of the generated reports, since there's (probably) not
a
> good way to divide the cube based on arbitrary Lucene queries.
>
> On Thu, Nov 12, 2009 at 1:03 AM, Michael Garski
> <mg...@myspace-inc.com>wrote:
>
> > Sounds like a full-text search with the results simply being facets
on
>
> > the organizations sorted by the funding amount?
> >
> > You mentioned adding the org ID once for each document. Do you do
the
>
> > same for the funding, with the funding for each corresponding
> organization?
> >
> > Michael
> >
> >
> > -----Original Message-----
> > From: Matt Honeycutt [mailto:mbhoneycutt@gmail.com]
> > Sent: Wed 11/11/2009 10:17 PM
> > To: lucene-net-user@incubator.apache.org
> > Subject: Re: FieldLookup for field with multiple values
> >
> > Well, let me prefix what I'm about to describe by saying that I know
> > that I'm doing something with Lucene that it wasn't meant to do.
This
>
> > is for a "proof of concept" system that I'm helping put together on
a
> > tight schedule with very limited resources, and we're trying to get
to
>
> > a mostly-working state as quickly as possible.
> >
> > That said, we are basically storing reports in Lucene. The reports
> > are fairly standard documents for the most part: they have a title,
> > body, abstract, etc, all of which we index and search with Lucene.
> > However, they also have a few fields that aren't standard, including
a
>
> > list of involved organizations as well as a dollar amount for each
> > report. The organizations are stored as IDs, and we add the org ID
> > field multiple times, once for each organization involved in the
> > report. The funding is also stored as a non-indexed field on the
> > Lucene document.
> >
> > What I'm trying to do is build a quick-and-dirty org-by-dollar
report
> > off of the reports that match the user's query. So, a query for
> > "aerospace" might match 50,000 documents, and I want to show the
user
> > the top 5 organizations in terms of dollars. Again, I know
reporting
> > like this isn't what Lucene was meant for, and we do have some ideas
> > on how to handle it long-term, but for now, I'm trying to get it
> > working as well as I can using Lucene alone, and Lucene does do a
> > great job of finding the relevant set of documents to build a report
> > from.
> >
> > On Wed, Nov 11, 2009 at 8:56 PM, Michael Garski
> > <mgarski@myspace-inc.com
> > >wrote:
> >
> > > Matt,
> > >
> > > StringIndex is for use when a field has only one value in it for
the
>
> > > purposes of sorting results, not for tokenized fields with
multiple
> > > values. TermVectors might be a better approach, but for 50K docs,
> > > you'll encounter an IO hit on reading them.
> > >
> > > I'm curious why you are looking to grab all of the terms for a
> > > ScoreDoc... can you shed some light on that?
> > >
> > > Michael
> > >
> > > -----Original Message-----
> > > From: Matt Honeycutt [mailto:mbhoneycutt@gmail.com]
> > > Sent: Wednesday, November 11, 2009 4:57 PM
> > > To: lucene-net-user@incubator.apache.org
> > > Subject: FieldLookup for field with multiple values
> > >
> > > It seems that the StringIndex returned by
> > > FieldCache.Fields.Default.GetStringIndex() only indexes one value
> > > for a document even when the document has multiple values for the
> > > field. Is there a performant want to get all the values for a
> > > particular field in a ScoreDoc? I'm having to do this across the
> > > entire result set of ScoreDocs (up to 50,000), and retrieving the
> > > values through LuceneDocument.GetFields is not going to cut it.
> > >
> > >
> >
> >
> >
>
>
Re: FieldLookup for field with multiple values
Posted by Matt Honeycutt <mb...@gmail.com>.
I can elaborate a little on what our *planned* approach for utilizing SQL
Server is. I don't know if this will work, but I've done similar things
with SQL CLR and haven't had it explode (yet), so I'm hopeful. Anyway:
Our system needs two types of output, basically: the full-text reports (I'll
call them documents to reduce ambiguity), and then statistical reports built
on those documents. The documents can easily be retrieved from Lucene, so
the challenge is building the reports. For that, my plan is to submit the
same query string to SQL Server that I sent to Lucene. Internally, SQL
Server would then pass the query back to Lucene and retrieve a list of
document IDs that matched. The communication may be over WCF or something
similar, and will be compressed during transit to reduce IO overhead. Once
SQL Server has the IDs, they will be loaded into a temporary table (with
indexes) or a table variable, which will then be used to filter the metadata
that the statistical reports are built from.
I have no idea how such a system would perform. I do hope to do some
feasibility tests sometime Real Soon (like in the next few weeks), and I'll
post my results if I manage to get it working.
If anyone has any other suggestions, please do share.
On Thu, Nov 12, 2009 at 8:43 AM, Moray McConnachie <
mmcconna@oxford-analytica.com> wrote:
> >While we're discussing this, anyone have any advice or suggestions for
> a better solution? We've considered a few things for our long-term
> solution.
>
> I'd be very interested to hear thoughts on intersecting SQL and Lucene
> too, as in our case we have very large lists of organisations which have
> different permissions (stored in SQL) for different documents stored in
> Lucene. Showing in the search results only those documents to which the
> organisation has permission is quite expensive for queries with lots of
> results. Storing it in the documents is not manageable because they need
> to be updated frequently across multiple documents. Currently we
> precompute a list for each organisation, cache that in memory, and
> recache it every time that organisation is updated. However this too is
> costly.
>
> Storing the Lucene document nos in SQL during indexing, and then passing
> the list of Lucene document nos matching a search to SQL for filtering
> seems the right way to go. But Matt is right, the problem is with
> scaling this to searches returning many thousands of documents.
>
> Yours,
> M.
> -------------------------------------
> Moray McConnachie
> Director of IT +44 1865 261 600
> Oxford Analytica http://www.oxan.com
>
> -----Original Message-----
> From: Matt Honeycutt [mailto:mbhoneycutt@gmail.com]
> Sent: 12 November 2009 13:53
> To: lucene-net-user@incubator.apache.org
> Subject: Re: FieldLookup for field with multiple values
>
> Yeah, it is sort of like your standard faceting scenario, except there
> are about 20,000 facets (organizations), and there's complex
> relationships among the facets.
>
> The reports we're dealing with only occasionally break the funding up by
> organization, so we decided (for now) to just store a single funding
> value, then break it up after-the-fact by dividing it by the number of
> organizations. So no, the funding is only stored once.
>
> While we're discussing this, anyone have any advice or suggestions for a
> better solution? We've considered a few things for our long-term
> solution.
> One is to put this metadata in a SQL Server instance, and use SQL CLR to
> build a temporary table based on document IDs from a Lucene index
> (hosted over WCF or something similar), then do the reporting within SQL
> Server. We plan to compress the list of IDs going back from Lucene to
> SQL Server to cut down on IO overhead, but we're still concerned that
> approach won't scale as we go from hundreds of thousands to millions of
> reports.
>
> Another option we've discussed is to precompute data cubes and use these
> to calculate reporting information. The concern here is the high
> dimensionality of the data (we have about 20,000 distinct organizations
> now, but fully expect that to increase by an order of magnitude) as well
> as the accuracy of the generated reports, since there's (probably) not a
> good way to divide the cube based on arbitrary Lucene queries.
>
> On Thu, Nov 12, 2009 at 1:03 AM, Michael Garski
> <mg...@myspace-inc.com>wrote:
>
> > Sounds like a full-text search with the results simply being facets on
>
> > the organizations sorted by the funding amount?
> >
> > You mentioned adding the org ID once for each document. Do you do the
>
> > same for the funding, with the funding for each corresponding
> organization?
> >
> > Michael
> >
> >
> > -----Original Message-----
> > From: Matt Honeycutt [mailto:mbhoneycutt@gmail.com]
> > Sent: Wed 11/11/2009 10:17 PM
> > To: lucene-net-user@incubator.apache.org
> > Subject: Re: FieldLookup for field with multiple values
> >
> > Well, let me prefix what I'm about to describe by saying that I know
> > that I'm doing something with Lucene that it wasn't meant to do. This
>
> > is for a "proof of concept" system that I'm helping put together on a
> > tight schedule with very limited resources, and we're trying to get to
>
> > a mostly-working state as quickly as possible.
> >
> > That said, we are basically storing reports in Lucene. The reports
> > are fairly standard documents for the most part: they have a title,
> > body, abstract, etc, all of which we index and search with Lucene.
> > However, they also have a few fields that aren't standard, including a
>
> > list of involved organizations as well as a dollar amount for each
> > report. The organizations are stored as IDs, and we add the org ID
> > field multiple times, once for each organization involved in the
> > report. The funding is also stored as a non-indexed field on the
> > Lucene document.
> >
> > What I'm trying to do is build a quick-and-dirty org-by-dollar report
> > off of the reports that match the user's query. So, a query for
> > "aerospace" might match 50,000 documents, and I want to show the user
> > the top 5 organizations in terms of dollars. Again, I know reporting
> > like this isn't what Lucene was meant for, and we do have some ideas
> > on how to handle it long-term, but for now, I'm trying to get it
> > working as well as I can using Lucene alone, and Lucene does do a
> > great job of finding the relevant set of documents to build a report
> > from.
> >
> > On Wed, Nov 11, 2009 at 8:56 PM, Michael Garski
> > <mgarski@myspace-inc.com
> > >wrote:
> >
> > > Matt,
> > >
> > > StringIndex is for use when a field has only one value in it for the
>
> > > purposes of sorting results, not for tokenized fields with multiple
> > > values. TermVectors might be a better approach, but for 50K docs,
> > > you'll encounter an IO hit on reading them.
> > >
> > > I'm curious why you are looking to grab all of the terms for a
> > > ScoreDoc... can you shed some light on that?
> > >
> > > Michael
> > >
> > > -----Original Message-----
> > > From: Matt Honeycutt [mailto:mbhoneycutt@gmail.com]
> > > Sent: Wednesday, November 11, 2009 4:57 PM
> > > To: lucene-net-user@incubator.apache.org
> > > Subject: FieldLookup for field with multiple values
> > >
> > > It seems that the StringIndex returned by
> > > FieldCache.Fields.Default.GetStringIndex() only indexes one value
> > > for a document even when the document has multiple values for the
> > > field. Is there a performant want to get all the values for a
> > > particular field in a ScoreDoc? I'm having to do this across the
> > > entire result set of ScoreDocs (up to 50,000), and retrieving the
> > > values through LuceneDocument.GetFields is not going to cut it.
> > >
> > >
> >
> >
> >
>
>
RE: FieldLookup for field with multiple values
Posted by Moray McConnachie <mm...@oxford-analytica.com>.
>While we're discussing this, anyone have any advice or suggestions for
a better solution? We've considered a few things for our long-term
solution.
I'd be very interested to hear thoughts on intersecting SQL and Lucene
too, as in our case we have very large lists of organisations which have
different permissions (stored in SQL) for different documents stored in
Lucene. Showing in the search results only those documents to which the
organisation has permission is quite expensive for queries with lots of
results. Storing it in the documents is not manageable because they need
to be updated frequently across multiple documents. Currently we
precompute a list for each organisation, cache that in memory, and
recache it every time that organisation is updated. However this too is
costly.
Storing the Lucene document nos in SQL during indexing, and then passing
the list of Lucene document nos matching a search to SQL for filtering
seems the right way to go. But Matt is right, the problem is with
scaling this to searches returning many thousands of documents.
Yours,
M.
-------------------------------------
Moray McConnachie
Director of IT +44 1865 261 600
Oxford Analytica http://www.oxan.com
-----Original Message-----
From: Matt Honeycutt [mailto:mbhoneycutt@gmail.com]
Sent: 12 November 2009 13:53
To: lucene-net-user@incubator.apache.org
Subject: Re: FieldLookup for field with multiple values
Yeah, it is sort of like your standard faceting scenario, except there
are about 20,000 facets (organizations), and there's complex
relationships among the facets.
The reports we're dealing with only occasionally break the funding up by
organization, so we decided (for now) to just store a single funding
value, then break it up after-the-fact by dividing it by the number of
organizations. So no, the funding is only stored once.
While we're discussing this, anyone have any advice or suggestions for a
better solution? We've considered a few things for our long-term
solution.
One is to put this metadata in a SQL Server instance, and use SQL CLR to
build a temporary table based on document IDs from a Lucene index
(hosted over WCF or something similar), then do the reporting within SQL
Server. We plan to compress the list of IDs going back from Lucene to
SQL Server to cut down on IO overhead, but we're still concerned that
approach won't scale as we go from hundreds of thousands to millions of
reports.
Another option we've discussed is to precompute data cubes and use these
to calculate reporting information. The concern here is the high
dimensionality of the data (we have about 20,000 distinct organizations
now, but fully expect that to increase by an order of magnitude) as well
as the accuracy of the generated reports, since there's (probably) not a
good way to divide the cube based on arbitrary Lucene queries.
On Thu, Nov 12, 2009 at 1:03 AM, Michael Garski
<mg...@myspace-inc.com>wrote:
> Sounds like a full-text search with the results simply being facets on
> the organizations sorted by the funding amount?
>
> You mentioned adding the org ID once for each document. Do you do the
> same for the funding, with the funding for each corresponding
organization?
>
> Michael
>
>
> -----Original Message-----
> From: Matt Honeycutt [mailto:mbhoneycutt@gmail.com]
> Sent: Wed 11/11/2009 10:17 PM
> To: lucene-net-user@incubator.apache.org
> Subject: Re: FieldLookup for field with multiple values
>
> Well, let me prefix what I'm about to describe by saying that I know
> that I'm doing something with Lucene that it wasn't meant to do. This
> is for a "proof of concept" system that I'm helping put together on a
> tight schedule with very limited resources, and we're trying to get to
> a mostly-working state as quickly as possible.
>
> That said, we are basically storing reports in Lucene. The reports
> are fairly standard documents for the most part: they have a title,
> body, abstract, etc, all of which we index and search with Lucene.
> However, they also have a few fields that aren't standard, including a
> list of involved organizations as well as a dollar amount for each
> report. The organizations are stored as IDs, and we add the org ID
> field multiple times, once for each organization involved in the
> report. The funding is also stored as a non-indexed field on the
> Lucene document.
>
> What I'm trying to do is build a quick-and-dirty org-by-dollar report
> off of the reports that match the user's query. So, a query for
> "aerospace" might match 50,000 documents, and I want to show the user
> the top 5 organizations in terms of dollars. Again, I know reporting
> like this isn't what Lucene was meant for, and we do have some ideas
> on how to handle it long-term, but for now, I'm trying to get it
> working as well as I can using Lucene alone, and Lucene does do a
> great job of finding the relevant set of documents to build a report
> from.
>
> On Wed, Nov 11, 2009 at 8:56 PM, Michael Garski
> <mgarski@myspace-inc.com
> >wrote:
>
> > Matt,
> >
> > StringIndex is for use when a field has only one value in it for the
> > purposes of sorting results, not for tokenized fields with multiple
> > values. TermVectors might be a better approach, but for 50K docs,
> > you'll encounter an IO hit on reading them.
> >
> > I'm curious why you are looking to grab all of the terms for a
> > ScoreDoc... can you shed some light on that?
> >
> > Michael
> >
> > -----Original Message-----
> > From: Matt Honeycutt [mailto:mbhoneycutt@gmail.com]
> > Sent: Wednesday, November 11, 2009 4:57 PM
> > To: lucene-net-user@incubator.apache.org
> > Subject: FieldLookup for field with multiple values
> >
> > It seems that the StringIndex returned by
> > FieldCache.Fields.Default.GetStringIndex() only indexes one value
> > for a document even when the document has multiple values for the
> > field. Is there a performant want to get all the values for a
> > particular field in a ScoreDoc? I'm having to do this across the
> > entire result set of ScoreDocs (up to 50,000), and retrieving the
> > values through LuceneDocument.GetFields is not going to cut it.
> >
> >
>
>
>
Re: FieldLookup for field with multiple values
Posted by Matt Honeycutt <mb...@gmail.com>.
Yeah, it is sort of like your standard faceting scenario, except there are
about 20,000 facets (organizations), and there's complex relationships among
the facets.
The reports we're dealing with only occasionally break the funding up by
organization, so we decided (for now) to just store a single funding value,
then break it up after-the-fact by dividing it by the number of
organizations. So no, the funding is only stored once.
While we're discussing this, anyone have any advice or suggestions for a
better solution? We've considered a few things for our long-term solution.
One is to put this metadata in a SQL Server instance, and use SQL CLR to
build a temporary table based on document IDs from a Lucene index (hosted
over WCF or something similar), then do the reporting within SQL Server. We
plan to compress the list of IDs going back from Lucene to SQL Server to cut
down on IO overhead, but we're still concerned that approach won't scale as
we go from hundreds of thousands to millions of reports.
Another option we've discussed is to precompute data cubes and use these to
calculate reporting information. The concern here is the high
dimensionality of the data (we have about 20,000 distinct organizations now,
but fully expect that to increase by an order of magnitude) as well as the
accuracy of the generated reports, since there's (probably) not a good way
to divide the cube based on arbitrary Lucene queries.
On Thu, Nov 12, 2009 at 1:03 AM, Michael Garski <mg...@myspace-inc.com>wrote:
> Sounds like a full-text search with the results simply being facets on the
> organizations sorted by the funding amount?
>
> You mentioned adding the org ID once for each document. Do you do the same
> for the funding, with the funding for each corresponding organization?
>
> Michael
>
>
> -----Original Message-----
> From: Matt Honeycutt [mailto:mbhoneycutt@gmail.com]
> Sent: Wed 11/11/2009 10:17 PM
> To: lucene-net-user@incubator.apache.org
> Subject: Re: FieldLookup for field with multiple values
>
> Well, let me prefix what I'm about to describe by saying that I know that
> I'm doing something with Lucene that it wasn't meant to do. This is for a
> "proof of concept" system that I'm helping put together on a tight schedule
> with very limited resources, and we're trying to get to a mostly-working
> state as quickly as possible.
>
> That said, we are basically storing reports in Lucene. The reports are
> fairly standard documents for the most part: they have a title, body,
> abstract, etc, all of which we index and search with Lucene. However, they
> also have a few fields that aren't standard, including a list of involved
> organizations as well as a dollar amount for each report. The
> organizations
> are stored as IDs, and we add the org ID field multiple times, once for
> each
> organization involved in the report. The funding is also stored as a
> non-indexed field on the Lucene document.
>
> What I'm trying to do is build a quick-and-dirty org-by-dollar report off
> of
> the reports that match the user's query. So, a query for "aerospace" might
> match 50,000 documents, and I want to show the user the top 5 organizations
> in terms of dollars. Again, I know reporting like this isn't what Lucene
> was meant for, and we do have some ideas on how to handle it long-term, but
> for now, I'm trying to get it working as well as I can using Lucene alone,
> and Lucene does do a great job of finding the relevant set of documents to
> build a report from.
>
> On Wed, Nov 11, 2009 at 8:56 PM, Michael Garski <mgarski@myspace-inc.com
> >wrote:
>
> > Matt,
> >
> > StringIndex is for use when a field has only one value in it for the
> > purposes of sorting results, not for tokenized fields with multiple
> > values. TermVectors might be a better approach, but for 50K docs,
> > you'll encounter an IO hit on reading them.
> >
> > I'm curious why you are looking to grab all of the terms for a
> > ScoreDoc... can you shed some light on that?
> >
> > Michael
> >
> > -----Original Message-----
> > From: Matt Honeycutt [mailto:mbhoneycutt@gmail.com]
> > Sent: Wednesday, November 11, 2009 4:57 PM
> > To: lucene-net-user@incubator.apache.org
> > Subject: FieldLookup for field with multiple values
> >
> > It seems that the StringIndex returned by
> > FieldCache.Fields.Default.GetStringIndex() only indexes one value for a
> > document even when the document has multiple values for the field. Is
> > there
> > a performant want to get all the values for a particular field in a
> > ScoreDoc? I'm having to do this across the entire result set of
> > ScoreDocs
> > (up to 50,000), and retrieving the values through
> > LuceneDocument.GetFields
> > is not going to cut it.
> >
> >
>
>
>
RE: FieldLookup for field with multiple values
Posted by Michael Garski <mg...@myspace-inc.com>.
Sounds like a full-text search with the results simply being facets on the organizations sorted by the funding amount?
You mentioned adding the org ID once for each document. Do you do the same for the funding, with the funding for each corresponding organization?
Michael
-----Original Message-----
From: Matt Honeycutt [mailto:mbhoneycutt@gmail.com]
Sent: Wed 11/11/2009 10:17 PM
To: lucene-net-user@incubator.apache.org
Subject: Re: FieldLookup for field with multiple values
Well, let me prefix what I'm about to describe by saying that I know that
I'm doing something with Lucene that it wasn't meant to do. This is for a
"proof of concept" system that I'm helping put together on a tight schedule
with very limited resources, and we're trying to get to a mostly-working
state as quickly as possible.
That said, we are basically storing reports in Lucene. The reports are
fairly standard documents for the most part: they have a title, body,
abstract, etc, all of which we index and search with Lucene. However, they
also have a few fields that aren't standard, including a list of involved
organizations as well as a dollar amount for each report. The organizations
are stored as IDs, and we add the org ID field multiple times, once for each
organization involved in the report. The funding is also stored as a
non-indexed field on the Lucene document.
What I'm trying to do is build a quick-and-dirty org-by-dollar report off of
the reports that match the user's query. So, a query for "aerospace" might
match 50,000 documents, and I want to show the user the top 5 organizations
in terms of dollars. Again, I know reporting like this isn't what Lucene
was meant for, and we do have some ideas on how to handle it long-term, but
for now, I'm trying to get it working as well as I can using Lucene alone,
and Lucene does do a great job of finding the relevant set of documents to
build a report from.
On Wed, Nov 11, 2009 at 8:56 PM, Michael Garski <mg...@myspace-inc.com>wrote:
> Matt,
>
> StringIndex is for use when a field has only one value in it for the
> purposes of sorting results, not for tokenized fields with multiple
> values. TermVectors might be a better approach, but for 50K docs,
> you'll encounter an IO hit on reading them.
>
> I'm curious why you are looking to grab all of the terms for a
> ScoreDoc... can you shed some light on that?
>
> Michael
>
> -----Original Message-----
> From: Matt Honeycutt [mailto:mbhoneycutt@gmail.com]
> Sent: Wednesday, November 11, 2009 4:57 PM
> To: lucene-net-user@incubator.apache.org
> Subject: FieldLookup for field with multiple values
>
> It seems that the StringIndex returned by
> FieldCache.Fields.Default.GetStringIndex() only indexes one value for a
> document even when the document has multiple values for the field. Is
> there
> a performant want to get all the values for a particular field in a
> ScoreDoc? I'm having to do this across the entire result set of
> ScoreDocs
> (up to 50,000), and retrieving the values through
> LuceneDocument.GetFields
> is not going to cut it.
>
>
Re: FieldLookup for field with multiple values
Posted by Matt Honeycutt <mb...@gmail.com>.
Well, let me prefix what I'm about to describe by saying that I know that
I'm doing something with Lucene that it wasn't meant to do. This is for a
"proof of concept" system that I'm helping put together on a tight schedule
with very limited resources, and we're trying to get to a mostly-working
state as quickly as possible.
That said, we are basically storing reports in Lucene. The reports are
fairly standard documents for the most part: they have a title, body,
abstract, etc, all of which we index and search with Lucene. However, they
also have a few fields that aren't standard, including a list of involved
organizations as well as a dollar amount for each report. The organizations
are stored as IDs, and we add the org ID field multiple times, once for each
organization involved in the report. The funding is also stored as a
non-indexed field on the Lucene document.
What I'm trying to do is build a quick-and-dirty org-by-dollar report off of
the reports that match the user's query. So, a query for "aerospace" might
match 50,000 documents, and I want to show the user the top 5 organizations
in terms of dollars. Again, I know reporting like this isn't what Lucene
was meant for, and we do have some ideas on how to handle it long-term, but
for now, I'm trying to get it working as well as I can using Lucene alone,
and Lucene does do a great job of finding the relevant set of documents to
build a report from.
On Wed, Nov 11, 2009 at 8:56 PM, Michael Garski <mg...@myspace-inc.com>wrote:
> Matt,
>
> StringIndex is for use when a field has only one value in it for the
> purposes of sorting results, not for tokenized fields with multiple
> values. TermVectors might be a better approach, but for 50K docs,
> you'll encounter an IO hit on reading them.
>
> I'm curious why you are looking to grab all of the terms for a
> ScoreDoc... can you shed some light on that?
>
> Michael
>
> -----Original Message-----
> From: Matt Honeycutt [mailto:mbhoneycutt@gmail.com]
> Sent: Wednesday, November 11, 2009 4:57 PM
> To: lucene-net-user@incubator.apache.org
> Subject: FieldLookup for field with multiple values
>
> It seems that the StringIndex returned by
> FieldCache.Fields.Default.GetStringIndex() only indexes one value for a
> document even when the document has multiple values for the field. Is
> there
> a performant want to get all the values for a particular field in a
> ScoreDoc? I'm having to do this across the entire result set of
> ScoreDocs
> (up to 50,000), and retrieving the values through
> LuceneDocument.GetFields
> is not going to cut it.
>
>
RE: FieldLookup for field with multiple values
Posted by Michael Garski <mg...@myspace-inc.com>.
Matt,
StringIndex is for use when a field has only one value in it for the
purposes of sorting results, not for tokenized fields with multiple
values. TermVectors might be a better approach, but for 50K docs,
you'll encounter an IO hit on reading them.
I'm curious why you are looking to grab all of the terms for a
ScoreDoc... can you shed some light on that?
Michael
-----Original Message-----
From: Matt Honeycutt [mailto:mbhoneycutt@gmail.com]
Sent: Wednesday, November 11, 2009 4:57 PM
To: lucene-net-user@incubator.apache.org
Subject: FieldLookup for field with multiple values
It seems that the StringIndex returned by
FieldCache.Fields.Default.GetStringIndex() only indexes one value for a
document even when the document has multiple values for the field. Is
there
a performant want to get all the values for a particular field in a
ScoreDoc? I'm having to do this across the entire result set of
ScoreDocs
(up to 50,000), and retrieving the values through
LuceneDocument.GetFields
is not going to cut it.