Simulate doc linking via post filter cache check

[tedsolr <ts...@sciquest.com>]

I'm pulling my hair out on this one - and there's not much of that to begin
with. The problem I have is that updating 10M denormalized docs in a
collection takes about 5 hours. Soon there will be collections with 100M
docs and a 50 hour update cycle will not be acceptable. The process involves
cleaning (deleting) the marker fields, querying the collection with user
defined saved searches, then updating the marker fields in every matched
doc. If I can normalize based on the searches the processing time should go
way down: delete marker docs, query the collection with user defined saved
searches, then insert marker docs. The time savings comes from 1) deleting
and inserting docs is faster than updating docs, 2) the number of saved
searches is at least 1000X less than the number of docs.

A doc may have a couple hundred fields, but looks sorta like this:
{"id":123_5677899","searchid":"34","name":"Johnson", ...}

To normalize I would remove the searchid into a new doc:
{"id":"S234","searchid":"34","doclist":["123_5677899","123_5677898",...]}

The "link" is established by the doclist field which is multivalued and
contains the ids from the real docs. All this is doable, the problem is that
when users create saved searches they must only match docs that have not
already been matched by another search. That's why there's only one doc
"type" now - every matched doc has a marker (searchid) which makes the Solr
search work. Since it's not possible to do a RDBMS like search joining the 2
doc types, I need to run the saved search: find docs where name=Johnson,
then drop the docs that are not in a doclist.

So, maybe if I manage a custom cache of matched doc ids, I can check each
returned id against the cache and drop the docs that are not in it. I think
this could be done in a post filter. There will be a big memory hit to
maintain this cache, but does this seem like a performant solution to my
problem?

Thanks!
v5.2.1
All collections are one shard with replication factor 2



--
View this message in context: http://lucene.472066.n3.nabble.com/Simulate-doc-linking-via-post-filter-cache-check-tp4275842.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Simulate doc linking via post filter cache check

[tedsolr <ts...@sciquest.com>]

Mikhail, that's an interesting idea. If a terms list could stand in for a
cache that may be helpful. What I don't fully see is how the search would
work. Building an explicit negative terms query with returned IDs doesn't
seem possible as that list would be in the millions. To drastically speed my
process up I need to stop updating the data docs and only update the marker
(linked) docs.

Starting with 0 terms indexed for field "doclist" the very first search is
easy:
- put all result IDs in the doclist
Second search must exclude results that are already represented in the
doclist field. How is that possible?

I should mention I do an explicit hard commit after running each saved
search, to prevent consecutive searches from overlapping. That is probably
costing me. I didn't know it was possible to do an explicit soft commit. How
do you do that with SolrJ (not by setting maxDocs=1 in the config I hope)?




--
View this message in context: http://lucene.472066.n3.nabble.com/Simulate-doc-linking-via-post-filter-cache-check-tp4275842p4275929.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Simulate doc linking via post filter cache check

[tedsolr <ts...@sciquest.com>]

The terms component will not work for me because it holds on to terms from
deleted documents. My indexes are too volatile.

I could perform a search for every match - but that would not perform. Maybe
I need something that can compare two searches. Anyone know of an existing
filter component does something similar?



--
View this message in context: http://lucene.472066.n3.nabble.com/Simulate-doc-linking-via-post-filter-cache-check-tp4275842p4281783.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Simulate doc linking via post filter cache check

[Mikhail Khludnev <mk...@griddynamics.com>]

The problem description is really long, you know.
I'd attack the statement:

> Since it's not possible to do a RDBMS like search joining the 2
> doc types, I need to run the saved search: find docs where name=Johnson,
> then drop the docs that are not in a doclist.
>

And also, if you remove all markers or starting from empty collection, and
do softCommit after every add, you can use /terms (TermsComponent) as a
"cache of" inserted doclist_ids.

For me it seems more like transient cache for ETL process, this state makes
sense only for single load operation; and not a search engine concern,
really.

Also, you can think from the opposite side:
after you search for the first request: q=name:Johnson and add it result to
markers, the second request might be q=name:Jacobson -name:Johnson etc,
until you exceed maxBooleanClauses limit, that can be leveraged by another
meanings.

and also every request can append list of responded ids into the growing
list of negative terms query:
q=name:Jacobson -{terms f=ids v=$alreadyseen}&alreadyseen=2,4,6,8,...

or they might be joined from markers, if you can afford often softCommit,
etc there are plenty of approaches to keep your hair.


On Tue, May 10, 2016 at 6:44 PM, tedsolr <ts...@sciquest.com> wrote:

> I'm pulling my hair out on this one - and there's not much of that to begin
> with. The problem I have is that updating 10M denormalized docs in a
> collection takes about 5 hours. Soon there will be collections with 100M
> docs and a 50 hour update cycle will not be acceptable. The process
> involves
> cleaning (deleting) the marker fields, querying the collection with user
> defined saved searches, then updating the marker fields in every matched
> doc. If I can normalize based on the searches the processing time should go
> way down: delete marker docs, query the collection with user defined saved
> searches, then insert marker docs. The time savings comes from 1) deleting
> and inserting docs is faster than updating docs, 2) the number of saved
> searches is at least 1000X less than the number of docs.
>
> A doc may have a couple hundred fields, but looks sorta like this:
> {"id":123_5677899","searchid":"34","name":"Johnson", ...}
>
> To normalize I would remove the searchid into a new doc:
> {"id":"S234","searchid":"34","doclist":["123_5677899","123_5677898",...]}
>
> The "link" is established by the doclist field which is multivalued and
> contains the ids from the real docs. All this is doable, the problem is
> that
> when users create saved searches they must only match docs that have not
> already been matched by another search. That's why there's only one doc
> "type" now - every matched doc has a marker (searchid) which makes the Solr
> search work. Since it's not possible to do a RDBMS like search joining the
> 2
> doc types, I need to run the saved search: find docs where name=Johnson,
> then drop the docs that are not in a doclist.
>
> So, maybe if I manage a custom cache of matched doc ids, I can check each
> returned id against the cache and drop the docs that are not in it. I think
> this could be done in a post filter. There will be a big memory hit to
> maintain this cache, but does this seem like a performant solution to my
> problem?
>
> Thanks!
> v5.2.1
> All collections are one shard with replication factor 2
>
>
>
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/Simulate-doc-linking-via-post-filter-cache-check-tp4275842.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>



-- 
Sincerely yours
Mikhail Khludnev
Principal Engineer,
Grid Dynamics

<http://www.griddynamics.com>
<mk...@griddynamics.com>