[jira] [Resolved] (CONNECTORS-989) Support virtual child document model

["Karl Wright (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/CONNECTORS-989?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Karl Wright resolved CONNECTORS-989.
------------------------------------

       Resolution: Fixed
    Fix Version/s: ManifoldCF 1.7

r1612102

> Support virtual child document model
> ------------------------------------
>
>                 Key: CONNECTORS-989
>                 URL: https://issues.apache.org/jira/browse/CONNECTORS-989
>             Project: ManifoldCF
>          Issue Type: Improvement
>          Components: Framework agents process
>    Affects Versions: ManifoldCF 1.7
>            Reporter: Karl Wright
>            Assignee: Karl Wright
>             Fix For: ManifoldCF 1.7
>
>
> In some cases, documents that are indexed may be virtual children of those that are queued.  A good example of this is RSS feeds where the data being indexed all comes from the feed.
> In order to implement this, the following changes would be required:
> (1) IProcessActivity.ingestDocument() has a variant which allows you to include a virtual child document identifier in addition to the main document identifier.
> (2) IIncrementalIngester's addOrReplaceDocument receives TWO document keys -- one for main (queued) document identifier, one for child virtual document identifier.
> (3) IIncrementalIngester has two new methods: beginDocument() and endDocument(), both of which take a main (queued) document identifier as an argument.
> (4) ingeststatus table has two additional columns: a state, and a child key.
> (5) The flow is: at beginDocument() time, put all records relating to a document into a "processing" state.  Documents that are seen have their state changed.  Documents never
>    encountered are deleted at the end.
> (6) Incremental decisions not to update an output record STILL will require that the record be touched and its state set.
> (7) DocumentIngest records for the entire set of children will be fetched when the document is queued.
> (8) The getDocumentVersions() method must be modified to allow return of version strings for all children, although there can be "shortcuts" as well (where a single version
>     string applies to all children.)
> (9) The decision about whether to refetch a document is based on the returned version strings and on those fetched by the stuffer thread.
> (10) Similarly, processDocuments() receives version strings for all virtual children.
> (11) There is no need to actively reset the state of document records on restart; the current logic should be robust enough to be able to generate the required deletions.
> (12) Deleting a document deletes ALL child virtual documents.  This happens within the incremental ingester.
> (13) Requeuing interval must be computed across all children, taking the minimum, since there's no requirement that an ingeststatus record exist for the parent.
> (14) All other logic, including making sure only one agent operates on a url at a time, is the same.
> (15) Interrupting the delete phase is safe because next time the doc is processed the records will be removed.
> Analysis:
> - The critical thing is making the non-virtual case no worse.
> - For a virtual child document, instead of one db access, there are two.
> - For document records that are not changed, there are two additional writes that were not needed before.
> - There's an additional index (or the document key index has another subfield).
> - If the queries written can be done in such a way as to treat the standard (no child document) case specially, we may be able to avoid much impact; only two index queries per document returning zero rows each
> - If we handle the standard case using the same mechanism, the WorkerThread logic dealing with deletions can go away.
> Summary:
> - Additional database overhead in the non-virtual indexing case consists of one additional write and one additional zero-row query, OR two additional zero-row queries.
> - Additional database overhead in the non-virtual skip case consists of two additional writes, OR two additional zero-row queries.
> - The overhead is low but is significant and will impact overall framework performance
> - The up-sides are as follows: (a) handling an important but infrequent case better; (b) less connector involvement in indexing (e.g., IProcessActivity.deleteDocument() does nothing now, and can be deprecated).



--
This message was sent by Atlassian JIRA
(v6.2#6252)