You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@manifoldcf.apache.org by Markus Schuch <ma...@web.de> on 2014/11/21 13:59:07 UTC

Document components

Hi,

is there any example implementation of the new document component feature invented with CONNECTORS-989?

I read the section Document components in [0] but i still do not know how to actually write a repository connector
that ingests multiple documents originating from a single document of a repository.

My first guess was to call the method "activities.ingestDocumentWithException" multiple times with the same identifier but distinct component identifiers during document processing.

I wrote a simple TestConnector: 
The processDocuments method looks like:

    public void processDocuments(String[] documentIdentifiers,
            String[] versions, IProcessActivity activities,
            DocumentSpecification spec, boolean[] scanOnly)
            throws ManifoldCFException, ServiceInterruption {
        
        int i = 0;
        for (String identifier : documentIdentifiers) {

            byte[] content1 = "test content 1".getBytes();
            byte[] content2 = "test content 2".getBytes();
            byte[] content3 = "test content 3".getBytes();

                        
            RepositoryDocument rd1 = new RepositoryDocument();
            rd1.setBinary(new ByteArrayInputStream(content1), content1.length);

            RepositoryDocument rd2 = new RepositoryDocument();
            rd2.setBinary(new ByteArrayInputStream(content2), content2.length);

            
            RepositoryDocument rd3 = new RepositoryDocument();
            rd3.setBinary(new ByteArrayInputStream(content3), content3.length);

            
            System.out.println("process " + identifier);
            
            try {
                activities.ingestDocumentWithException(identifier, "comp1", versions[i], identifier+"/comp1", rd1);
                activities.ingestDocumentWithException(identifier, "comp2", versions[i], identifier+"/comp2", rd2);
                activities.ingestDocumentWithException(identifier, "comp3", versions[i], identifier+"/comp3", rd3);
            } catch (IOException e) {
                e.printStackTrace();
            }
            
            i++;
        }
        
    }

For seeding the method getDocumentIdentifiers() returns a stream with a single document identifier "testidentifier1".
Full Code available at [1].

But subsequent calls of ingestDocumentWithException result in deletions of a previously added component.

job end	1416573910333(copmtest1) 0	1	
document ingest 	testidentifier1/comp3 OK 12	8	
document deletion 	testidentifier1/comp1 OK 0	2	
document ingest 	testidentifier1/comp2 OK 12	10	
document deletion 	testidentifier1/comp1 OK 0	3	
document ingest 	testidentifier1/comp1 OK 12	13	
job start	1416573910333(copmtest1) 0	1

Only testidentifier1/comp2 and testidentifier1/comp3 exist in the output connection after the job is finished.

I feed i might have a false understanding of the concept...

Any help is appreciated.

Thanks in advance
Markus

--
Using ManifoldCF 1.7.1 
[0] http://manifoldcf.apache.org/release/release-1.7.2/en_US/writing-repository-connectors.html
[1] https://gist.github.com/schuch/43809594ad8f81ddc625

Re: Document components

Posted by Karl Wright <da...@gmail.com>.
See CONNECTORS-1115.  I looked into this; looked relatively easy to add a
method to IProcessActivity that does what you request.  Please give it a
try and let me know how it works for you.

Thanks,
Karl


On Tue, Nov 25, 2014 at 6:22 PM, Karl Wright <da...@gmail.com> wrote:

> Hi Markus,
>
> >>>>>>
> noDocument() removes the document or the specified component from the
> output but keeps track of the version in the status queue. The decision of
> not indexing the document/component is considered persistent as long as the
> version string does not change.
>
> deleteDocument() removes the document and all its components from output
> and the status queue. The decision of not indexing the document will have
> to be made again when the document is processed the next time (version
> string is irrelevant)
>
> removeDocument() removes the primary document from the output and from the
> status queue but keeps components in the output. The decision of not
> indexing the document will have to be made again when the document is
> processed the next time (version string is irrelevant)
>
> Is this correct?
> <<<<<<
>
> Yes.
>
> >>>>>>
> The scenario is indexing documents with embedded documents. The embedded
> documents are ingested as components.
>
> We assume a document with multiple components was ingested. For the next
> processing the version does not change.
> So the whole document should not be refetched.
> But how i can prevent the deletion of the components when the document is
> not re-fetched?
> I saw the method "retainDocument" which seems to be the way to go, but the
> problem is that without fetching the document
> i have no knowledge about the available components.
> Is there any other way to retain all components without knowing them?
> <<<<<<
>
> Not at present; the assumption for components is that the processing of a
> primary document will allow your connector to determine disposition of all
> components of the primary document every time that processDocuments() is
> called for it.  Effectively that means that the assumption is that
> determining what components are in a document is a relatively inexpensive
> operation.  It's necessary to make that assumption, because that's the only
> way the bookkeeping can work - MCF needs to know what happens with the
> components, when all it has is a processDocuments() call.  I'll look into
> how hard it would be to add the functionality you are looking for though,
> and get back to you.
>
> >>>>>>
> About a patch for a Test Connector:
> I think i could contribute something.
> Do you have general requirements/guideline for test connectors?
> Are there examples of a similar test connector?
> <<<<<<
>
> Look at
> framework/pull-agent/src/test/java/org/apache/manifoldcf/crawler/tests.
> There are a number of test connectors there, and tests that use them.
>
> Thanks,
> Karl
>
>
> On Tue, Nov 25, 2014 at 5:53 PM, Markus Schuch <ma...@web.de>
> wrote:
>
>> Hi Karl,
>>
>> thanks for the clarification about primary document disposition.
>>
>> I'm still not 100% sure if i understand the differences... i try to
>> explain it in my own words:
>>
>> noDocument() removes the document or the specified component from the
>> output but keeps track of the version in the status queue. The decision of
>> not indexing the document/component is considered persistent as long as the
>> version string does not change.
>>
>> deleteDocument() removes the document and all its components from output
>> and the status queue. The decision of not indexing the document will have
>> to be made again when the document is processed the next time (version
>> string is irrelevant)
>>
>> removeDocument() removes the primary document from the output and from
>> the status queue but keeps components in the output. The decision of not
>> indexing the document will have to be made again when the document is
>> processed the next time (version string is irrelevant)
>>
>> Is this correct?
>>
>> -----------------------------
>>
>> An new question i have:
>>
>> The scenario is indexing documents with embedded documents. The embedded
>> documents are ingested as components.
>>
>> We assume a document with multiple components was ingested. For the next
>> processing the version does not change.
>> So the whole document should not be refetched.
>> But how i can prevent the deletion of the components when the document is
>> not re-fetched?
>> I saw the method "retainDocument" which seems to be the way to go, but
>> the problem is that without fetching the document
>> i have no knowledge about the available components.
>> Is there any other way to retain all components without knowing them?
>>
>> ----------------------------
>>
>> About a patch for a Test Connector:
>> I think i could contribute something.
>> Do you have general requirements/guideline for test connectors?
>> Are there examples of a similar test connector?
>>
>> Regards,
>> Markus
>>
>
>

Re: Document components

Posted by Karl Wright <da...@gmail.com>.
Hi Markus,

>>>>>>
noDocument() removes the document or the specified component from the
output but keeps track of the version in the status queue. The decision of
not indexing the document/component is considered persistent as long as the
version string does not change.

deleteDocument() removes the document and all its components from output
and the status queue. The decision of not indexing the document will have
to be made again when the document is processed the next time (version
string is irrelevant)

removeDocument() removes the primary document from the output and from the
status queue but keeps components in the output. The decision of not
indexing the document will have to be made again when the document is
processed the next time (version string is irrelevant)

Is this correct?
<<<<<<

Yes.

>>>>>>
The scenario is indexing documents with embedded documents. The embedded
documents are ingested as components.

We assume a document with multiple components was ingested. For the next
processing the version does not change.
So the whole document should not be refetched.
But how i can prevent the deletion of the components when the document is
not re-fetched?
I saw the method "retainDocument" which seems to be the way to go, but the
problem is that without fetching the document
i have no knowledge about the available components.
Is there any other way to retain all components without knowing them?
<<<<<<

Not at present; the assumption for components is that the processing of a
primary document will allow your connector to determine disposition of all
components of the primary document every time that processDocuments() is
called for it.  Effectively that means that the assumption is that
determining what components are in a document is a relatively inexpensive
operation.  It's necessary to make that assumption, because that's the only
way the bookkeeping can work - MCF needs to know what happens with the
components, when all it has is a processDocuments() call.  I'll look into
how hard it would be to add the functionality you are looking for though,
and get back to you.

>>>>>>
About a patch for a Test Connector:
I think i could contribute something.
Do you have general requirements/guideline for test connectors?
Are there examples of a similar test connector?
<<<<<<

Look at
framework/pull-agent/src/test/java/org/apache/manifoldcf/crawler/tests.
There are a number of test connectors there, and tests that use them.

Thanks,
Karl


On Tue, Nov 25, 2014 at 5:53 PM, Markus Schuch <ma...@web.de> wrote:

> Hi Karl,
>
> thanks for the clarification about primary document disposition.
>
> I'm still not 100% sure if i understand the differences... i try to
> explain it in my own words:
>
> noDocument() removes the document or the specified component from the
> output but keeps track of the version in the status queue. The decision of
> not indexing the document/component is considered persistent as long as the
> version string does not change.
>
> deleteDocument() removes the document and all its components from output
> and the status queue. The decision of not indexing the document will have
> to be made again when the document is processed the next time (version
> string is irrelevant)
>
> removeDocument() removes the primary document from the output and from the
> status queue but keeps components in the output. The decision of not
> indexing the document will have to be made again when the document is
> processed the next time (version string is irrelevant)
>
> Is this correct?
>
> -----------------------------
>
> An new question i have:
>
> The scenario is indexing documents with embedded documents. The embedded
> documents are ingested as components.
>
> We assume a document with multiple components was ingested. For the next
> processing the version does not change.
> So the whole document should not be refetched.
> But how i can prevent the deletion of the components when the document is
> not re-fetched?
> I saw the method "retainDocument" which seems to be the way to go, but the
> problem is that without fetching the document
> i have no knowledge about the available components.
> Is there any other way to retain all components without knowing them?
>
> ----------------------------
>
> About a patch for a Test Connector:
> I think i could contribute something.
> Do you have general requirements/guideline for test connectors?
> Are there examples of a similar test connector?
>
> Regards,
> Markus
>

Re: Document components

Posted by Markus Schuch <ma...@web.de>.
Hi Karl,
 
thanks for the clarification about primary document disposition.

I'm still not 100% sure if i understand the differences... i try to explain it in my own words:

noDocument() removes the document or the specified component from the output but keeps track of the version in the status queue. The decision of not indexing the document/component is considered persistent as long as the version string does not change.

deleteDocument() removes the document and all its components from output and the status queue. The decision of not indexing the document will have to be made again when the document is processed the next time (version string is irrelevant)

removeDocument() removes the primary document from the output and from the status queue but keeps components in the output. The decision of not indexing the document will have to be made again when the document is processed the next time (version string is irrelevant)

Is this correct?

-----------------------------

An new question i have:

The scenario is indexing documents with embedded documents. The embedded documents are ingested as components.

We assume a document with multiple components was ingested. For the next processing the version does not change.
So the whole document should not be refetched.
But how i can prevent the deletion of the components when the document is not re-fetched?
I saw the method "retainDocument" which seems to be the way to go, but the problem is that without fetching the document
i have no knowledge about the available components. 
Is there any other way to retain all components without knowing them?

----------------------------

About a patch for a Test Connector:
I think i could contribute something.
Do you have general requirements/guideline for test connectors?
Are there examples of a similar test connector?

Regards,
Markus

Re: Document components

Posted by Karl Wright <da...@gmail.com>.
Ok, code committed.

Since you are obviously the first person to really try out the component
feature, I wondered if you would be willing to submit a test connector that
uses it, as a patch.  I apologize for not fully testing this feature in the
1.7 release -- the person who requested the feature obviously didn't
actually use it, and although I'd intended to write a test, that did not
get done.

Thanks,
Karl


On Tue, Nov 25, 2014 at 5:03 AM, Karl Wright <da...@gmail.com> wrote:

> I believe I've found the problem with removeDocument(), and will commit a
> fix shortly.
>
> To clarify your question about primary document disposition:
>
> For the case where you have no document, and you never expect there to be
> a document again (because, for instance, it was deleted), then
> removeDocument() is the right thing to call.  If the case is different,
> namely that the document exists but is no longer indexable for whatever
> reason, it's better to call noDocument() instead, because you can supply a
> version string, and then MCF will know not to ask you to process it again
> unless that string changes.
>
> Karl
>
>
> On Tue, Nov 25, 2014 at 4:46 AM, Karl Wright <da...@gmail.com> wrote:
>
>> Hi Markus,
>>
>> I've created a ticket for the exception.  CONNECTORS-1114.
>>
>> As for removal of a primary document that is not mentioned, do you mean
>> that within processDocuments(), if you don't call any disposition method
>> for a primary document, then that document is left around?  If so, that
>> behavior is intended -- it was necessary for backwards compatibility.  The
>> document should, of course, be cleaned up at the end of the job, as long as
>> you are not doing a minimal crawl.
>>
>> If you are seeing some other kind of behavior, please try to describe it
>> more completely so that I have a better idea what you mean.
>>
>> Thanks,
>> Karl
>>
>>
>> On Tue, Nov 25, 2014 at 3:25 AM, Markus Schuch <ma...@web.de>
>> wrote:
>>
>>> Hi Karl,
>>>
>>> the patch for CONNECTORS-1111 fixes the cleanup issue.
>>>
>>> Another question about primary documents and their components:
>>>
>>> I have ingested a primary document with some components.
>>> During the next processing the primary document should no longer be
>>> indexed, but the sub components of it should still be indexed.
>>>
>>> My understanding is, that not mentioned components are automatically
>>> removed.
>>> Since the primary document is the "null" component, i expected the
>>> framework would remove the primary document component if not mentioned, too.
>>>
>>> But this is not the case. Is this another bug or do i have to remove the
>>> primary document somehow manually?
>>>
>>> There is an activity method removeDocument(identifier) which seems
>>> related.
>>> But i do not fully understand the described usage scenario in the
>>> method's javadoc.
>>>
>>> I tried the method. The result was the following database exception:
>>> (Patches for CONNECTORS-1110 and CONNECTORS-1111 are applied)
>>>
>>> 2014-11-25 08:30:07,868 ERROR [Worker thread '1']
>>> org.apache.manifoldcf.crawlerthreads: Worker thread aborting and restarting
>>> due to database connection reset: Database exception: SQLException doing
>>> query (HY0000): You need to set exactly 3 parameters on the prepared
>>> statement
>>> org.apache.manifoldcf.core.interfaces.ManifoldCFException: Database
>>> exception: SQLException doing query (HY0000): You need to set exactly 3
>>> parameters on the prepared statement
>>>     at
>>> org.apache.manifoldcf.core.database.Database$ExecuteQueryThread.finishUp(Database.java:702)
>>>     at
>>> org.apache.manifoldcf.core.database.Database.executeViaThread(Database.java:728)
>>>     at
>>> org.apache.manifoldcf.core.database.Database.executeUncachedQuery(Database.java:762)
>>>     at
>>> org.apache.manifoldcf.core.database.Database$QueryCacheExecutor.create(Database.java:1435)
>>>     at
>>> org.apache.manifoldcf.core.cachemanager.CacheManager.findObjectsAndExecute(CacheManager.java:146)
>>>     at
>>> org.apache.manifoldcf.core.database.Database.executeQuery(Database.java:191)
>>>     at
>>> org.apache.manifoldcf.core.database.DBInterfaceMySQL.performQuery(DBInterfaceMySQL.java:875)
>>>     at
>>> org.apache.manifoldcf.core.database.BaseTable.performQuery(BaseTable.java:221)
>>>     at
>>> org.apache.manifoldcf.agents.incrementalingest.IncrementalIngester.findRowIdsForDocIds(IncrementalIngester.java:1518)
>>>     at
>>> org.apache.manifoldcf.agents.incrementalingest.IncrementalIngester.documentRemoveMultiple(IncrementalIngester.java:1377)
>>>     at
>>> org.apache.manifoldcf.agents.incrementalingest.IncrementalIngester.documentRemove(IncrementalIngester.java:803)
>>>     at
>>> org.apache.manifoldcf.crawler.system.WorkerThread$ProcessActivity.removeDocument(WorkerThread.java:1674)
>>>     at
>>> com.example.mcf.TestConnector.processDocuments(TestConnector.java:278)
>>>     at
>>> org.apache.manifoldcf.crawler.connectors.BaseRepositoryConnector.processDocuments(BaseRepositoryConnector.java:670)
>>>     at
>>> org.apache.manifoldcf.crawler.connectors.BaseRepositoryConnector.processDocuments(BaseRepositoryConnector.java:649)
>>>     at
>>> org.apache.manifoldcf.crawler.connectors.BaseRepositoryConnector.processDocuments(BaseRepositoryConnector.java:402)
>>>     at
>>> org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:380)
>>> Caused by: java.sql.SQLException: You need to set exactly 3 parameters
>>> on the prepared statement
>>>     at
>>> org.mariadb.jdbc.internal.SQLExceptionMapper.get(SQLExceptionMapper.java:149)
>>>     at
>>> org.mariadb.jdbc.internal.SQLExceptionMapper.throwException(SQLExceptionMapper.java:106)
>>>     at
>>> org.mariadb.jdbc.MySQLStatement.executeQueryEpilog(MySQLStatement.java:264)
>>>     at org.mariadb.jdbc.MySQLStatement.execute(MySQLStatement.java:288)
>>>     at
>>> org.mariadb.jdbc.MySQLStatement.executeQuery(MySQLStatement.java:302)
>>>     at
>>> org.mariadb.jdbc.MySQLPreparedStatement.executeQuery(MySQLPreparedStatement.java:112)
>>>     at
>>> org.apache.manifoldcf.core.database.Database.execute(Database.java:880)
>>>     at
>>> org.apache.manifoldcf.core.database.Database$ExecuteQueryThread.run(Database.java:683)
>>> Caused by: org.mariadb.jdbc.internal.common.QueryException: You need to
>>> set exactly 3 parameters on the prepared statement
>>>     at
>>> org.mariadb.jdbc.internal.common.query.MySQLParameterizedQuery.validate(MySQLParameterizedQuery.java:117)
>>>     at
>>> org.mariadb.jdbc.internal.mysql.MySQLProtocol.executeQuery(MySQLProtocol.java:976)
>>>     at org.mariadb.jdbc.MySQLStatement.execute(MySQLStatement.java:281)
>>>
>>> Regards,
>>> Markus
>>>
>>
>>
>

Re: Document components

Posted by Karl Wright <da...@gmail.com>.
I believe I've found the problem with removeDocument(), and will commit a
fix shortly.

To clarify your question about primary document disposition:

For the case where you have no document, and you never expect there to be a
document again (because, for instance, it was deleted), then
removeDocument() is the right thing to call.  If the case is different,
namely that the document exists but is no longer indexable for whatever
reason, it's better to call noDocument() instead, because you can supply a
version string, and then MCF will know not to ask you to process it again
unless that string changes.

Karl


On Tue, Nov 25, 2014 at 4:46 AM, Karl Wright <da...@gmail.com> wrote:

> Hi Markus,
>
> I've created a ticket for the exception.  CONNECTORS-1114.
>
> As for removal of a primary document that is not mentioned, do you mean
> that within processDocuments(), if you don't call any disposition method
> for a primary document, then that document is left around?  If so, that
> behavior is intended -- it was necessary for backwards compatibility.  The
> document should, of course, be cleaned up at the end of the job, as long as
> you are not doing a minimal crawl.
>
> If you are seeing some other kind of behavior, please try to describe it
> more completely so that I have a better idea what you mean.
>
> Thanks,
> Karl
>
>
> On Tue, Nov 25, 2014 at 3:25 AM, Markus Schuch <ma...@web.de>
> wrote:
>
>> Hi Karl,
>>
>> the patch for CONNECTORS-1111 fixes the cleanup issue.
>>
>> Another question about primary documents and their components:
>>
>> I have ingested a primary document with some components.
>> During the next processing the primary document should no longer be
>> indexed, but the sub components of it should still be indexed.
>>
>> My understanding is, that not mentioned components are automatically
>> removed.
>> Since the primary document is the "null" component, i expected the
>> framework would remove the primary document component if not mentioned, too.
>>
>> But this is not the case. Is this another bug or do i have to remove the
>> primary document somehow manually?
>>
>> There is an activity method removeDocument(identifier) which seems
>> related.
>> But i do not fully understand the described usage scenario in the
>> method's javadoc.
>>
>> I tried the method. The result was the following database exception:
>> (Patches for CONNECTORS-1110 and CONNECTORS-1111 are applied)
>>
>> 2014-11-25 08:30:07,868 ERROR [Worker thread '1']
>> org.apache.manifoldcf.crawlerthreads: Worker thread aborting and restarting
>> due to database connection reset: Database exception: SQLException doing
>> query (HY0000): You need to set exactly 3 parameters on the prepared
>> statement
>> org.apache.manifoldcf.core.interfaces.ManifoldCFException: Database
>> exception: SQLException doing query (HY0000): You need to set exactly 3
>> parameters on the prepared statement
>>     at
>> org.apache.manifoldcf.core.database.Database$ExecuteQueryThread.finishUp(Database.java:702)
>>     at
>> org.apache.manifoldcf.core.database.Database.executeViaThread(Database.java:728)
>>     at
>> org.apache.manifoldcf.core.database.Database.executeUncachedQuery(Database.java:762)
>>     at
>> org.apache.manifoldcf.core.database.Database$QueryCacheExecutor.create(Database.java:1435)
>>     at
>> org.apache.manifoldcf.core.cachemanager.CacheManager.findObjectsAndExecute(CacheManager.java:146)
>>     at
>> org.apache.manifoldcf.core.database.Database.executeQuery(Database.java:191)
>>     at
>> org.apache.manifoldcf.core.database.DBInterfaceMySQL.performQuery(DBInterfaceMySQL.java:875)
>>     at
>> org.apache.manifoldcf.core.database.BaseTable.performQuery(BaseTable.java:221)
>>     at
>> org.apache.manifoldcf.agents.incrementalingest.IncrementalIngester.findRowIdsForDocIds(IncrementalIngester.java:1518)
>>     at
>> org.apache.manifoldcf.agents.incrementalingest.IncrementalIngester.documentRemoveMultiple(IncrementalIngester.java:1377)
>>     at
>> org.apache.manifoldcf.agents.incrementalingest.IncrementalIngester.documentRemove(IncrementalIngester.java:803)
>>     at
>> org.apache.manifoldcf.crawler.system.WorkerThread$ProcessActivity.removeDocument(WorkerThread.java:1674)
>>     at
>> com.example.mcf.TestConnector.processDocuments(TestConnector.java:278)
>>     at
>> org.apache.manifoldcf.crawler.connectors.BaseRepositoryConnector.processDocuments(BaseRepositoryConnector.java:670)
>>     at
>> org.apache.manifoldcf.crawler.connectors.BaseRepositoryConnector.processDocuments(BaseRepositoryConnector.java:649)
>>     at
>> org.apache.manifoldcf.crawler.connectors.BaseRepositoryConnector.processDocuments(BaseRepositoryConnector.java:402)
>>     at
>> org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:380)
>> Caused by: java.sql.SQLException: You need to set exactly 3 parameters on
>> the prepared statement
>>     at
>> org.mariadb.jdbc.internal.SQLExceptionMapper.get(SQLExceptionMapper.java:149)
>>     at
>> org.mariadb.jdbc.internal.SQLExceptionMapper.throwException(SQLExceptionMapper.java:106)
>>     at
>> org.mariadb.jdbc.MySQLStatement.executeQueryEpilog(MySQLStatement.java:264)
>>     at org.mariadb.jdbc.MySQLStatement.execute(MySQLStatement.java:288)
>>     at
>> org.mariadb.jdbc.MySQLStatement.executeQuery(MySQLStatement.java:302)
>>     at
>> org.mariadb.jdbc.MySQLPreparedStatement.executeQuery(MySQLPreparedStatement.java:112)
>>     at
>> org.apache.manifoldcf.core.database.Database.execute(Database.java:880)
>>     at
>> org.apache.manifoldcf.core.database.Database$ExecuteQueryThread.run(Database.java:683)
>> Caused by: org.mariadb.jdbc.internal.common.QueryException: You need to
>> set exactly 3 parameters on the prepared statement
>>     at
>> org.mariadb.jdbc.internal.common.query.MySQLParameterizedQuery.validate(MySQLParameterizedQuery.java:117)
>>     at
>> org.mariadb.jdbc.internal.mysql.MySQLProtocol.executeQuery(MySQLProtocol.java:976)
>>     at org.mariadb.jdbc.MySQLStatement.execute(MySQLStatement.java:281)
>>
>> Regards,
>> Markus
>>
>
>

Re: Document components

Posted by Karl Wright <da...@gmail.com>.
Hi Markus,

I've created a ticket for the exception.  CONNECTORS-1114.

As for removal of a primary document that is not mentioned, do you mean
that within processDocuments(), if you don't call any disposition method
for a primary document, then that document is left around?  If so, that
behavior is intended -- it was necessary for backwards compatibility.  The
document should, of course, be cleaned up at the end of the job, as long as
you are not doing a minimal crawl.

If you are seeing some other kind of behavior, please try to describe it
more completely so that I have a better idea what you mean.

Thanks,
Karl


On Tue, Nov 25, 2014 at 3:25 AM, Markus Schuch <ma...@web.de> wrote:

> Hi Karl,
>
> the patch for CONNECTORS-1111 fixes the cleanup issue.
>
> Another question about primary documents and their components:
>
> I have ingested a primary document with some components.
> During the next processing the primary document should no longer be
> indexed, but the sub components of it should still be indexed.
>
> My understanding is, that not mentioned components are automatically
> removed.
> Since the primary document is the "null" component, i expected the
> framework would remove the primary document component if not mentioned, too.
>
> But this is not the case. Is this another bug or do i have to remove the
> primary document somehow manually?
>
> There is an activity method removeDocument(identifier) which seems related.
> But i do not fully understand the described usage scenario in the method's
> javadoc.
>
> I tried the method. The result was the following database exception:
> (Patches for CONNECTORS-1110 and CONNECTORS-1111 are applied)
>
> 2014-11-25 08:30:07,868 ERROR [Worker thread '1']
> org.apache.manifoldcf.crawlerthreads: Worker thread aborting and restarting
> due to database connection reset: Database exception: SQLException doing
> query (HY0000): You need to set exactly 3 parameters on the prepared
> statement
> org.apache.manifoldcf.core.interfaces.ManifoldCFException: Database
> exception: SQLException doing query (HY0000): You need to set exactly 3
> parameters on the prepared statement
>     at
> org.apache.manifoldcf.core.database.Database$ExecuteQueryThread.finishUp(Database.java:702)
>     at
> org.apache.manifoldcf.core.database.Database.executeViaThread(Database.java:728)
>     at
> org.apache.manifoldcf.core.database.Database.executeUncachedQuery(Database.java:762)
>     at
> org.apache.manifoldcf.core.database.Database$QueryCacheExecutor.create(Database.java:1435)
>     at
> org.apache.manifoldcf.core.cachemanager.CacheManager.findObjectsAndExecute(CacheManager.java:146)
>     at
> org.apache.manifoldcf.core.database.Database.executeQuery(Database.java:191)
>     at
> org.apache.manifoldcf.core.database.DBInterfaceMySQL.performQuery(DBInterfaceMySQL.java:875)
>     at
> org.apache.manifoldcf.core.database.BaseTable.performQuery(BaseTable.java:221)
>     at
> org.apache.manifoldcf.agents.incrementalingest.IncrementalIngester.findRowIdsForDocIds(IncrementalIngester.java:1518)
>     at
> org.apache.manifoldcf.agents.incrementalingest.IncrementalIngester.documentRemoveMultiple(IncrementalIngester.java:1377)
>     at
> org.apache.manifoldcf.agents.incrementalingest.IncrementalIngester.documentRemove(IncrementalIngester.java:803)
>     at
> org.apache.manifoldcf.crawler.system.WorkerThread$ProcessActivity.removeDocument(WorkerThread.java:1674)
>     at
> com.example.mcf.TestConnector.processDocuments(TestConnector.java:278)
>     at
> org.apache.manifoldcf.crawler.connectors.BaseRepositoryConnector.processDocuments(BaseRepositoryConnector.java:670)
>     at
> org.apache.manifoldcf.crawler.connectors.BaseRepositoryConnector.processDocuments(BaseRepositoryConnector.java:649)
>     at
> org.apache.manifoldcf.crawler.connectors.BaseRepositoryConnector.processDocuments(BaseRepositoryConnector.java:402)
>     at
> org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:380)
> Caused by: java.sql.SQLException: You need to set exactly 3 parameters on
> the prepared statement
>     at
> org.mariadb.jdbc.internal.SQLExceptionMapper.get(SQLExceptionMapper.java:149)
>     at
> org.mariadb.jdbc.internal.SQLExceptionMapper.throwException(SQLExceptionMapper.java:106)
>     at
> org.mariadb.jdbc.MySQLStatement.executeQueryEpilog(MySQLStatement.java:264)
>     at org.mariadb.jdbc.MySQLStatement.execute(MySQLStatement.java:288)
>     at
> org.mariadb.jdbc.MySQLStatement.executeQuery(MySQLStatement.java:302)
>     at
> org.mariadb.jdbc.MySQLPreparedStatement.executeQuery(MySQLPreparedStatement.java:112)
>     at
> org.apache.manifoldcf.core.database.Database.execute(Database.java:880)
>     at
> org.apache.manifoldcf.core.database.Database$ExecuteQueryThread.run(Database.java:683)
> Caused by: org.mariadb.jdbc.internal.common.QueryException: You need to
> set exactly 3 parameters on the prepared statement
>     at
> org.mariadb.jdbc.internal.common.query.MySQLParameterizedQuery.validate(MySQLParameterizedQuery.java:117)
>     at
> org.mariadb.jdbc.internal.mysql.MySQLProtocol.executeQuery(MySQLProtocol.java:976)
>     at org.mariadb.jdbc.MySQLStatement.execute(MySQLStatement.java:281)
>
> Regards,
> Markus
>

Re: Document components

Posted by Markus Schuch <ma...@web.de>.
Hi Karl,
 
the patch for CONNECTORS-1111 fixes the cleanup issue.
 
Another question about primary documents and their components:
 
I have ingested a primary document with some components.
During the next processing the primary document should no longer be indexed, but the sub components of it should still be indexed.
 
My understanding is, that not mentioned components are automatically removed.
Since the primary document is the "null" component, i expected the framework would remove the primary document component if not mentioned, too.
 
But this is not the case. Is this another bug or do i have to remove the primary document somehow manually?

There is an activity method removeDocument(identifier) which seems related.
But i do not fully understand the described usage scenario in the method's javadoc.

I tried the method. The result was the following database exception:
(Patches for CONNECTORS-1110 and CONNECTORS-1111 are applied)
 
2014-11-25 08:30:07,868 ERROR [Worker thread '1'] org.apache.manifoldcf.crawlerthreads: Worker thread aborting and restarting due to database connection reset: Database exception: SQLException doing query (HY0000): You need to set exactly 3 parameters on the prepared statement
org.apache.manifoldcf.core.interfaces.ManifoldCFException: Database exception: SQLException doing query (HY0000): You need to set exactly 3 parameters on the prepared statement
    at org.apache.manifoldcf.core.database.Database$ExecuteQueryThread.finishUp(Database.java:702)
    at org.apache.manifoldcf.core.database.Database.executeViaThread(Database.java:728)
    at org.apache.manifoldcf.core.database.Database.executeUncachedQuery(Database.java:762)
    at org.apache.manifoldcf.core.database.Database$QueryCacheExecutor.create(Database.java:1435)
    at org.apache.manifoldcf.core.cachemanager.CacheManager.findObjectsAndExecute(CacheManager.java:146)
    at org.apache.manifoldcf.core.database.Database.executeQuery(Database.java:191)
    at org.apache.manifoldcf.core.database.DBInterfaceMySQL.performQuery(DBInterfaceMySQL.java:875)
    at org.apache.manifoldcf.core.database.BaseTable.performQuery(BaseTable.java:221)
    at org.apache.manifoldcf.agents.incrementalingest.IncrementalIngester.findRowIdsForDocIds(IncrementalIngester.java:1518)
    at org.apache.manifoldcf.agents.incrementalingest.IncrementalIngester.documentRemoveMultiple(IncrementalIngester.java:1377)
    at org.apache.manifoldcf.agents.incrementalingest.IncrementalIngester.documentRemove(IncrementalIngester.java:803)
    at org.apache.manifoldcf.crawler.system.WorkerThread$ProcessActivity.removeDocument(WorkerThread.java:1674)
    at com.example.mcf.TestConnector.processDocuments(TestConnector.java:278)
    at org.apache.manifoldcf.crawler.connectors.BaseRepositoryConnector.processDocuments(BaseRepositoryConnector.java:670)
    at org.apache.manifoldcf.crawler.connectors.BaseRepositoryConnector.processDocuments(BaseRepositoryConnector.java:649)
    at org.apache.manifoldcf.crawler.connectors.BaseRepositoryConnector.processDocuments(BaseRepositoryConnector.java:402)
    at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:380)
Caused by: java.sql.SQLException: You need to set exactly 3 parameters on the prepared statement
    at org.mariadb.jdbc.internal.SQLExceptionMapper.get(SQLExceptionMapper.java:149)
    at org.mariadb.jdbc.internal.SQLExceptionMapper.throwException(SQLExceptionMapper.java:106)
    at org.mariadb.jdbc.MySQLStatement.executeQueryEpilog(MySQLStatement.java:264)
    at org.mariadb.jdbc.MySQLStatement.execute(MySQLStatement.java:288)
    at org.mariadb.jdbc.MySQLStatement.executeQuery(MySQLStatement.java:302)
    at org.mariadb.jdbc.MySQLPreparedStatement.executeQuery(MySQLPreparedStatement.java:112)
    at org.apache.manifoldcf.core.database.Database.execute(Database.java:880)
    at org.apache.manifoldcf.core.database.Database$ExecuteQueryThread.run(Database.java:683)
Caused by: org.mariadb.jdbc.internal.common.QueryException: You need to set exactly 3 parameters on the prepared statement
    at org.mariadb.jdbc.internal.common.query.MySQLParameterizedQuery.validate(MySQLParameterizedQuery.java:117)
    at org.mariadb.jdbc.internal.mysql.MySQLProtocol.executeQuery(MySQLProtocol.java:976)
    at org.mariadb.jdbc.MySQLStatement.execute(MySQLStatement.java:281)
 
Regards,
Markus

Re: Document components

Posted by Karl Wright <da...@gmail.com>.
Hi Markus,

You can mix and match, yes.  The non-component version of the method is
equivalent to using a component ID of null.

I've created CONNECTORS-1111 for the cleanup issue.

Thanks,
Karl


On Fri, Nov 21, 2014 at 9:20 AM, Markus Schuch <ma...@web.de> wrote:

> Hi Karl,
>
> as i already documented in CONNECTORS-1110, the provided patch solved the
> problem.
>
> Another question about an implementation detail when using components:
> Is it admissible to mix calls of ingestDocumentWithException(...) with and
> without the component identifier argument?
>
> E.g. call ingestDocumentWithException without component identifier for the
> "main" document and then call ingestDocumentWithException with component
> identifiers for all its subcomponents?
>
> Another issue:
> When deleting a job, the cleanup process only removes the first component
> of a document from the output connection.
> In my example only "testidentifier1/comp1" is cleaned up.
>
> Regards,
> Markus
>
>

Re: Document components

Posted by Markus Schuch <ma...@web.de>.
Hi Karl,
 
as i already documented in CONNECTORS-1110, the provided patch solved the problem.
 
Another question about an implementation detail when using components:
Is it admissible to mix calls of ingestDocumentWithException(...) with and without the component identifier argument?

E.g. call ingestDocumentWithException without component identifier for the "main" document and then call ingestDocumentWithException with component identifiers for all its subcomponents?

Another issue:
When deleting a job, the cleanup process only removes the first component of a document from the output connection.
In my example only "testidentifier1/comp1" is cleaned up.
 
Regards,
Markus
 

Re: Document components

Posted by Karl Wright <da...@gmail.com>.
Hi Markus,

Your example looks correct.  I suspect there may be a bug.  I'll open a
ticket.  CONNECTORS-1110.


Karl


On Fri, Nov 21, 2014 at 7:59 AM, Markus Schuch <ma...@web.de> wrote:

> Hi,
>
> is there any example implementation of the new document component feature
> invented with CONNECTORS-989?
>
> I read the section Document components in [0] but i still do not know how
> to actually write a repository connector
> that ingests multiple documents originating from a single document of a
> repository.
>
> My first guess was to call the method
> "activities.ingestDocumentWithException" multiple times with the same
> identifier but distinct component identifiers during document processing.
>
> I wrote a simple TestConnector:
> The processDocuments method looks like:
>
>     public void processDocuments(String[] documentIdentifiers,
>             String[] versions, IProcessActivity activities,
>             DocumentSpecification spec, boolean[] scanOnly)
>             throws ManifoldCFException, ServiceInterruption {
>
>         int i = 0;
>         for (String identifier : documentIdentifiers) {
>
>             byte[] content1 = "test content 1".getBytes();
>             byte[] content2 = "test content 2".getBytes();
>             byte[] content3 = "test content 3".getBytes();
>
>
>             RepositoryDocument rd1 = new RepositoryDocument();
>             rd1.setBinary(new ByteArrayInputStream(content1),
> content1.length);
>
>             RepositoryDocument rd2 = new RepositoryDocument();
>             rd2.setBinary(new ByteArrayInputStream(content2),
> content2.length);
>
>
>             RepositoryDocument rd3 = new RepositoryDocument();
>             rd3.setBinary(new ByteArrayInputStream(content3),
> content3.length);
>
>
>             System.out.println("process " + identifier);
>
>             try {
>                 activities.ingestDocumentWithException(identifier,
> "comp1", versions[i], identifier+"/comp1", rd1);
>                 activities.ingestDocumentWithException(identifier,
> "comp2", versions[i], identifier+"/comp2", rd2);
>                 activities.ingestDocumentWithException(identifier,
> "comp3", versions[i], identifier+"/comp3", rd3);
>             } catch (IOException e) {
>                 e.printStackTrace();
>             }
>
>             i++;
>         }
>
>     }
>
> For seeding the method getDocumentIdentifiers() returns a stream with a
> single document identifier "testidentifier1".
> Full Code available at [1].
>
> But subsequent calls of ingestDocumentWithException result in deletions of
> a previously added component.
>
> job end 1416573910333(copmtest1) 0      1
> document ingest         testidentifier1/comp3 OK 12     8
> document deletion       testidentifier1/comp1 OK 0      2
> document ingest         testidentifier1/comp2 OK 12     10
> document deletion       testidentifier1/comp1 OK 0      3
> document ingest         testidentifier1/comp1 OK 12     13
> job start       1416573910333(copmtest1) 0      1
>
> Only testidentifier1/comp2 and testidentifier1/comp3 exist in the output
> connection after the job is finished.
>
> I feed i might have a false understanding of the concept...
>
> Any help is appreciated.
>
> Thanks in advance
> Markus
>
> --
> Using ManifoldCF 1.7.1
> [0]
> http://manifoldcf.apache.org/release/release-1.7.2/en_US/writing-repository-connectors.html
> [1] https://gist.github.com/schuch/43809594ad8f81ddc625
>