[jira] [Updated] (CONNECTORS-1547) No activity record for for excluded documents in WebCrawlerConnector

["Karl Wright (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/CONNECTORS-1547?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Karl Wright updated CONNECTORS-1547:
------------------------------------
    Fix Version/s: ManifoldCF 2.12

> No activity record for for excluded documents in WebCrawlerConnector
> --------------------------------------------------------------------
>
>                 Key: CONNECTORS-1547
>                 URL: https://issues.apache.org/jira/browse/CONNECTORS-1547
>             Project: ManifoldCF
>          Issue Type: Bug
>          Components: Web connector
>            Reporter: Olivier Tavard
>            Assignee: Karl Wright
>            Priority: Minor
>             Fix For: ManifoldCF 2.12
>
>         Attachments: manifoldcf_local_files.log, manifoldcf_web.log, simple_history_files.jpg, simple_history_web.jpg
>
>
> Hi,
> I noticed that there is no activity record logged for documents excluded by the Document Filter transformation connector  in the WebCrawler connector.
> To reproduce the issue on MCF out of the box :
> Null output connector 
> Web repository connector 
> Job :
> - DocumentFilter added which only accepts application/msword (doc/docx) documents
> The simple history does not mention the documents excluded (excepted for html documents). They have fetch activity and that's all (see simple_history_web.jpeg).
> We can only see the documents excluded by the MCF log (with DEBUG verbosity activity on connectors) :
> {code:java}
> Removing url 'https://www.datafari.com/assets/img/Logo_Datafari_4_Condensed_No_D_20180606_30x30.png' because it had the wrong content type ('image/png'){code}
> (see manifoldcf_local_files.log)
> The related code is in WebcrawlerConnector.java l.904 :
> {code:java}
> fetchStatus.contextMessage = "it had the wrong content type ('"+contentType+"')";
>  fetchStatus.resultSignal = RESULT_NO_DOCUMENT;
>  activityResultCode = null;{code}
> The activityResultCode is null.
>  
>  
> If we configure the same job but for a Local File system connector with the same Document Filter transformation connector, the simple history mentions all the documents excluded in the simple history (see simple_history_files.jpeg)  and the code mentions a specific error code with an activity record logged (class FileConnector l. 415) : 
> {code:java}
> if (!activities.checkMimeTypeIndexable(mimeType))
>  {
>  errorCode = activities.EXCLUDED_MIMETYPE;
>  errorDesc = "Excluded because mime type ('"+mimeType+"')";
>  Logging.connectors.debug("Skipping file '"+documentIdentifier+"' because mime type ('"+mimeType+"') was excluded by output connector.");
>  activities.noDocument(documentIdentifier,versionString);
>  continue;
>  }{code}
>  
> So the Web Crawler connector should have the same behaviour than for FileConnector and explicitly mention all the documents excluded by the user I think.
>  
> Best regards,
> Olivier



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)