You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@manifoldcf.apache.org by Shinichiro Abe <sh...@gmail.com> on 2013/10/08 08:24:45 UTC

SSL Basic Authentication

Hi,
I'm sure that Web Connector supports Basic Authentication and can crawl http sites,
but I'm not sure about the case that https SSL site with https basic Authentication.
I can register basic auth with https:// regex, user and password, but crawling failed. 
Does this support https basic Authentication?
Now I watch the log, the logs shows "WEB: Decided not to ingest 
'https://server.com/url/' because it did not match ingestability criteria".
What does it mean about this message?

Regards,
Shinichiro Abe

Re: SSL Basic Authentication

Posted by Shinichiro Abe <sh...@gmail.com>.

Thanks Karl,

I'm guessing (1).
Seeing logs, at first 401 response returned then passed auth and tried to fetch document.
After that, below message is shown.
I'll try to debug isDataIngestable().

Thank you,
Shinichiro Abe

On 2013/10/08, at 15:42, Karl Wright <da...@gmail.com> wrote:

> Hi Abe-san,
> 
> This means that the method:
> 
>   protected boolean isDataIngestable(IFingerprintActivity activities, String documentIdentifier, DocumentURLFilter filter)
>     throws ServiceInterruption, ManifoldCFException
> 
> ... has returned false.  If you look at the method, you will see that it checks the following:
> 
> (1) That the response code is 200 (could be this if the login is failing, because then 401 is returned).
> (2) That the output connector will accept documents of that length.
> (3) Whether the output connector will accept the URL as given.
> (4) Whether the url matches the urls allowed for indexing in the web job.
> (5) If there is no mime type at all (which I think is not correct; it should probably ask the output connector even in this case).
> (6) Whether the output connector will accept the document's mime type.
> 
> In most cases, the method logs its decision, so you may see additional output that could clarify why the document is being excluded.  If no additional message is being output, then it is either case (1) or (5).  You would have to add logging code to figure out which one it is.
> 
> Thanks,
> Karl
> 
> 
> 
> On Tue, Oct 8, 2013 at 2:24 AM, Shinichiro Abe <sh...@gmail.com> wrote:
> Hi,
> I'm sure that Web Connector supports Basic Authentication and can crawl http sites,
> but I'm not sure about the case that https SSL site with https basic Authentication.
> I can register basic auth with https:// regex, user and password, but crawling failed.
> Does this support https basic Authentication?
> Now I watch the log, the logs shows "WEB: Decided not to ingest
> 'https://server.com/url/' because it did not match ingestability criteria".
> What does it mean about this message?
> 
> Regards,
> Shinichiro Abe
>

Re: SSL Basic Authentication

Posted by Karl Wright <da...@gmail.com>.

Hi Abe-san,

This means that the method:

  protected boolean isDataIngestable(IFingerprintActivity activities,
String documentIdentifier, DocumentURLFilter filter)
    throws ServiceInterruption, ManifoldCFException

... has returned false.  If you look at the method, you will see that it
checks the following:

(1) That the response code is 200 (could be this if the login is failing,
because then 401 is returned).
(2) That the output connector will accept documents of that length.
(3) Whether the output connector will accept the URL as given.
(4) Whether the url matches the urls allowed for indexing in the web job.
(5) If there is no mime type at all (which I think is not correct; it
should probably ask the output connector even in this case).
(6) Whether the output connector will accept the document's mime type.

In most cases, the method logs its decision, so you may see additional
output that could clarify why the document is being excluded.  If no
additional message is being output, then it is either case (1) or (5).  You
would have to add logging code to figure out which one it is.

Thanks,
Karl

On Tue, Oct 8, 2013 at 2:24 AM, Shinichiro Abe
<sh...@gmail.com>wrote:

> Hi,
> I'm sure that Web Connector supports Basic Authentication and can crawl
> http sites,
> but I'm not sure about the case that https SSL site with https basic
> Authentication.
> I can register basic auth with https:// regex, user and password, but
> crawling failed.
> Does this support https basic Authentication?
> Now I watch the log, the logs shows "WEB: Decided not to ingest
> 'https://server.com/url/' because it did not match ingestability
> criteria".
> What does it mean about this message?
>
> Regards,
> Shinichiro Abe