[jira] [Created] (CONNECTORS-1598) session based authentication cannot register 401

["roel goovaerts (JIRA)" <ji...@apache.org>]

roel goovaerts created CONNECTORS-1598:
------------------------------------------

             Summary: session based authentication cannot register 401
                 Key: CONNECTORS-1598
                 URL: https://issues.apache.org/jira/browse/CONNECTORS-1598
             Project: ManifoldCF
          Issue Type: Bug
          Components: Web connector
    Affects Versions: ManifoldCF 2.12
            Reporter: roel goovaerts


Description:

Access to a specific domain is restricted by being A) an intranet service B) based on an employee/costumer profile.
For manifold to be able to be authenticated there is a specific '\{{domain}}/login' page with a form where manifold was configured to enter it's username and password. A session-cookie is then set so manifold is authenticated to access all resources. If a request for a resource is not authenticated the service throws a 401. When the service returns a 401 the actual content of the resource includes the same form as is present in '\{{domain}}/login'.

Problem:

The only way we have been able to configure manifold to be authenticated was by specifying session-based credentials AND providing '\{{domain}}/login' as a seed in the job as well. The only other seed in the job is a sitemap.
This is of course not ideal since it can easily happen that the seed for the sitemap gets processed first, which then throws a 401 on the sitemap and the job stops.
Another possible scenario with this configuration is that the cookie expires and all other resources throw 401 and get deleted from the index (elasticsearch). There is also another job (different language, same domain), usage of the cookie from the previous job has also been registered.

Current session-based access credentials configuration:

--url regular expression : https://\{{domain}}/
--login pages: 
---login url regexp : 'login'
---page type : form
---identification regexp is set to match the form-name
---form parameters are filled with the correct parameters

This is verified to work, but as my understanding this only works because the login-page is part of the seeds and so it matches the url when it comes across it when crawling. There is no configuration yet which redirects (for example) to this page when manifold receives a 401.

My goal was then to remove the login-page from the seeds and configure the job so that each time a fetch returns a 401, manifold knows to go to the login page. in pseudo code:

--If authenticated
---process 
--else
---redirect to login
---retry resource

 

Based on the documentation here: https://manifoldcf.apache.org/release/release-2.12/en_US/end-user-documentation.html#webrepository I tried a few different configurations. The first thing to notice is in the comparison table, 'page based authentication' only mentions 4xx and 'session based authentication' only mentions 3xx.

At this time my biggest question is; are these response codes bound to the difference in settings between page and session based? As far I have been able to see, whenever manifold receives a 401 it logs "ignoring url \{‌{url}‌} because it failed to fetch (status=401, ..."
Am I not able to work with session based authentication when the service returns 401's?

 

Configuration attempts (all failed):
- for all attempts the login page was removed from the seeds.
- in general I have kept the above configuration of page type 'form', in the case I was able to redirect manifold to this page.
- The kinds of content that a web connection can recognize as a login page specified in the documentation lists an option "A page that has specific content on it, as described by a regular expression". As the description of this case specified I tried the page type 'content' setting, with identification regexp set to '.*' for testing and an override url set to '\{{domain}}/login'. My hopes were that in this test the match-all-regexp would override to the login page for every url it fetches.
- Since the content of a 401 also includes the same form as the login page, i tried with page type 'form', supplied identification regexp en override form parameters, just like above, only with the "login url regexp" set to '.*'. My hopes were that each page has the possibility to have the form recognized if it is returned as a 401.

In both cases the only thing I could see is that manifold fetched the sitemap, received a 401 and in manifold logged "ignoring url \{‌{url}‌} because it failed to fetch (status=401, ..."

Some questions:
- Is there anything to be done when manifold receives a 401?
- is 4xx tied to page base authentication and 3xx tied to session based authentication?
- is there some other configuration/logic that I am missing, that I could try out?



A minimal effort solution would be if there was a way to make manifold start at the login and not do any crawling (most importanly no deleting) when it is unable to be authenticated. Together with this a way to remove the session cookie when the job is done would also be necessary, so as to avoid the expiry of the cookie as a result of manifold using an old cookie.

Side-note; is there any way to make manifold not delete documents when it receives a 401?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)