[DISCUSS] ldap_auth donation

[Jay Doane <ja...@apache.org>]

Greetings,

In 2015 IBM Cloudant developed an LDAP based authentication handler for
its CouchDB 2.x-based Cloudant Local offering. Since then, it has been used
in production on several large Cloudant Local deployments, accruing many
bug fixes and enhancements in the process.

Over the years, there has clearly been interest in using LDAP with CouchDB
[1], so it seems like the ldap_auth functionality might be something the
greater community could benefit from. If there are no objections from the
community or PMC, I'd be happy to open a PR in the hopes of getting it
included in CouchDB 3.2. To give an idea of what it's all about, I've
included the README.md contents below.

Thanks,
Jay

[1] https://couchdb.markmail.org/search/?q=LDAP

README.md:

# Delegating Basic and Cookie authentication and authorization to LDAP

CouchDB includes a built-in security model, with self-contained
authentication and authorization which rely on .ini files and/or user
databases to store user data, including usernames (uids), password
hashes, and user roles for accessing database resources.

For an organization which already uses an LDAP service for access
control, it may make more sense to delegate authentication and
authorization services to LDAP when accessing CouchDB. This is where
`ldap_auth` comes in.

At a high level, when `ldap_auth` is configured, it completely
replaces the default Basic and Cookie authentication and
authorization. It ignores .ini files and user databases altogether,
and attempts to validate user credentials using the configured LDAP
service. Once credentials have been authentication by the LDAP
service, `ldap_auth` determines the user's roles (which in turn
determine what the user is authorized to access) based on the LDAP
groups of which the user is a member.

## ldap_interface

This low-level interface module uses
[eldap](http://www.erlang.org/doc/man/eldap.html) to connect to,
authenticate, and search LDAP server(s) which are configured to model
user accounts and their associated CouchDB roles.

### Configuration

All configuration is done via `.ini` file, mostly in the `[ldap_auth]`
section. The following parameters can be modified, and have the
associated defaults in parentheses:

- `servers (127.0.0.1)` one or more LDAP servers; defaults to a single
host, but could be a comma separated list (note that all servers must
use the same port, a limitation of the underlying eldap library)

- `port (389)` LDAP server port for un-encrypted communication

- `ssl_port (636)` LDAP server port for encrypted communication

- `use_ssl (true)` if `true`, use TLS to encrypt traffic to LDAP
servers

- `timeout (5000)` milliseconds to wait for a response from an LDAP
server before throwing an error

- `user_base_dn (ou=users,dc=example,dc=com)` defines a directory
location to start searching for users

- `user_classes (person)` defines which `objectClass`es indicate
a particular entry as a user during search

- `user_uid_attribute (uid)` defines which attribute maps to
username

- `group_base_dn (ou=groups,dc=example,dc=com)` defines directory
location to start searching for groups

- `group_classes (posixGroup)` defines which `objectClass`es indicate
a particular entry as a group during search

- `group_member_attribute (memberUid)` defines which group attribute
maps to user Uid

- `group_role_attribute (description)` defines which group attribute
maps to a particular role

- `searcher_dn (uid=ldapsearch,ou=users,dc=example,dc=com)` defines
the DN to use when searching for users and groups

- `searcher_password (secret)` defines the password for the
`searcher_dn` above

- `user_bind_dns ([])` defines one or more base DNs into which the
authenticating user's username can be inserted as the
`user_uid_attribute`, and used to bind directly. See the "Efficiency
Considerations" section below for details

**Please note** that at least one of the above parameters must be set
inside the `[ldap_auth]` section of a .ini configuration file in order
for ldap_auth to be considered "configured", and to function as an
authentication/authorization handler. Failure to explicitly set at
least one `[ldap_auth]` parameter will result in the system using the
default basic and cookie authentication/authorization handlers instead.

### Efficiency Considerations

  `ldap_auth` effectively has 2 modes of operation, depending on
whether `user_bind_dns` has been defined or not. Both modes return a
list of roles associated with a user upon authentication success.

  By default, `ldap_auth` opens a connection to an LDAP server and
binds the connection handle with the searcher DN and password. It then
uses that connection to search for a user record with the
authenticating username. If a matching user DN is found, it opens a
second connection to the server and attempts to bind that connection
using the username and password credentials. If the bind is
successful, it closes that connection, and uses the original
"searcher" bound connection to search for groups which contain the
`group_member_attribute` matching the user's username. Those groups
have a `group_role_attribute` which indicates the actual roles for the
user.

  On the other hand, if you know in advance that all your users' DNs
can be constructed by inserting their usernames into the following
pattern: `$uid_attribute=$username,$user_bind_dn`
(e.g. `uid=jay,ou=users,dc=example,dc=com`), then you can configure
one or more `user_bind_dns`, and `ldap_auth` will not make use of the
searcher DN, nor `user_base_dn`, but instead attempt to bind directly
with the constructed user DNs. If the bind is successful, that bound
connection is further used to make the same group search as described
in the above paragraph. Overall, this technique can be used to
eliminate several additional network round trips.

## ldap_auth

  Implements Basic and Cookie based authentication handlers, using
`ldap_interface:authorized_roles(Username, Password)` to obtain the
roles associated with a particular user.

  Using `ldap_auth` with basic authentication requires no client side
changes. A properly configured database will automatically attempt to
authenticate via LDAP using the supplied credentials.

  Similarly, for cookie authentication, POST credentials to the
`_session` endpoint to obtain an AuthSession cookie, which contains a
signed hash of its contents: name, time issued, and authorized
roles. Subsequent requests using that cookie will automatically use
the roles in the cookie until it expires. The following configuration
parameters in the `[couch_httpd_auth]` section are used:

- `timeout (600)` seconds until the AuthSession cookie expires

- `secret` the shared secret used to sign the AuthSession cookie,
it should be created when the cluster is provisioned

Re: [DISCUSS] ldap_auth donation

[Jay Doane <ja...@gmail.com>]

Hi Joan,

Yes, I'm planning to maintain it, and hopefully it's readable enough for
others to do so if necessary. In any case, since current Cloudant Local
customers are being encouraged (with few on-prem alternatives) to migrate
to CouchDB, there should be a built-in user base for as well. This is
definitely not abandonware!

And, yes, I'm still planning to get weatherreport into 3.2 as well. Thanks
for the encouragement :)

Jay

On Tue, Sep 1, 2020 at 1:42 PM Joan Touzet <wo...@apache.org> wrote:

> I remember this code!
>
> Sure, let's get it out there, *as long as* someone is going to maintain
> it going forward.
>
> I had been hoping to see more rallying behind how to use the JWT
> integration for OAuth and SAML workflows, but no one's done any
> walkthroughs / blogposts that I've seen. Putting those things in front
> of LDAP may be easier (but would be outside the scope of CouchDB...)
>
> While we're on the topic of donation, any chance of weatherreport
> getting donated? You had brought this up a few months ago, Jay, and I'd
> love to see that in 3.2 as well.
>
> -Joan
>
>
>
> On 2020-09-01 3:02 p.m., Jay Doane wrote:
> > Greetings,
> >
> > In 2015 IBM Cloudant developed an LDAP based authentication handler for
> > its CouchDB 2.x-based Cloudant Local offering. Since then, it has been
> used
> > in production on several large Cloudant Local deployments, accruing many
> > bug fixes and enhancements in the process.
> >
> > Over the years, there has clearly been interest in using LDAP with
> CouchDB
> > [1], so it seems like the ldap_auth functionality might be something the
> > greater community could benefit from. If there are no objections from the
> > community or PMC, I'd be happy to open a PR in the hopes of getting it
> > included in CouchDB 3.2. To give an idea of what it's all about, I've
> > included the README.md contents below.
> >
> > Thanks,
> > Jay
> >
> > [1] https://couchdb.markmail.org/search/?q=LDAP
> >
> > README.md:
> >
> > # Delegating Basic and Cookie authentication and authorization to LDAP
> >
> > CouchDB includes a built-in security model, with self-contained
> > authentication and authorization which rely on .ini files and/or user
> > databases to store user data, including usernames (uids), password
> > hashes, and user roles for accessing database resources.
> >
> > For an organization which already uses an LDAP service for access
> > control, it may make more sense to delegate authentication and
> > authorization services to LDAP when accessing CouchDB. This is where
> > `ldap_auth` comes in.
> >
> > At a high level, when `ldap_auth` is configured, it completely
> > replaces the default Basic and Cookie authentication and
> > authorization. It ignores .ini files and user databases altogether,
> > and attempts to validate user credentials using the configured LDAP
> > service. Once credentials have been authentication by the LDAP
> > service, `ldap_auth` determines the user's roles (which in turn
> > determine what the user is authorized to access) based on the LDAP
> > groups of which the user is a member.
> >
> > ## ldap_interface
> >
> > This low-level interface module uses
> > [eldap](http://www.erlang.org/doc/man/eldap.html) to connect to,
> > authenticate, and search LDAP server(s) which are configured to model
> > user accounts and their associated CouchDB roles.
> >
> > ### Configuration
> >
> > All configuration is done via `.ini` file, mostly in the `[ldap_auth]`
> > section. The following parameters can be modified, and have the
> > associated defaults in parentheses:
> >
> > - `servers (127.0.0.1)` one or more LDAP servers; defaults to a single
> > host, but could be a comma separated list (note that all servers must
> > use the same port, a limitation of the underlying eldap library)
> >
> > - `port (389)` LDAP server port for un-encrypted communication
> >
> > - `ssl_port (636)` LDAP server port for encrypted communication
> >
> > - `use_ssl (true)` if `true`, use TLS to encrypt traffic to LDAP
> > servers
> >
> > - `timeout (5000)` milliseconds to wait for a response from an LDAP
> > server before throwing an error
> >
> > - `user_base_dn (ou=users,dc=example,dc=com)` defines a directory
> > location to start searching for users
> >
> > - `user_classes (person)` defines which `objectClass`es indicate
> > a particular entry as a user during search
> >
> > - `user_uid_attribute (uid)` defines which attribute maps to
> > username
> >
> > - `group_base_dn (ou=groups,dc=example,dc=com)` defines directory
> > location to start searching for groups
> >
> > - `group_classes (posixGroup)` defines which `objectClass`es indicate
> > a particular entry as a group during search
> >
> > - `group_member_attribute (memberUid)` defines which group attribute
> > maps to user Uid
> >
> > - `group_role_attribute (description)` defines which group attribute
> > maps to a particular role
> >
> > - `searcher_dn (uid=ldapsearch,ou=users,dc=example,dc=com)` defines
> > the DN to use when searching for users and groups
> >
> > - `searcher_password (secret)` defines the password for the
> > `searcher_dn` above
> >
> > - `user_bind_dns ([])` defines one or more base DNs into which the
> > authenticating user's username can be inserted as the
> > `user_uid_attribute`, and used to bind directly. See the "Efficiency
> > Considerations" section below for details
> >
> > **Please note** that at least one of the above parameters must be set
> > inside the `[ldap_auth]` section of a .ini configuration file in order
> > for ldap_auth to be considered "configured", and to function as an
> > authentication/authorization handler. Failure to explicitly set at
> > least one `[ldap_auth]` parameter will result in the system using the
> > default basic and cookie authentication/authorization handlers instead.
> >
> > ### Efficiency Considerations
> >
> >    `ldap_auth` effectively has 2 modes of operation, depending on
> > whether `user_bind_dns` has been defined or not. Both modes return a
> > list of roles associated with a user upon authentication success.
> >
> >    By default, `ldap_auth` opens a connection to an LDAP server and
> > binds the connection handle with the searcher DN and password. It then
> > uses that connection to search for a user record with the
> > authenticating username. If a matching user DN is found, it opens a
> > second connection to the server and attempts to bind that connection
> > using the username and password credentials. If the bind is
> > successful, it closes that connection, and uses the original
> > "searcher" bound connection to search for groups which contain the
> > `group_member_attribute` matching the user's username. Those groups
> > have a `group_role_attribute` which indicates the actual roles for the
> > user.
> >
> >    On the other hand, if you know in advance that all your users' DNs
> > can be constructed by inserting their usernames into the following
> > pattern: `$uid_attribute=$username,$user_bind_dn`
> > (e.g. `uid=jay,ou=users,dc=example,dc=com`), then you can configure
> > one or more `user_bind_dns`, and `ldap_auth` will not make use of the
> > searcher DN, nor `user_base_dn`, but instead attempt to bind directly
> > with the constructed user DNs. If the bind is successful, that bound
> > connection is further used to make the same group search as described
> > in the above paragraph. Overall, this technique can be used to
> > eliminate several additional network round trips.
> >
> > ## ldap_auth
> >
> >    Implements Basic and Cookie based authentication handlers, using
> > `ldap_interface:authorized_roles(Username, Password)` to obtain the
> > roles associated with a particular user.
> >
> >    Using `ldap_auth` with basic authentication requires no client side
> > changes. A properly configured database will automatically attempt to
> > authenticate via LDAP using the supplied credentials.
> >
> >    Similarly, for cookie authentication, POST credentials to the
> > `_session` endpoint to obtain an AuthSession cookie, which contains a
> > signed hash of its contents: name, time issued, and authorized
> > roles. Subsequent requests using that cookie will automatically use
> > the roles in the cookie until it expires. The following configuration
> > parameters in the `[couch_httpd_auth]` section are used:
> >
> > - `timeout (600)` seconds until the AuthSession cookie expires
> >
> > - `secret` the shared secret used to sign the AuthSession cookie,
> > it should be created when the cluster is provisioned
> >
>

Re: [DISCUSS] ldap_auth donation

[Joan Touzet <wo...@apache.org>]

I remember this code!

Sure, let's get it out there, *as long as* someone is going to maintain 
it going forward.

I had been hoping to see more rallying behind how to use the JWT 
integration for OAuth and SAML workflows, but no one's done any 
walkthroughs / blogposts that I've seen. Putting those things in front 
of LDAP may be easier (but would be outside the scope of CouchDB...)

While we're on the topic of donation, any chance of weatherreport 
getting donated? You had brought this up a few months ago, Jay, and I'd 
love to see that in 3.2 as well.

-Joan



On 2020-09-01 3:02 p.m., Jay Doane wrote:
> Greetings,
> 
> In 2015 IBM Cloudant developed an LDAP based authentication handler for
> its CouchDB 2.x-based Cloudant Local offering. Since then, it has been used
> in production on several large Cloudant Local deployments, accruing many
> bug fixes and enhancements in the process.
> 
> Over the years, there has clearly been interest in using LDAP with CouchDB
> [1], so it seems like the ldap_auth functionality might be something the
> greater community could benefit from. If there are no objections from the
> community or PMC, I'd be happy to open a PR in the hopes of getting it
> included in CouchDB 3.2. To give an idea of what it's all about, I've
> included the README.md contents below.
> 
> Thanks,
> Jay
> 
> [1] https://couchdb.markmail.org/search/?q=LDAP
> 
> README.md:
> 
> # Delegating Basic and Cookie authentication and authorization to LDAP
> 
> CouchDB includes a built-in security model, with self-contained
> authentication and authorization which rely on .ini files and/or user
> databases to store user data, including usernames (uids), password
> hashes, and user roles for accessing database resources.
> 
> For an organization which already uses an LDAP service for access
> control, it may make more sense to delegate authentication and
> authorization services to LDAP when accessing CouchDB. This is where
> `ldap_auth` comes in.
> 
> At a high level, when `ldap_auth` is configured, it completely
> replaces the default Basic and Cookie authentication and
> authorization. It ignores .ini files and user databases altogether,
> and attempts to validate user credentials using the configured LDAP
> service. Once credentials have been authentication by the LDAP
> service, `ldap_auth` determines the user's roles (which in turn
> determine what the user is authorized to access) based on the LDAP
> groups of which the user is a member.
> 
> ## ldap_interface
> 
> This low-level interface module uses
> [eldap](http://www.erlang.org/doc/man/eldap.html) to connect to,
> authenticate, and search LDAP server(s) which are configured to model
> user accounts and their associated CouchDB roles.
> 
> ### Configuration
> 
> All configuration is done via `.ini` file, mostly in the `[ldap_auth]`
> section. The following parameters can be modified, and have the
> associated defaults in parentheses:
> 
> - `servers (127.0.0.1)` one or more LDAP servers; defaults to a single
> host, but could be a comma separated list (note that all servers must
> use the same port, a limitation of the underlying eldap library)
> 
> - `port (389)` LDAP server port for un-encrypted communication
> 
> - `ssl_port (636)` LDAP server port for encrypted communication
> 
> - `use_ssl (true)` if `true`, use TLS to encrypt traffic to LDAP
> servers
> 
> - `timeout (5000)` milliseconds to wait for a response from an LDAP
> server before throwing an error
> 
> - `user_base_dn (ou=users,dc=example,dc=com)` defines a directory
> location to start searching for users
> 
> - `user_classes (person)` defines which `objectClass`es indicate
> a particular entry as a user during search
> 
> - `user_uid_attribute (uid)` defines which attribute maps to
> username
> 
> - `group_base_dn (ou=groups,dc=example,dc=com)` defines directory
> location to start searching for groups
> 
> - `group_classes (posixGroup)` defines which `objectClass`es indicate
> a particular entry as a group during search
> 
> - `group_member_attribute (memberUid)` defines which group attribute
> maps to user Uid
> 
> - `group_role_attribute (description)` defines which group attribute
> maps to a particular role
> 
> - `searcher_dn (uid=ldapsearch,ou=users,dc=example,dc=com)` defines
> the DN to use when searching for users and groups
> 
> - `searcher_password (secret)` defines the password for the
> `searcher_dn` above
> 
> - `user_bind_dns ([])` defines one or more base DNs into which the
> authenticating user's username can be inserted as the
> `user_uid_attribute`, and used to bind directly. See the "Efficiency
> Considerations" section below for details
> 
> **Please note** that at least one of the above parameters must be set
> inside the `[ldap_auth]` section of a .ini configuration file in order
> for ldap_auth to be considered "configured", and to function as an
> authentication/authorization handler. Failure to explicitly set at
> least one `[ldap_auth]` parameter will result in the system using the
> default basic and cookie authentication/authorization handlers instead.
> 
> ### Efficiency Considerations
> 
>    `ldap_auth` effectively has 2 modes of operation, depending on
> whether `user_bind_dns` has been defined or not. Both modes return a
> list of roles associated with a user upon authentication success.
> 
>    By default, `ldap_auth` opens a connection to an LDAP server and
> binds the connection handle with the searcher DN and password. It then
> uses that connection to search for a user record with the
> authenticating username. If a matching user DN is found, it opens a
> second connection to the server and attempts to bind that connection
> using the username and password credentials. If the bind is
> successful, it closes that connection, and uses the original
> "searcher" bound connection to search for groups which contain the
> `group_member_attribute` matching the user's username. Those groups
> have a `group_role_attribute` which indicates the actual roles for the
> user.
> 
>    On the other hand, if you know in advance that all your users' DNs
> can be constructed by inserting their usernames into the following
> pattern: `$uid_attribute=$username,$user_bind_dn`
> (e.g. `uid=jay,ou=users,dc=example,dc=com`), then you can configure
> one or more `user_bind_dns`, and `ldap_auth` will not make use of the
> searcher DN, nor `user_base_dn`, but instead attempt to bind directly
> with the constructed user DNs. If the bind is successful, that bound
> connection is further used to make the same group search as described
> in the above paragraph. Overall, this technique can be used to
> eliminate several additional network round trips.
> 
> ## ldap_auth
> 
>    Implements Basic and Cookie based authentication handlers, using
> `ldap_interface:authorized_roles(Username, Password)` to obtain the
> roles associated with a particular user.
> 
>    Using `ldap_auth` with basic authentication requires no client side
> changes. A properly configured database will automatically attempt to
> authenticate via LDAP using the supplied credentials.
> 
>    Similarly, for cookie authentication, POST credentials to the
> `_session` endpoint to obtain an AuthSession cookie, which contains a
> signed hash of its contents: name, time issued, and authorized
> roles. Subsequent requests using that cookie will automatically use
> the roles in the cookie until it expires. The following configuration
> parameters in the `[couch_httpd_auth]` section are used:
> 
> - `timeout (600)` seconds until the AuthSession cookie expires
> 
> - `secret` the shared secret used to sign the AuthSession cookie,
> it should be created when the cluster is provisioned
>