You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@ponymail.apache.org by hu...@apache.org on 2020/09/07 20:04:28 UTC
[incubator-ponymail-foal] branch master updated (1124f96 -> 38f7459)
This is an automated email from the ASF dual-hosted git repository.
humbedooh pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-ponymail-foal.git.
from 1124f96 Typo
new 04a17a6 Add wrappers for delete and index
new a1b4e77 Implement sticky sessions across restarts
new f2cace0 Expand comments
new 7b6665e Conform to new session plugin standards, lint.
new eda22fa some OAuths use 'fullname'
new 812ae8f spit out creds if available.
new a0c85c7 Implement logout feature
new a85a6c1 credentials may be None, so this is a better (working) check.
new 83b5836 tweak some names to match code
new 38f7459 Merge branch 'master' of github.com:apache/incubator-ponymail-foal
The 10 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
server/endpoints/oauth.py | 20 ++++--
server/endpoints/preferences.py | 21 +++++-
server/plugins/aaa.py | 2 +-
server/plugins/database.py | 12 ++++
server/plugins/session.py | 155 +++++++++++++++++++++++++++++++++-------
tools/mappings.yaml | 8 +--
6 files changed, 177 insertions(+), 41 deletions(-)
[incubator-ponymail-foal] 01/10: Add wrappers for delete and index
Posted by hu...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
humbedooh pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-ponymail-foal.git
commit 04a17a69780a403627ecfcd2694f3ce8559e5b4d
Author: Daniel Gruno <hu...@apache.org>
AuthorDate: Mon Sep 7 21:55:07 2020 +0200
Add wrappers for delete and index
---
server/plugins/database.py | 12 ++++++++++++
1 file changed, 12 insertions(+)
diff --git a/server/plugins/database.py b/server/plugins/database.py
index 0739ee7..910073a 100644
--- a/server/plugins/database.py
+++ b/server/plugins/database.py
@@ -71,3 +71,15 @@ class Database:
index = self.dbs.mbox
res = await self.client.get(index=index, **kwargs)
return res
+
+ async def delete(self, index="", **kwargs):
+ if not index:
+ index = self.dbs.session
+ res = await self.client.delete(index=index, **kwargs)
+ return res
+
+ async def index(self, index="", **kwargs):
+ if not index:
+ index = self.dbs.session
+ res = await self.client.index(index=index, **kwargs)
+ return res
[incubator-ponymail-foal] 06/10: spit out creds if available.
Posted by hu...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
humbedooh pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-ponymail-foal.git
commit 812ae8f88ebb664441ff7729e98d6bf649c5a8db
Author: Daniel Gruno <hu...@apache.org>
AuthorDate: Mon Sep 7 22:01:23 2020 +0200
spit out creds if available.
---
server/endpoints/preferences.py | 11 ++++++++++-
1 file changed, 10 insertions(+), 1 deletion(-)
diff --git a/server/endpoints/preferences.py b/server/endpoints/preferences.py
index bfd7e3b..9e6a965 100644
--- a/server/endpoints/preferences.py
+++ b/server/endpoints/preferences.py
@@ -17,13 +17,14 @@
import plugins.server
import plugins.aaa
+import plugins.session
""" Generic preferences endpoint for Pony Mail codename Foal"""
""" This is incomplete, but will work for anonymous tests. """
async def process(
- server: plugins.server.BaseServer, session: dict, indata: dict
+ server: plugins.server.BaseServer, session: plugins.session.SessionObject, indata: dict
) -> dict:
prefs = {"login": {}}
lists = {}
@@ -38,6 +39,14 @@ async def process(
lists[ldomain] = {}
lists[ldomain][lname] = entry["count"]
prefs["lists"] = lists
+ if session and session.credentials:
+ prefs['login'] = {
+ "credentials": {
+ "uid": session.credentials.uid,
+ "email": session.credentials.email,
+ "fullname": session.credentials.name,
+ }
+ }
return prefs
[incubator-ponymail-foal] 03/10: Expand comments
Posted by hu...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
humbedooh pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-ponymail-foal.git
commit f2cace0371a7249c58df42b2c01e94c18ce97e44
Author: Daniel Gruno <hu...@apache.org>
AuthorDate: Mon Sep 7 21:58:38 2020 +0200
Expand comments
---
server/plugins/session.py | 6 ++++--
1 file changed, 4 insertions(+), 2 deletions(-)
diff --git a/server/plugins/session.py b/server/plugins/session.py
index 81d7295..aae98b7 100644
--- a/server/plugins/session.py
+++ b/server/plugins/session.py
@@ -169,14 +169,16 @@ async def set_session(server: plugins.server.BaseServer, cid, **credentials):
session = SessionObject(server, last_accessed=time.time(), cookie=session_id, cid=cid)
session.credentials = SessionCredentials(credentials)
server.data.sessions[session_id] = session
- # Grab temporary DB handle
+
+ # Grab temporary DB handle since session objects at init do not have this
+ # We just need this to be able to save the session in ES.
session.database = await server.dbpool.get()
# Save session and account data
await save_session(session)
await save_credentials(session)
- # Put DB handle back
+ # Put DB handle back into the pool
server.dbpool.put_nowait(session.database)
return cookie["ponymail"].OutputString()
[incubator-ponymail-foal] 04/10: Conform to new session plugin
standards, lint.
Posted by hu...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
humbedooh pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-ponymail-foal.git
commit 7b6665e20c64e089b8f19290a48da043eaee92be
Author: Daniel Gruno <hu...@apache.org>
AuthorDate: Mon Sep 7 22:00:29 2020 +0200
Conform to new session plugin standards, lint.
---
server/endpoints/oauth.py | 18 +++++++++++++-----
1 file changed, 13 insertions(+), 5 deletions(-)
diff --git a/server/endpoints/oauth.py b/server/endpoints/oauth.py
index b6d06c0..cf67773 100644
--- a/server/endpoints/oauth.py
+++ b/server/endpoints/oauth.py
@@ -22,6 +22,7 @@ import plugins.session
import plugins.oauthGeneric
import typing
import aiohttp.web
+import hashlib
async def process(
@@ -43,22 +44,29 @@ async def process(
if not uid:
uid = rv.get("email")
if uid:
+ cid = hashlib.shake_128(
+ ("%s-%s" % (rv.get("oauth_domain", "generic"), uid)).encode(
+ "ascii", "ignore"
+ )
+ ).hexdigest(16)
cookie = await plugins.session.set_session(
server,
+ cid,
uid=uid,
name=rv.get("name"),
email=rv.get("email"),
# Authoritative if OAuth domain is in the authoritative oauth section in ponymail.yaml
# Required for access to private emails
- authoritative=rv.get('oauth_domain', 'generic') in server.config.oauth.authoritative_domains,
+ authoritative=rv.get("oauth_domain", "generic")
+ in server.config.oauth.authoritative_domains,
+ oauth_provider=rv.get("oauth_domain", "generic"),
oauth_data=rv,
)
# This could be improved upon, instead of a raw response return value
return aiohttp.web.Response(
- headers={
- 'set-cookie': cookie,
- 'content-type': 'application/json'
- }, status=200, text='{"okay": true}'
+ headers={"set-cookie": cookie, "content-type": "application/json"},
+ status=200,
+ text='{"okay": true}',
)
[incubator-ponymail-foal] 09/10: tweak some names to match code
Posted by hu...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
humbedooh pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-ponymail-foal.git
commit 83b5836da8279200b49cecfd285c525333438479
Author: Daniel Gruno <hu...@apache.org>
AuthorDate: Mon Sep 7 22:03:44 2020 +0200
tweak some names to match code
---
tools/mappings.yaml | 8 ++------
1 file changed, 2 insertions(+), 6 deletions(-)
diff --git a/tools/mappings.yaml b/tools/mappings.yaml
index bd2b7b5..8828396 100644
--- a/tools/mappings.yaml
+++ b/tools/mappings.yaml
@@ -11,17 +11,13 @@ account:
type: object
email:
type: keyword
- fullname:
+ name:
type: keyword
uid:
type: keyword
internal:
properties:
- cookie:
- type: keyword
- ip:
- type: keyword
- oauth_used:
+ oauth_provider:
type: keyword
oauth_data:
type: object
[incubator-ponymail-foal] 08/10: credentials may be None,
so this is a better (working) check.
Posted by hu...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
humbedooh pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-ponymail-foal.git
commit a85a6c178408a703fd0e47617a0021afce73237f
Author: Daniel Gruno <hu...@apache.org>
AuthorDate: Mon Sep 7 22:03:19 2020 +0200
credentials may be None, so this is a better (working) check.
---
server/plugins/aaa.py | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/server/plugins/aaa.py b/server/plugins/aaa.py
index 5f23578..c654a4c 100644
--- a/server/plugins/aaa.py
+++ b/server/plugins/aaa.py
@@ -40,7 +40,7 @@ def can_access_email(session: plugins.session.SessionObject, email) -> bool:
def can_access_list(session: plugins.session.SessionObject, listid) -> bool:
"""Determine if a list can be accessed by the current user"""
# If logged in via a known oauth, we assume access for now...TO BE CHANGED
- if session and session.credentials.authoritative:
+ if session.credentials and session.credentials.authoritative:
return True
else:
return False
[incubator-ponymail-foal] 02/10: Implement sticky sessions across
restarts
Posted by hu...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
humbedooh pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-ponymail-foal.git
commit a1b4e77b70987a3cd4a1e10a1a2829a0a1824dbf
Author: Daniel Gruno <hu...@apache.org>
AuthorDate: Mon Sep 7 21:57:35 2020 +0200
Implement sticky sessions across restarts
---
server/plugins/session.py | 155 +++++++++++++++++++++++++++++++++++++---------
1 file changed, 127 insertions(+), 28 deletions(-)
diff --git a/server/plugins/session.py b/server/plugins/session.py
index 6c6f870..81d7295 100644
--- a/server/plugins/session.py
+++ b/server/plugins/session.py
@@ -28,7 +28,8 @@ import plugins.database
import plugins.server
import copy
-PYPONY_MAX_SESSION_AGE = 86400 * 7 # Max 1 week between visits before voiding a session
+FOAL_MAX_SESSION_AGE = 86400 * 7 # Max 1 week between visits before voiding a session
+FOAL_SAVE_SESSION_INTERVAL = 3600 # Update sessions on disk max once per hour
class SessionCredentials:
@@ -40,52 +41,55 @@ class SessionCredentials:
admin: bool
oauth_data: dict
- def __init__(self, doc: typing.Dict = None):
+ def __init__(self, doc):
if doc:
- self.uid = doc.get('uid', '')
- self.name = doc.get('name', '')
- self.email = doc.get('email', '')
- self.provider = doc.get('provider', 'generic')
- self.authoritative = doc.get('authoritative', False)
- self.admin = doc.get('admin', False)
- self.oauth_data = doc.get('oauth_data', {})
+ self.uid = doc.get("uid", "")
+ self.name = doc.get("name", "")
+ self.email = doc.get("email", "")
+ self.oauth_provider = doc.get("oauth_provider", "generic")
+ self.authoritative = doc.get("authoritative", False)
+ self.admin = doc.get("admin", False)
+ self.oauth_data = doc.get("oauth_data", {})
else:
self.uid = ""
self.name = ""
self.email = ""
- self.provider = "generic"
+ self.oauth_provider = "generic"
self.authoritative = False
self.admin = False
self.oauth_data = {}
class SessionObject:
- uid: str
+ cid: str
+ cookie: str
created: int
last_accessed: int
credentials: SessionCredentials
database: typing.Optional[plugins.database.Database]
- def __init__(self, server: plugins.server.BaseServer, doc=None):
+ def __init__(self, server: plugins.server.BaseServer, **kwargs):
self.database = None
- if not doc:
+ if not kwargs:
now = int(time.time())
self.created = now
self.last_accessed = now
- self.credentials = SessionCredentials()
- self.uid = str(uuid.uuid4())
+ self.credentials = None
+ self.cookie = str(uuid.uuid4())
+ self.cid = None
else:
- self.created = doc["created"]
- self.last_accessed = doc["last_accessed"]
- self.credentials = SessionCredentials(doc["credentials"])
- self.uid = doc["uid"]
+ self.last_accessed = kwargs.get("last_accessed")
+ self.credentials = SessionCredentials(kwargs.get("credentials"))
+ self.cookie = kwargs.get("cookie")
+ self.cid = kwargs.get("cid")
async def get_session(
server: plugins.server.BaseServer, request: aiohttp.web.BaseRequest
- ) -> SessionObject:
+) -> SessionObject:
session_id = None
session = None
+ now = int(time.time())
if request.headers.get("cookie"):
for cookie_header in request.headers.getall("cookie"):
cookies: http.cookies.SimpleCookie = http.cookies.SimpleCookie(
@@ -95,29 +99,124 @@ async def get_session(
session_id = cookies["ponymail"].value
break
+ # Do we have the session in local memory?
if session_id in server.data.sessions:
x_session = server.data.sessions[session_id]
- now = int(time.time())
- if (now - x_session.last_accessed) > PYPONY_MAX_SESSION_AGE:
+ if (now - x_session.last_accessed) > FOAL_MAX_SESSION_AGE:
del server.data.sessions[session_id]
else:
+
+ # Do we need to update the timestamp in ES?
+ if (now - x_session.last_accessed) > FOAL_SAVE_SESSION_INTERVAL:
+ x_session.last_accessed = now
+ await save_session(x_session)
+
# Make a copy so we don't have a race condition with the database pool object
# In case the session is used twice within the same loop
session = copy.copy(x_session)
- if not session:
- session = SessionObject(server)
+ session.database = await server.dbpool.get()
+ return session
+
+ # If not in local memory, start a new session object
+ session = SessionObject(server)
session.database = await server.dbpool.get()
+
+ # If a cookie was supplied, look for a session object in ES
+ if session_id:
+ try:
+ session_doc = await session.database.get(
+ session.database.dbs.session, id=session_id
+ )
+ last_update = session_doc["_source"]["updated"]
+ session.cookie = session_id
+ # Check that this cookie ain't too old. If it is, delete it and return bare-bones session object
+ if (now - last_update) > FOAL_MAX_SESSION_AGE:
+ session.database.delete(
+ index=session.database.dbs.session, id=session_id
+ )
+ return session
+
+ # Get CID and fecth the account data
+ cid = session_doc["_source"]["cid"]
+ if cid:
+ account_doc = await session.database.get(session.database.dbs.account, id=cid)
+ creds = account_doc["_source"]['credentials']
+ internal = account_doc['_source']['internal']
+
+ # Set session data
+ session.cid = cid
+ session.last_accessed = last_update
+ creds["authoritative"] = (
+ internal.get("oauth_provider") in server.config.oauth.authoritative_domains
+ )
+ creds['oauth_provider'] = internal.get('oauth_provider', 'generic')
+ creds['oauth_data'] = internal.get('oauth_data', {})
+ session.credentials = SessionCredentials(creds)
+
+ # Save in memory storage
+ server.data.sessions[session_id] = session
+
+ except plugins.database.DBError:
+ pass
return session
-async def set_session(server: plugins.server.BaseServer, **credentials):
+async def set_session(server: plugins.server.BaseServer, cid, **credentials):
"""Create a new user session in the database"""
session_id = str(uuid.uuid4())
cookie: http.cookies.SimpleCookie = http.cookies.SimpleCookie()
cookie["ponymail"] = session_id
- session = SessionObject(server)
+ session = SessionObject(server, last_accessed=time.time(), cookie=session_id, cid=cid)
session.credentials = SessionCredentials(credentials)
server.data.sessions[session_id] = session
- return cookie['ponymail'].OutputString()
-
+ # Grab temporary DB handle
+ session.database = await server.dbpool.get()
+ # Save session and account data
+ await save_session(session)
+ await save_credentials(session)
+
+ # Put DB handle back
+ server.dbpool.put_nowait(session.database)
+ return cookie["ponymail"].OutputString()
+
+
+async def save_session(session: SessionObject):
+ """Save a session object in the ES database"""
+ await session.database.index(
+ index=session.database.dbs.session,
+ id=session.cookie,
+ body={
+ "cookie": session.cookie,
+ "cid": session.cid,
+ "updated": session.last_accessed,
+ },
+ )
+
+
+async def remove_session(session: SessionObject):
+ """Remove a session object in the ES database"""
+ await session.database.delete(
+ index=session.database.dbs.session,
+ id=session.cookie
+ )
+
+
+async def save_credentials(session: SessionObject):
+ """Save a user account object in the ES database"""
+ await session.database.index(
+ index=session.database.dbs.account,
+ id=session.cid,
+ body={
+ "cid": session.cid,
+ "credentials": {
+ "email": session.credentials.email,
+ "name": session.credentials.name,
+ "uid": session.credentials.uid,
+ },
+ "internal": {
+ "oauth_provider": session.credentials.oauth_provider,
+ "oauth_data": session.credentials.oauth_data,
+ }
+ },
+ )
[incubator-ponymail-foal] 07/10: Implement logout feature
Posted by hu...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
humbedooh pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-ponymail-foal.git
commit a0c85c791b46ae93155b714162271a54ba1cf9e8
Author: Daniel Gruno <hu...@apache.org>
AuthorDate: Mon Sep 7 22:02:38 2020 +0200
Implement logout feature
---
server/endpoints/preferences.py | 10 ++++++++++
1 file changed, 10 insertions(+)
diff --git a/server/endpoints/preferences.py b/server/endpoints/preferences.py
index 9e6a965..7908a22 100644
--- a/server/endpoints/preferences.py
+++ b/server/endpoints/preferences.py
@@ -48,6 +48,16 @@ async def process(
}
}
+ # Logging out??
+ if indata.get('logout'):
+ # Remove session from ElasticSearch
+ await plugins.session.remove_session(session)
+
+ # If stored in memory, remove from there.
+ if session.cookie in server.data.sessions:
+ del server.data.sessions[session.cookie]
+ session.credentials = None
+
return prefs
[incubator-ponymail-foal] 05/10: some OAuths use 'fullname'
Posted by hu...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
humbedooh pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-ponymail-foal.git
commit eda22fa89ba964a4db8ccb8649f00e58cab6dfdd
Author: Daniel Gruno <hu...@apache.org>
AuthorDate: Mon Sep 7 22:00:43 2020 +0200
some OAuths use 'fullname'
---
server/endpoints/oauth.py | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/server/endpoints/oauth.py b/server/endpoints/oauth.py
index cf67773..ec7a30c 100644
--- a/server/endpoints/oauth.py
+++ b/server/endpoints/oauth.py
@@ -53,7 +53,7 @@ async def process(
server,
cid,
uid=uid,
- name=rv.get("name"),
+ name=rv.get("name") or rv.get("fullname"),
email=rv.get("email"),
# Authoritative if OAuth domain is in the authoritative oauth section in ponymail.yaml
# Required for access to private emails
[incubator-ponymail-foal] 10/10: Merge branch 'master' of
github.com:apache/incubator-ponymail-foal
Posted by hu...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
humbedooh pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-ponymail-foal.git
commit 38f7459baf67e7db8e9d0c079fb3b5ac274f0dd8
Merge: 83b5836 1124f96
Author: Daniel Gruno <hu...@apache.org>
AuthorDate: Mon Sep 7 22:04:16 2020 +0200
Merge branch 'master' of github.com:apache/incubator-ponymail-foal
server/endpoints/source.py | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)