You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@ponymail.apache.org by hu...@apache.org on 2020/09/10 08:27:58 UTC
[incubator-ponymail-foal] branch master updated (3438dd5 -> 0140cae)
This is an automated email from the ASF dual-hosted git repository.
humbedooh pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-ponymail-foal.git.
from 3438dd5 7.8.1 has url_prefix bugs in it, switch to 7.9.1
new 2fe14be Switch to threaded pool, let Python decide on worker count
new cc61e64 Exceptions may be thrown when finally fetching result()
new 2f81480 debug print if exception
new 83e42b1 let Python decide on number of runners
new 0140cae bump ES module
The 5 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
requirements.txt | 2 +-
server/main.py | 2 +-
server/plugins/offloader.py | 13 ++++++++++---
3 files changed, 12 insertions(+), 5 deletions(-)
[incubator-ponymail-foal] 02/05: Exceptions may be thrown when
finally fetching result()
Posted by hu...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
humbedooh pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-ponymail-foal.git
commit cc61e64465df3f0ca487b76af10af12d59da5842
Author: Daniel Gruno <hu...@apache.org>
AuthorDate: Thu Sep 10 10:23:59 2020 +0200
Exceptions may be thrown when finally fetching result()
---
server/plugins/offloader.py | 5 ++++-
1 file changed, 4 insertions(+), 1 deletion(-)
diff --git a/server/plugins/offloader.py b/server/plugins/offloader.py
index bd38a0e..0ee9823 100644
--- a/server/plugins/offloader.py
+++ b/server/plugins/offloader.py
@@ -37,7 +37,10 @@ class ExecutorPool:
print("[Runner] Waiting for task %r to finish" % func)
while runner.running():
await asyncio.sleep(0.01)
- rv = runner.result()
+ try:
+ rv = runner.result()
+ except Exception as e:
+ rv = e
if DEBUG:
print("[Runner] Done with task %r, put runner back in queue" % func)
if isinstance(rv, BaseException):
[incubator-ponymail-foal] 03/05: debug print if exception
Posted by hu...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
humbedooh pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-ponymail-foal.git
commit 2f814806d09964ca3d8a0157e72ed14bfde08797
Author: Daniel Gruno <hu...@apache.org>
AuthorDate: Thu Sep 10 10:26:21 2020 +0200
debug print if exception
---
server/plugins/offloader.py | 2 ++
1 file changed, 2 insertions(+)
diff --git a/server/plugins/offloader.py b/server/plugins/offloader.py
index 0ee9823..0f6fea1 100644
--- a/server/plugins/offloader.py
+++ b/server/plugins/offloader.py
@@ -44,5 +44,7 @@ class ExecutorPool:
if DEBUG:
print("[Runner] Done with task %r, put runner back in queue" % func)
if isinstance(rv, BaseException):
+ if DEBUG:
+ print("[Runner] Task %r encountered an exception during run." % func)
raise rv
return rv
[incubator-ponymail-foal] 05/05: bump ES module
Posted by hu...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
humbedooh pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-ponymail-foal.git
commit 0140cae8859f75383c03645abbac125cec165a6f
Author: Daniel Gruno <hu...@apache.org>
AuthorDate: Thu Sep 10 10:27:42 2020 +0200
bump ES module
Lots of bugfixes in 7.9.1, so wouldn't hurt to bump it.
---
requirements.txt | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/requirements.txt b/requirements.txt
index cd8d391..f2440c5 100644
--- a/requirements.txt
+++ b/requirements.txt
@@ -1,6 +1,6 @@
PyYAML~=5.3.1
elasticsearch-dsl>=7.0.0,<8.0.0
-elasticsearch~=7.8.1
+elasticsearch~=7.9.1
certifi~=2020.6.20
chardet~=3.0.4
netaddr~=0.8.0
[incubator-ponymail-foal] 04/05: let Python decide on number of
runners
Posted by hu...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
humbedooh pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-ponymail-foal.git
commit 83e42b1fdc3a546b74721ef73b27737344f2bb69
Author: Daniel Gruno <hu...@apache.org>
AuthorDate: Thu Sep 10 10:26:46 2020 +0200
let Python decide on number of runners
---
server/main.py | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/server/main.py b/server/main.py
index 3ffb88b..45c64ec 100644
--- a/server/main.py
+++ b/server/main.py
@@ -52,7 +52,7 @@ class Server(plugins.server.BaseServer):
self.data = plugins.configuration.InterData()
self.handlers = dict()
self.dbpool = asyncio.Queue()
- self.runners = plugins.offloader.ExecutorPool(threads=10)
+ self.runners = plugins.offloader.ExecutorPool()
self.server = None
# Make a pool of 15 database connections for async queries
[incubator-ponymail-foal] 01/05: Switch to threaded pool,
let Python decide on worker count
Posted by hu...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
humbedooh pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-ponymail-foal.git
commit 2fe14bef3e3b9093500a8a2c0f860013b3328ace
Author: Daniel Gruno <hu...@apache.org>
AuthorDate: Thu Sep 10 10:22:41 2020 +0200
Switch to threaded pool, let Python decide on worker count
---
server/plugins/offloader.py | 6 ++++--
1 file changed, 4 insertions(+), 2 deletions(-)
diff --git a/server/plugins/offloader.py b/server/plugins/offloader.py
index 4a6b95e..bd38a0e 100644
--- a/server/plugins/offloader.py
+++ b/server/plugins/offloader.py
@@ -24,8 +24,10 @@ DEBUG = False
class ExecutorPool:
- def __init__(self, threads=10):
- self.threads = concurrent.futures.ProcessPoolExecutor(max_workers=threads)
+ """A pool of runners for offloading blocking processes to threads, so that async processing can continue"""
+ def __init__(self, threads=None):
+ # If no thread count is specified, will default to: min(32, os.cpu_count() + 4)
+ self.threads = concurrent.futures.ThreadPoolExecutor(max_workers=threads)
async def run(self, func, *args, **kwargs):
if DEBUG: