[incubator-ponymail-foal] branch master updated: Use os.cpu_count to find a good parallel ops count

[hu...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

humbedooh pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-ponymail-foal.git


The following commit(s) were added to refs/heads/master by this push:
     new cab6e5b  Use os.cpu_count to find a good parallel ops count
cab6e5b is described below

commit cab6e5b354d2af546475cd83313931868f7af730
Author: Daniel Gruno <hu...@apache.org>
AuthorDate: Mon Sep 6 15:28:52 2021 -0500

    Use os.cpu_count to find a good parallel ops count
    
    if more than one core present, use around 75% of cores at most, but never more than cores-1.
---
 tools/migrate.py | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)

diff --git a/tools/migrate.py b/tools/migrate.py
index d3cd792..ff295f0 100644
--- a/tools/migrate.py
+++ b/tools/migrate.py
@@ -31,12 +31,14 @@ import base64
 import hashlib
 import multiprocessing
 import time
+import os
 
 # Increment this number whenever breaking changes happen in the migration workflow:
 MIGRATION_MAGIC_NUMBER = "2"
 
-# Max number of parallel conversions to perform before pushing
-MAX_PARALLEL_OPS = 12
+# Max number of parallel conversions to perform before pushing. 75-ish percent of max cores.
+cores = os.cpu_count()
+MAX_PARALLEL_OPS = max(min(int((cores + 1) * 0.75), cores-1), 1)
 
 
 class MultiDocProcessor:
@@ -205,6 +207,7 @@ def process_attachment(old_es, doc, dbname_attachment):
 async def main():
     print("Welcome to the Apache Pony Mail -> Foal migrator.")
     print("This will copy your old database, adjust the structure, and insert the emails into your new foal database.")
+    print("We will be utilizing %u cores for this operation." % MAX_PARALLEL_OPS)
     print("------------------------------------")
     old_es_url = (
         input("Enter the full URL (including http/https) of your old ES server: ") or "http://localhost:9200/"