[kibble-scanners] branch master updated: Move to using cloc 1.76 as minimum

[hu...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

humbedooh pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/kibble-scanners.git


The following commit(s) were added to refs/heads/master by this push:
     new 5b6f9c7  Move to using cloc 1.76 as minimum
5b6f9c7 is described below

commit 5b6f9c7fc53ec83fb0a8e03695ebf2db1b8e0be6
Author: Daniel Gruno <hu...@apache.org>
AuthorDate: Wed Jan 9 09:23:09 2019 +0100

    Move to using cloc 1.76 as minimum
    
    - Change minimum required version of cloc for counting to 1.76
    - Start utilizing multiprocessing for cloc
    - Fix Readme to reflect this change, note that it's optional
---
 README.md                 | 11 ++---------
 src/plugins/utils/sloc.py |  6 +++++-
 2 files changed, 7 insertions(+), 10 deletions(-)

diff --git a/README.md b/README.md
index dfdeccd..6797af7 100644
--- a/README.md
+++ b/README.md
@@ -65,19 +65,12 @@ The Kibble Scanners collect information for the Kibble Suite.
 
 ## Requirements:
 
- - [cloc](https://github.com/AlDanial/cloc) version 1.70 or later
+ - [cloc](https://github.com/AlDanial/cloc) version 1.76 or later `(optional)`
  - git binaries
  - python3 (3.3 or later)
  - python3-elasticsearch 
  - python3-dateutils
- 
-### Multi-threaded CLoC counter
- To speed up things, you can use the multi-threaded version of cloc
- found at [this location](https://raw.githubusercontent.com/stsnel/cloc/multithread/cloc).
- On machines with many cores, you may experience a tenfold speed increase in
- the SLoC and evolution scans.
- It requires ForkManager, which may be installed via CPAN:
- `cpan Parallel::ForkManager`
+
  
 # Get involved
   TBD. Please see https://kibble.apache.org/ for details!
diff --git a/src/plugins/utils/sloc.py b/src/plugins/utils/sloc.py
index 25f1a5b..38c0e7d 100644
--- a/src/plugins/utils/sloc.py
+++ b/src/plugins/utils/sloc.py
@@ -22,10 +22,14 @@ import sys
 import subprocess
 import time
 import re
+import multiprocessing
 
 def count(path):
     """ Count lines of Code """
-    inp = subprocess.check_output("cloc --quiet --progress-rate=0 %s" % path, shell = True).decode('ascii', 'replace')
+    # We determine how many cores there are, and adjust the
+    # process count based on that. Max 4 procs.
+    my_core_count = min((4, int( multiprocessing.cpu_count() )))
+    inp = subprocess.check_output("cloc --quiet --progress-rate=0 --processes=%u %s" % (my_core_count, path), shell = True).decode('ascii', 'replace')
     m = re.search(r".*Language\s+files\s+blank\s+comment\s+code[\s\S]+?-+([\s\S]+?)-+[\s\S]+?SUM:\s+(\d+)\s+(\d+)\s+(\d+)\s+(\d+)", inp, flags=re.MULTILINE|re.UNICODE)
     languages = {}
     ccount = 0