svn commit: r1712176 - in /comdev/reporter.apache.org/trunk: mailglomper2.py scripts/errtee.py scripts/mailglomper2.py scripts/urlutils.py urlutils.py

[se...@apache.org]

Author: sebb
Date: Mon Nov  2 22:58:12 2015
New Revision: 1712176

URL: http://svn.apache.org/viewvc?rev=1712176&view=rev
Log:
Move mailglomper to scripts

Added:
    comdev/reporter.apache.org/trunk/scripts/errtee.py   (with props)
    comdev/reporter.apache.org/trunk/scripts/mailglomper2.py
      - copied, changed from r1712089, comdev/reporter.apache.org/trunk/mailglomper2.py
    comdev/reporter.apache.org/trunk/scripts/urlutils.py
      - copied unchanged from r1711863, comdev/reporter.apache.org/trunk/urlutils.py
Removed:
    comdev/reporter.apache.org/trunk/mailglomper2.py
    comdev/reporter.apache.org/trunk/urlutils.py

Added: comdev/reporter.apache.org/trunk/scripts/errtee.py
URL: http://svn.apache.org/viewvc/comdev/reporter.apache.org/trunk/scripts/errtee.py?rev=1712176&view=auto
==============================================================================
--- comdev/reporter.apache.org/trunk/scripts/errtee.py (added)
+++ comdev/reporter.apache.org/trunk/scripts/errtee.py Mon Nov  2 22:58:12 2015
@@ -0,0 +1,36 @@
+"""
+    Class to copy stderr to stdout.
+
+    This is intended for use in cronjobs,
+    where it is useful to have both stderr and stdout in the log file
+    but still have stderr so that the mailer gets any errors
+    The main script should be run unbuffered (-u)
+
+    Unfortunately Python does not seem to offer the equivalent of:
+    perl -[mM][-]module  execute "use/no module..." before executing program
+    ruby -rlibrary       require the library before executing your script
+    lua  -l name         require library 'name'
+    Thus the module has to be explicitly imported by the user script,
+    and the user must defing the 'ERRTEE' environment variable to enable it
+  
+"""
+import sys
+import os
+
+class ErrTee(object):
+    def write(self, buf):
+        sys.__stderr__.write(buf)
+        sys.stdout.write(buf)
+    def flush(self):
+        sys.__stderr__.flush()
+
+# only enable the override if it is needed
+if 'ERRTEE' in os.environ:
+    sys.stderr=ErrTee()
+
+if __name__ == '__main__': # simple test
+    print("STDOUT1")
+    print("STDERR2", file=sys.stderr)
+    sys.stderr=ErrTee() # enable for testing
+    print("STDERR3", file=sys.stderr)
+    raise Exception("STDERR4")
\ No newline at end of file

Propchange: comdev/reporter.apache.org/trunk/scripts/errtee.py
------------------------------------------------------------------------------
    svn:eol-style = native

Copied: comdev/reporter.apache.org/trunk/scripts/mailglomper2.py (from r1712089, comdev/reporter.apache.org/trunk/mailglomper2.py)
URL: http://svn.apache.org/viewvc/comdev/reporter.apache.org/trunk/scripts/mailglomper2.py?p2=comdev/reporter.apache.org/trunk/scripts/mailglomper2.py&p1=comdev/reporter.apache.org/trunk/mailglomper2.py&r1=1712089&r2=1712176&rev=1712176&view=diff
==============================================================================
--- comdev/reporter.apache.org/trunk/mailglomper2.py (original)
+++ comdev/reporter.apache.org/trunk/scripts/mailglomper2.py Mon Nov  2 22:58:12 2015
@@ -17,11 +17,19 @@ from datetime import datetime
 import urlutils
 import urllib.error
 import traceback
-import data.errtee
+import errtee
 
 SECS_PER_DAY = 86400
 SECS_PER_WEEK = 604800
 
+# We assume that the script is run from the scripts subdirectory
+
+__RAO_HOME = "../"
+
+__MAILDATA_EXTENDED = __RAO_HOME + "data/maildata_extended.json"
+
+__MAILDATA_CACHE    = __RAO_HOME + "data/cache/maildata_weekly.json"
+
 def tsprint(s): # print with timestamp
     msg = "%s %s" % (time.strftime("%Y-%m-%d %H:%M:%S", time.localtime()), s)
     if isinstance(s, Exception):
@@ -49,10 +57,6 @@ def handle(signum, frame):
 
 tsprint("Start")
 
-__MAILDATA_EXTENDED = "data/maildata_extended2.json" # TODO change to normal name
-
-__MAILDATA_CACHE    = "data/cache/maildata_weekly.json"
-
 try:
     with open(__MAILDATA_EXTENDED,'r') as f:
         mls = json.loads(f.read())