You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@warble.apache.org by hu...@apache.org on 2018/06/12 14:11:11 UTC
[incubator-warble-node] branch master created (now 3bf383f)
This is an automated email from the ASF dual-hosted git repository.
humbedooh pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-warble-node.git.
at 3bf383f add a brief readme
This branch includes the following new commits:
new eb42553 Initial checkout of base design
new cfa7a25 add gitignore
new 8765257 remove test (moved to unittests.py)
new 0d5d510 add generic tcp test class
new 3bf383f add a brief readme
The 5 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
--
To stop receiving notification emails like this one, please contact
humbedooh@apache.org.
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@warble.apache.org
For additional commands, e-mail: commits-help@warble.apache.org
[incubator-warble-node] 01/05: Initial checkout of base design
Posted by hu...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
humbedooh pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-warble-node.git
commit eb425537ccd11772163550b2befd69ff47fdec52
Author: Daniel Gruno <hu...@apache.org>
AuthorDate: Tue Jun 12 09:01:37 2018 -0500
Initial checkout of base design
includes two parred tests and a unit test, not much else right now.
---
LICENSE | 201 ++++++++++++++++++++++++++++++++++++++++++++
NOTICE | 2 +
conf/node.yaml.sample | 11 +++
node.py | 98 +++++++++++++++++++++
plugins/basics/__init__.py | 8 ++
plugins/basics/misc.py | 88 +++++++++++++++++++
plugins/basics/socket.py | 148 ++++++++++++++++++++++++++++++++
plugins/basics/unittests.py | 81 ++++++++++++++++++
plugins/reports/__init__.py | 5 ++
plugins/reports/generic.py | 66 +++++++++++++++
plugins/tests/__init__.py | 8 ++
plugins/tests/http.py | 176 ++++++++++++++++++++++++++++++++++++++
plugins/tests/smtp.py | 67 +++++++++++++++
requirements.txt | 5 ++
14 files changed, 964 insertions(+)
diff --git a/LICENSE b/LICENSE
new file mode 100644
index 0000000..ad410e1
--- /dev/null
+++ b/LICENSE
@@ -0,0 +1,201 @@
+Apache License
+ Version 2.0, January 2004
+ http://www.apache.org/licenses/
+
+ TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+ 1. Definitions.
+
+ "License" shall mean the terms and conditions for use, reproduction,
+ and distribution as defined by Sections 1 through 9 of this document.
+
+ "Licensor" shall mean the copyright owner or entity authorized by
+ the copyright owner that is granting the License.
+
+ "Legal Entity" shall mean the union of the acting entity and all
+ other entities that control, are controlled by, or are under common
+ control with that entity. For the purposes of this definition,
+ "control" means (i) the power, direct or indirect, to cause the
+ direction or management of such entity, whether by contract or
+ otherwise, or (ii) ownership of fifty percent (50%) or more of the
+ outstanding shares, or (iii) beneficial ownership of such entity.
+
+ "You" (or "Your") shall mean an individual or Legal Entity
+ exercising permissions granted by this License.
+
+ "Source" form shall mean the preferred form for making modifications,
+ including but not limited to software source code, documentation
+ source, and configuration files.
+
+ "Object" form shall mean any form resulting from mechanical
+ transformation or translation of a Source form, including but
+ not limited to compiled object code, generated documentation,
+ and conversions to other media types.
+
+ "Work" shall mean the work of authorship, whether in Source or
+ Object form, made available under the License, as indicated by a
+ copyright notice that is included in or attached to the work
+ (an example is provided in the Appendix below).
+
+ "Derivative Works" shall mean any work, whether in Source or Object
+ form, that is based on (or derived from) the Work and for which the
+ editorial revisions, annotations, elaborations, or other modifications
+ represent, as a whole, an original work of authorship. For the purposes
+ of this License, Derivative Works shall not include works that remain
+ separable from, or merely link (or bind by name) to the interfaces of,
+ the Work and Derivative Works thereof.
+
+ "Contribution" shall mean any work of authorship, including
+ the original version of the Work and any modifications or additions
+ to that Work or Derivative Works thereof, that is intentionally
+ submitted to Licensor for inclusion in the Work by the copyright owner
+ or by an individual or Legal Entity authorized to submit on behalf of
+ the copyright owner. For the purposes of this definition, "submitted"
+ means any form of electronic, verbal, or written communication sent
+ to the Licensor or its representatives, including but not limited to
+ communication on electronic mailing lists, source code control systems,
+ and issue tracking systems that are managed by, or on behalf of, the
+ Licensor for the purpose of discussing and improving the Work, but
+ excluding communication that is conspicuously marked or otherwise
+ designated in writing by the copyright owner as "Not a Contribution."
+
+ "Contributor" shall mean Licensor and any individual or Legal Entity
+ on behalf of whom a Contribution has been received by Licensor and
+ subsequently incorporated within the Work.
+
+ 2. Grant of Copyright License. Subject to the terms and conditions of
+ this License, each Contributor hereby grants to You a perpetual,
+ worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+ copyright license to reproduce, prepare Derivative Works of,
+ publicly display, publicly perform, sublicense, and distribute the
+ Work and such Derivative Works in Source or Object form.
+
+ 3. Grant of Patent License. Subject to the terms and conditions of
+ this License, each Contributor hereby grants to You a perpetual,
+ worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+ (except as stated in this section) patent license to make, have made,
+ use, offer to sell, sell, import, and otherwise transfer the Work,
+ where such license applies only to those patent claims licensable
+ by such Contributor that are necessarily infringed by their
+ Contribution(s) alone or by combination of their Contribution(s)
+ with the Work to which such Contribution(s) was submitted. If You
+ institute patent litigation against any entity (including a
+ cross-claim or counterclaim in a lawsuit) alleging that the Work
+ or a Contribution incorporated within the Work constitutes direct
+ or contributory patent infringement, then any patent licenses
+ granted to You under this License for that Work shall terminate
+ as of the date such litigation is filed.
+
+ 4. Redistribution. You may reproduce and distribute copies of the
+ Work or Derivative Works thereof in any medium, with or without
+ modifications, and in Source or Object form, provided that You
+ meet the following conditions:
+
+ (a) You must give any other recipients of the Work or
+ Derivative Works a copy of this License; and
+
+ (b) You must cause any modified files to carry prominent notices
+ stating that You changed the files; and
+
+ (c) You must retain, in the Source form of any Derivative Works
+ that You distribute, all copyright, patent, trademark, and
+ attribution notices from the Source form of the Work,
+ excluding those notices that do not pertain to any part of
+ the Derivative Works; and
+
+ (d) If the Work includes a "NOTICE" text file as part of its
+ distribution, then any Derivative Works that You distribute must
+ include a readable copy of the attribution notices contained
+ within such NOTICE file, excluding those notices that do not
+ pertain to any part of the Derivative Works, in at least one
+ of the following places: within a NOTICE text file distributed
+ as part of the Derivative Works; within the Source form or
+ documentation, if provided along with the Derivative Works; or,
+ within a display generated by the Derivative Works, if and
+ wherever such third-party notices normally appear. The contents
+ of the NOTICE file are for informational purposes only and
+ do not modify the License. You may add Your own attribution
+ notices within Derivative Works that You distribute, alongside
+ or as an addendum to the NOTICE text from the Work, provided
+ that such additional attribution notices cannot be construed
+ as modifying the License.
+
+ You may add Your own copyright statement to Your modifications and
+ may provide additional or different license terms and conditions
+ for use, reproduction, or distribution of Your modifications, or
+ for any such Derivative Works as a whole, provided Your use,
+ reproduction, and distribution of the Work otherwise complies with
+ the conditions stated in this License.
+
+ 5. Submission of Contributions. Unless You explicitly state otherwise,
+ any Contribution intentionally submitted for inclusion in the Work
+ by You to the Licensor shall be under the terms and conditions of
+ this License, without any additional terms or conditions.
+ Notwithstanding the above, nothing herein shall supersede or modify
+ the terms of any separate license agreement you may have executed
+ with Licensor regarding such Contributions.
+
+ 6. Trademarks. This License does not grant permission to use the trade
+ names, trademarks, service marks, or product names of the Licensor,
+ except as required for reasonable and customary use in describing the
+ origin of the Work and reproducing the content of the NOTICE file.
+
+ 7. Disclaimer of Warranty. Unless required by applicable law or
+ agreed to in writing, Licensor provides the Work (and each
+ Contributor provides its Contributions) on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+ implied, including, without limitation, any warranties or conditions
+ of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+ PARTICULAR PURPOSE. You are solely responsible for determining the
+ appropriateness of using or redistributing the Work and assume any
+ risks associated with Your exercise of permissions under this License.
+
+ 8. Limitation of Liability. In no event and under no legal theory,
+ whether in tort (including negligence), contract, or otherwise,
+ unless required by applicable law (such as deliberate and grossly
+ negligent acts) or agreed to in writing, shall any Contributor be
+ liable to You for damages, including any direct, indirect, special,
+ incidental, or consequential damages of any character arising as a
+ result of this License or out of the use or inability to use the
+ Work (including but not limited to damages for loss of goodwill,
+ work stoppage, computer failure or malfunction, or any and all
+ other commercial damages or losses), even if such Contributor
+ has been advised of the possibility of such damages.
+
+ 9. Accepting Warranty or Additional Liability. While redistributing
+ the Work or Derivative Works thereof, You may choose to offer,
+ and charge a fee for, acceptance of support, warranty, indemnity,
+ or other liability obligations and/or rights consistent with this
+ License. However, in accepting such obligations, You may act only
+ on Your own behalf and on Your sole responsibility, not on behalf
+ of any other Contributor, and only if You agree to indemnify,
+ defend, and hold each Contributor harmless for any liability
+ incurred by, or claims asserted against, such Contributor by reason
+ of your accepting any such warranty or additional liability.
+
+ END OF TERMS AND CONDITIONS
+
+ APPENDIX: How to apply the Apache License to your work.
+
+ To apply the Apache License to your work, attach the following
+ boilerplate notice, with the fields enclosed by brackets "{}"
+ replaced with your own identifying information. (Don't include
+ the brackets!) The text should be enclosed in the appropriate
+ comment syntax for the file format. We also recommend that a
+ file or class name and description of purpose be included on the
+ same "printed page" as the copyright notice for easier
+ identification within third-party archives.
+
+ Copyright {yyyy} {name of copyright owner}
+
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
\ No newline at end of file
diff --git a/NOTICE b/NOTICE
new file mode 100644
index 0000000..5a43b3a
--- /dev/null
+++ b/NOTICE
@@ -0,0 +1,2 @@
+Apache Warble (incubating) Monitoring Node Package
+Copyright 2018 The Apache Software Foundation and individual contributors
diff --git a/conf/node.yaml.sample b/conf/node.yaml.sample
new file mode 100644
index 0000000..ab95cbd
--- /dev/null
+++ b/conf/node.yaml.sample
@@ -0,0 +1,11 @@
+client:
+ # Warble Master hostname
+ server: demo.warble.apache.org
+ # APP ID and Key for node to access master data (and for master to know who this is)
+ # This typically gets set by the program after talking to the master.
+ appid: UNSET
+ appkey: foobar
+
+misc:
+ # NTP server or pool for adjusting time inside the node.
+ ntpserver: pool.ntp.org
diff --git a/node.py b/node.py
new file mode 100644
index 0000000..93d5399
--- /dev/null
+++ b/node.py
@@ -0,0 +1,98 @@
+#!/usr/bin/env python3.4
+# -*- coding: utf-8 -*-
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+ #the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""
+This is the main node script for Apache Warble (incubating)
+"""
+_VERSION = '0.1.0'
+
+# Basic imports
+import os
+import sys
+import ruamel.yaml
+import requests
+import datetime
+import argparse
+
+# Warble-specific libraries
+import plugins.tests
+import plugins.basics.misc
+
+basepath = os.path.dirname(os.path.realpath(__file__))
+configpath = "%s/conf/node.yaml" % basepath
+
+if __name__ == "__main__":
+
+ parser = argparse.ArgumentParser(description = "Run-time configuration options for Apache Warble (incubating)")
+ parser.add_argument('--version', action = 'store_true', help = 'Print node version and exit')
+ parser.add_argument('--test', action = 'store_true', help = 'Run debug unit tests')
+ parser.add_argument('--config', type = str, help = 'Load a specific configuration file')
+ args = parser.parse_args()
+
+ # Miscellaneous CLI args
+ if args.version: # --version: print version and exit
+ print(_VERSION)
+ sys.exit(0)
+
+ # Specific conf file to load?
+ if args.config:
+ if os.path.exists(args.config):
+ configpath = args.config
+ else:
+ print("Bork: --config passed to program, but could not find config file %s" % args.config)
+ sys.exit(-1)
+
+ # Init yaml, load configuration.
+ # We use ruamel.yaml here, because it preserves the existing structure and
+ # comments, unlike the traditional yaml library.
+ yaml = ruamel.yaml.YAML()
+ yaml.indent(sequence=4, offset=2)
+ conftext = open(configpath).read()
+ gconf = yaml.load(conftext)
+
+ # Unit test mode?
+ if args.test:
+ print("Running tests...")
+ import plugins.basics.unittests
+ gconf['version'] = _VERSION
+ plugins.basics.unittests.run(gconf)
+ sys.exit(0)
+
+
+ # If no app id set, get a unique app id for this node.
+ if gconf['client'].get('appid', 'UNSET') == 'UNSET':
+ gconf['client']['appid'] = plugins.basics.misc.appid()
+ print("Uninitialized node, setting base App ID to %s" % gconf['client']['appid'])
+ # Save updated changes to disk
+ yaml.dump(gconf, open(configpath, "w"))
+
+ # Set node software version for tests
+ gconf['version'] = _VERSION
+
+ # Get local time offset from NTP
+ toffset = plugins.basics.misc.adjustTime(gconf['misc']['ntpserver'])
+ gconf['misc']['offset'] = toffset
+
+ # Connect to master (or at least try to), leave a calling card if not known.
+
+
+ # Init an SMTP test object
+ t = plugins.tests.smtp.test(gconf)
+
+ # Run a test against hermes
+ t.run({'host': 'hermes.apache.org', 'port': '2025'})
+
diff --git a/plugins/basics/__init__.py b/plugins/basics/__init__.py
new file mode 100644
index 0000000..8f21a81
--- /dev/null
+++ b/plugins/basics/__init__.py
@@ -0,0 +1,8 @@
+import plugins.basics.socket
+import plugins.basics.misc
+
+__all__ = [
+ 'misc',
+ 'socket'
+]
+
diff --git a/plugins/basics/misc.py b/plugins/basics/misc.py
new file mode 100644
index 0000000..a59376b
--- /dev/null
+++ b/plugins/basics/misc.py
@@ -0,0 +1,88 @@
+#!/usr/bin/env python3.4
+# -*- coding: utf-8 -*-
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+ #the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+""" This is the library for miscellaneous auxiliary functions for
+ Apache Warble (incubating) nodes.
+"""
+
+# Socket imports
+import select
+import socket
+import ssl
+import struct
+from socket import AF_INET, SOCK_DGRAM
+import time
+import uuid
+
+def hostname():
+ return socket.gethostname()
+
+def appid():
+ return "%s/%s" % (hostname(), uuid.uuid4())
+
+def adjustTime(host):
+ TS_1970 = 2208988800
+ client = socket.socket( AF_INET, SOCK_DGRAM )
+ client.settimeout(5)
+ data = b'\x1b' + 47 * b'\0'
+ ipaddr = socket.gethostbyname(host)
+ client.sendto( data, ( ipaddr, 123 ))
+ try:
+ data, address = client.recvfrom( 1024 )
+ if data:
+ t = struct.unpack( '!12I', data )[10]
+ t -= TS_1970
+ offset = time.time() - t
+ if offset > 0:
+ print("NTP: Offsetting time by %d miliseconds (machine clock is slightly ahead of real time)" % (offset * 1000))
+ elif offset < 0:
+ print("NTP: Offsetting time by %d miliseconds (machine clock is slightly behind real time)" % (offset * 1000))
+ return offset
+ else:
+ return 0
+ except Exception as err:
+ return 0
+
+
+
+class timer():
+ def __init__(self):
+ self.started = time.time() - toffset
+ self.last = time.time();
+ self.log = {}
+ def add(self, logtype):
+ self.log['time_' + logtype] = time.time() - toffset
+ if (time.time() - self.last) > 60:
+ raise Exception("Monitoring step took more than 60 seconds to complete")
+ self.last = time.time()
+
+class debugger():
+ def __init__(self):
+ self.started = time.time()
+ self.log = ""
+ def add(self, message):
+ self.log += "[%s]: %s\r\n" % (time.asctime(time.gmtime()), message)
+ print("[%s]: %s\r\n" % (time.asctime(time.gmtime()), message))
+
+
+def makeError(component, errmsg):
+ log = {}
+ log['time'] = time.time() - toffset
+ log['component'] = component
+ log['error'] = errmsg
+ return log
+
diff --git a/plugins/basics/socket.py b/plugins/basics/socket.py
new file mode 100644
index 0000000..860a058
--- /dev/null
+++ b/plugins/basics/socket.py
@@ -0,0 +1,148 @@
+#!/usr/bin/env python3.4
+# -*- coding: utf-8 -*-
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+ #the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""
+This is the TCP/UDP Socket Library for Apache Warble (incubating)
+"""
+
+# Socket imports
+import select
+import socket
+import ssl
+import struct
+from socket import AF_INET, SOCK_DGRAM
+import time
+
+
+class tcp():
+ def __init__(self, testParameters, report):
+ self.report = report
+ self.iptype = socket.AF_INET6 if (testParameters.get('ipv6', False) == True) else socket.AF_INET
+ self.host = testParameters.get('host')
+ self.port = int(testParameters.get('port', 80))
+ self.error = None
+ self.socket = socket.socket(self.iptype, socket.SOCK_STREAM)
+ self.socket.settimeout(3)
+ self.bytes = 0
+ self.report.debug("Initialising socket")
+ self.report.timer('init')
+ self.status_code = None
+ self.location = None
+ self.server = None
+ self.realip = None
+ self.cert = None
+
+ try:
+ self.report.debug("Looking up hostname %s..." % self.host)
+ af, socktype, proto, canonname, sa = socket.getaddrinfo(self.host, self.port, self.iptype, socket.SOCK_STREAM)[0]
+ self.report.timer('dns')
+ self.sa = sa
+ except Exception as err:
+ raise Exception("Could not resolve hostname: %s" % err)
+
+
+ self.realip = sa[0];
+ if not self.realip:
+ self.report.error('dns', "Could not resolve host %s" % self.host)
+ return None
+ if self.realip and self.realip == '127.0.0.1' or self.realip == '::1':
+ self.report.error('dns', "Hostname %s points to localhost!" % self.host)
+ return None
+ self.report.debug("Connecting to %s:%u" % (self.realip, self.port))
+ self.socket = socket.socket(af, socktype, proto)
+
+ def __del__(self):
+ # Close socket if not already closed
+ try:
+ self.socket.close()
+ except:
+ pass
+
+ def secure(self, SNI = None, verify = False):
+ """ Wrap socket in OpenSSL """
+ self.report.debug("Wrapping socket for TLS")
+ if SNI:
+ self.report.debug("Using SNI extension for %s" % SNI)
+ context = ssl.SSLContext(ssl.PROTOCOL_TLSv1_2) # SSL, TLS1, TLS1.1 is largely deprecated now.
+ context.verify_mode = ssl.CERT_OPTIONAL
+ # Are we going to test the certificate for validity?
+ if verify == True:
+ context.verify_mode = ssl.CERT_REQUIRED
+ context.check_hostname = True
+ context.load_default_certs()
+
+ self.socket = context.wrap_socket(self.socket, server_hostname = SNI)
+
+ while True:
+ try:
+ self.socket.do_handshake()
+ break
+ except ssl.SSLWantReadError:
+ select.select([self.socket], [], [])
+ except ssl.SSLWantWriteError:
+ select.select([], [self.socket], [])
+ self.report.debug("Shook hands, TLS ready")
+
+ return context
+ else:
+ self.socket = ssl.wrap_socket(self.socket)
+
+
+ def connect(self):
+ try:
+ self.socket.connect(self.sa)
+ self.report.timer('connect')
+ except Exception as err:
+ print("Connection to %s failed" % self.realip)
+ raise Exception("Could not connect to host: %s" % str(err))
+
+ def send(self, b):
+ """ Send bytes (or convert string to bytes) to socket """
+ if type(b) is str:
+ self.socket.send(b.encode('ascii', errors = 'replace'))
+ else:
+ self.socket.send(b)
+
+ def readline(self, recv_buffer=256, delim=b'\n'):
+ """ Reads a line from a TCP (SSL?) socket, if presented within 60 seconds """
+ buffer = b''
+ data = True
+ self.socket.setblocking(0)
+ while data:
+ try:
+ data = self.socket.recv(recv_buffer)
+ buffer += data
+ self.bytes += len(data)
+ while delim in buffer:
+ line, buffer = buffer.split(delim, 1)
+ yield line
+ except BlockingIOError as err:
+ ready = select.select([self.socket], [], [], 60)
+ if ready[0]:
+ continue
+ else:
+ raise Exception("Socket timeout after 60 seconds")
+ except ssl.SSLWantReadError as err:
+ ready = select.select([self.socket], [], [], 60)
+ if ready[0]:
+ continue
+ else:
+ raise Exception("Socket timeout after 60 seconds")
+ except Exception as err:
+ print(type(err))
+ raise err
+
diff --git a/plugins/basics/unittests.py b/plugins/basics/unittests.py
new file mode 100644
index 0000000..4353ffa
--- /dev/null
+++ b/plugins/basics/unittests.py
@@ -0,0 +1,81 @@
+#!/usr/bin/env python3.4
+# -*- coding: utf-8 -*-
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+ #the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+""" This is a generic unit testing lib for
+ Apache Warble (incubating) nodes.
+"""
+
+import plugins.basics
+import plugins.tests
+import datetime
+
+def spit(t):
+ # All done, spit out report
+ print("TEST %s COMPLETED" % t.report.id)
+ print()
+
+ print("DEBUG:")
+ print('-' * 80)
+ for k, v in t.report._debug:
+ print(datetime.datetime.fromtimestamp(k).strftime("%Y-%m-%d %H:%M:%S.%f"), v)
+ print('-' * 80)
+ print("TIMESTAMPS:")
+ print('-' * 80)
+ previous = None
+ first = None
+ for k, v in sorted(t.report.timeseries.items(), key = lambda x: x[1]):
+ if not previous:
+ previous = v
+ first = v
+ print("%-10s: +%3dms (%3dms)" % (k, ((v - previous) * 1000), (v - first) * 1000))
+ previous = v
+ print('-' * 80)
+
+def uprint(t, params):
+ print("Running test %s:" % t.report.id)
+ print('-' * 80)
+ for k, v in params.items():
+ print("%-16s: %s" % (k, v))
+ print('-' * 80)
+
+def run(gc):
+ gc['debug'] = True
+
+ # HTTP test
+ params = {
+ 'host': 'www-eu.apache.org',
+ 'vhost': 'www.apache.org',
+ 'URI': '/',
+ 'port': 80
+ }
+ t = plugins.tests.http.test(gc)
+ uprint(t, params)
+ t.run(params)
+ spit(t)
+
+ # SMTP test
+ params = {
+ 'host': 'mail-relay.apache.org',
+ 'type': 'smtp',
+ 'SSL': True,
+ 'port': 465
+ }
+ uprint(t, params)
+ t = plugins.tests.smtp.test(gc)
+ t.run(params)
+ spit(t)
+
\ No newline at end of file
diff --git a/plugins/reports/__init__.py b/plugins/reports/__init__.py
new file mode 100644
index 0000000..255ec92
--- /dev/null
+++ b/plugins/reports/__init__.py
@@ -0,0 +1,5 @@
+import plugins.reports.generic
+
+__all__ = [
+ 'generic'
+]
diff --git a/plugins/reports/generic.py b/plugins/reports/generic.py
new file mode 100644
index 0000000..2cfab24
--- /dev/null
+++ b/plugins/reports/generic.py
@@ -0,0 +1,66 @@
+#!/usr/bin/env python3.4
+# -*- coding: utf-8 -*-
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+ #the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""
+This is the generic node report class for Apache Warble (incubating)
+"""
+
+import time
+import plugins.basics.misc
+import uuid
+
+class template:
+
+ def __init__(self, globalConfig):
+ self._debug = [] # Generic debug array with tuples in it
+ self._warn = [] # Generic warning array with tuples in it
+ self._alert = [] # Generic alert array with tuples in it
+ self._error = None # Error placeholder
+ self.timeseries = {} # Generic dictionary timeseries
+ self.id = uuid.uuid4() # Report ID
+ self.config = globalConfig
+ self.offset = globalConfig['misc'].get('offset', 0) # timestamp offset
+
+ def debug(self, string):
+ """ Logs a debug string in the report with a timestamp """
+ now = time.time() - self.offset
+ self._debug.append( (now, string) )
+ if self.config.get('debug', False) == True:
+ print(string)
+
+ def error(self, tag, string):
+ self._error = {
+ 'time': time.time() - self.offset,
+ 'component': tag,
+ 'message': string
+ }
+
+ def warn(self, string):
+ """ Logs a warning message """
+ now = time.time() - self.offset
+ self._warn.append( (now, string) )
+
+ def alert(self, string):
+ """ Logs an alert message """
+ now = time.time() - self.offset
+ self._alert.append( (now, string) )
+
+ def timer(self, tag):
+ """ Logs an event in a timeseries list """
+ now = time.time() - self.offset
+ self.timeseries[tag] = now
+
diff --git a/plugins/tests/__init__.py b/plugins/tests/__init__.py
new file mode 100644
index 0000000..6138558
--- /dev/null
+++ b/plugins/tests/__init__.py
@@ -0,0 +1,8 @@
+import plugins.tests.http
+import plugins.tests.smtp
+
+__all__ = [
+ 'http',
+ 'smtp'
+]
+
diff --git a/plugins/tests/http.py b/plugins/tests/http.py
new file mode 100644
index 0000000..687692d
--- /dev/null
+++ b/plugins/tests/http.py
@@ -0,0 +1,176 @@
+#!/usr/bin/env python3.4
+# -*- coding: utf-8 -*-
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+ #the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""
+This is the HTTP(S) test suite for Apache Warble (incubating).
+"""
+
+import plugins.basics
+import plugins.reports
+import ssl
+import re
+
+class test:
+ def __init__(self, globalConfig):
+ self.config = globalConfig
+ # Initialize a report object to store our findings
+ self.report = plugins.reports.generic.template(self.config)
+
+ def getCertData(cert):
+ """ Collates certificate data for HTTPS checks """
+ cn = ["none"]
+ ou = ["none"]
+ on = ["none"]
+ if 'subjectAltName' in cert:
+ cn = [x[1] for x in cert['subjectAltName']
+ if x[0].lower() == 'dns']
+ else:
+ cn = [x[0][1] for x in cert['subject']
+ if x[0][0].lower() == 'commonname']
+ ou = [x[0][1] for x in cert['subject']
+ if x[0][0].lower() == 'organizationalunitname']
+ on = [x[0][1] for x in cert['subject']
+ if x[0][0].lower() == 'organizationname']
+ return ("O=%s/OU=%s/CN=%s" % (str(on[0]) if len(on) > 0 else "Unknown", str(ou[0]) if len(ou) > 0 else "Unknown", str(cn[0]) if len(cn) > 0 else "Unknown"))
+
+
+ def run(self, testParameters):
+ request = plugins.basics.socket.tcp(testParameters, self.report)
+
+ try:
+ # Basic initialization and settings
+ pid = testParameters.get('id')
+ try:
+ request.init(testParameters)
+ except Exception as err:
+ self.report.error('init', str(err))
+
+ SSL = True if testParameters.get('type') == "https" else False # SSL/TLS request?
+ ise = testParameters.get('ise', 999) # Which status code(s) to treat as Internal Server Error/failure
+ method = testParameters.get('method', 'GET')
+ vhost = testParameters.get('vhost', testParameters.get('host', 'localhost'))
+
+ # Try to connect, if fail, report and return
+ try:
+ request.connect()
+ except Exception as err:
+ self.report.error('connect', str(err))
+ return report
+
+ # If SSL/TLS, initiate OpenSSL context
+ if SSL:
+ context = ssl.SSLContext(ssl.PROTOCOL_TLSv1_2) # SSL, TLS1, TLS1.1 is largely deprecated now.
+ context.verify_mode = ssl.CERT_OPTIONAL
+ # Are we going to test the certificate for validity?
+ if testParameters.get('checkcert', False) == True:
+ context.verify_mode = ssl.CERT_REQUIRED
+ context.check_hostname = True
+ context.load_default_certs()
+
+ # Hope for a vhost setting, fall back to host name or 'localhost'
+ request.socket = context.wrap_socket(sock.socket, server_hostname = vhost)
+
+ self.report.debug("Connected, sending HTTPS payload.")
+ request.cert = {}
+
+ cert = request.socket.getpeercert(binary_form=False)
+ cipher = request.socket.cipher()
+ request.cert['protocol'] = cipher[1]
+ request.cert['algorithm'] = cipher[0]
+ if cert:
+ self.report.debug("Analyzing server certificate")
+ request.cert['notbefore'] = cert['notBefore'] if 'notBefore' in cert else None
+ request.cert['notafter'] = cert['notAfter']
+ request.cert['subject'] = self.getCertData(cert)
+ request.cert['issuer'] = "Validated Certificate Authority"
+ now = time.time() - toffset
+ if not pid in certDates or certDates[pid] <= (now - (86400)):
+ self.report.debug("Saving certificate data")
+ if testParameters.get('checkcert', False) == True:
+ first = ssl.cert_time_to_seconds(cert['notBefore'])
+ last = ssl.cert_time_to_seconds(cert['notAfter'])
+ if first > now:
+ self.report.error('certificate', "HTTPS certificate is not yet valid (notBefore is greater than today)")
+ return
+ if last < now:
+ self.report.error('certificate', "HTTPS certificate has expired (notAfter is less than today)")
+ return
+ if testParameters.get('warncert', False) == True:
+ last = ssl.cert_time_to_seconds(cert['notAfter'])
+ if last < (time.time() + (86400*7)):
+ days = int((last - time.time()) / 86400)
+ self.report.error('certificate', "HTTPS certificate is about to expire (%u days from now)!" % days)
+ return
+
+ else:
+ self.report.debug("Connected, sending HTTP payload.")
+ request.send("%s %s HTTP/1.1\r\nConnection: close\r\nHost: %s\r\nUser-Agent: Apache Warble/%s\r\n\r\n" % (method.upper(), testParameters.get('uri', '/'), vhost, self.config.get('version')))
+ self.report.timer('send')
+ status = None
+ ISE = None
+ self.report.debug("Reading response header from server")
+ for line in request.readline():
+ line = str(line, 'utf-8')
+ if not status:
+ self.report.timer('read')
+ match = re.match("HTTP/[0-9.]+ (\d+)(.*)", line, flags=re.I)
+ if match:
+ rc = int(match.group(1))
+ request.status_code = match.group(1) + match.group(2)
+ self.report.debug("Server response code: %s" % request.status_code)
+ if rc > ise and ise > 0:
+ ISE = line
+ status = True
+ else:
+ raise Exception("Invalid HTTP response received: " + line)
+ if not request.status_code:
+ request.status_code = line
+ if line == "" or line == "\r":
+ break
+ match = re.match("Server: (.+)", line, flags=re.I)
+ if match:
+ request.server = match.group(1)
+ self.report.debug("Server software is: %s" % request.server)
+ match = re.match("Location: (.+)", line, flags=re.I)
+ if match:
+ request.location = match.group(1)
+
+ # Did we catch an internal server error or equivalent? bork!
+ if ISE:
+ self.report.error('response', "Internal Server Error or equivalent bad message received: " + ISE)
+ return
+
+ self.report.debug("Reading response body (up to 10kb)")
+ data = ""
+ while len(data) < 10240:
+ try:
+ bucket = request.socket.recv(1024)
+ if not bucket:
+ break
+ data += bucket
+ except Exception as err:
+ break
+ self.report.timer('data')
+ if data:
+ request.bytes += len(data)
+ self.report.debug("All went well, closing socket.")
+ request.socket.close()
+ self.report.timer('end')
+ except Exception as err:
+ print("Caught error:" + str(err))
+ self.report.error('response', str(err))
+
diff --git a/plugins/tests/smtp.py b/plugins/tests/smtp.py
new file mode 100644
index 0000000..5c4aeea
--- /dev/null
+++ b/plugins/tests/smtp.py
@@ -0,0 +1,67 @@
+#!/usr/bin/env python3
+# -*- coding: utf-8 -*-
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""
+This is the SMTP test suite for Apache Warble (incubating).
+It basically just connects to an SMTP service, checks the hello, and disconnects.
+"""
+
+import plugins.basics
+import plugins.reports
+
+class test:
+ def __init__(self, globalConfig):
+ self.config = globalConfig
+
+ # Initialize a report object to store our findings
+ self.report = plugins.reports.generic.template(self.config)
+
+ def run(self, testParameters):
+
+
+ try:
+ # Open up a TCP socket, tie to the report object and pass test parameters (host, port etc)
+ request = plugins.basics.socket.tcp(testParameters, self.report)
+
+ # Connect to host
+ request.connect()
+
+ # If SSL, wrap the socket to OpenSSL via the built-in secure() call.
+ SSL = testParameters.get('SSL', False)
+ if SSL == True:
+ request.secure(SNI = testParameters.get('host'))
+
+ # Now we basically just read the first line of response and assume
+ # eeeeverything is okay if that worked!
+ self.report.debug("Connected, reading response")
+ status = None
+ firstLine = next(request.readline()) # Just get the first line
+ self.report.debug("Got a line")
+ self.report.timer('read')
+ request.server = str(firstLine, 'utf-8') # convert from bytes to string
+ request.status_code = "Connection accepted"
+ status = True
+ self.report.timer('data')
+ self.report.debug("Response from server was: %s" % request.server)
+ self.report.debug("All went well, closing socket.")
+ self.report.timer('end')
+
+ except Exception as err:
+ print("Caught error:" + str(err))
+ if not self.report.error:
+ self.report.error('response', str(err))
+
diff --git a/requirements.txt b/requirements.txt
new file mode 100644
index 0000000..c35ac55
--- /dev/null
+++ b/requirements.txt
@@ -0,0 +1,5 @@
+# Warble requirements file
+ruamel.yaml
+requests
+dnspython
+ldap
--
To stop receiving notification emails like this one, please contact
humbedooh@apache.org.
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@warble.apache.org
For additional commands, e-mail: commits-help@warble.apache.org
[incubator-warble-node] 03/05: remove test (moved to unittests.py)
Posted by hu...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
humbedooh pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-warble-node.git
commit 87652572b9f8e93a91395238876dfcf4ee27c1b1
Author: Daniel Gruno <hu...@apache.org>
AuthorDate: Tue Jun 12 09:04:44 2018 -0500
remove test (moved to unittests.py)
---
node.py | 7 +------
1 file changed, 1 insertion(+), 6 deletions(-)
diff --git a/node.py b/node.py
index 93d5399..4de0eeb 100644
--- a/node.py
+++ b/node.py
@@ -88,11 +88,6 @@ if __name__ == "__main__":
gconf['misc']['offset'] = toffset
# Connect to master (or at least try to), leave a calling card if not known.
+ # TDB
- # Init an SMTP test object
- t = plugins.tests.smtp.test(gconf)
-
- # Run a test against hermes
- t.run({'host': 'hermes.apache.org', 'port': '2025'})
-
--
To stop receiving notification emails like this one, please contact
humbedooh@apache.org.
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@warble.apache.org
For additional commands, e-mail: commits-help@warble.apache.org
[incubator-warble-node] 02/05: add gitignore
Posted by hu...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
humbedooh pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-warble-node.git
commit cfa7a2538a1e9ddb7171ae0f966ce61988823327
Author: Daniel Gruno <hu...@apache.org>
AuthorDate: Tue Jun 12 09:02:15 2018 -0500
add gitignore
---
.gitignore | 2 ++
1 file changed, 2 insertions(+)
diff --git a/.gitignore b/.gitignore
new file mode 100644
index 0000000..c20c2ab
--- /dev/null
+++ b/.gitignore
@@ -0,0 +1,2 @@
+__pycache__
+
--
To stop receiving notification emails like this one, please contact
humbedooh@apache.org.
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@warble.apache.org
For additional commands, e-mail: commits-help@warble.apache.org
[incubator-warble-node] 05/05: add a brief readme
Posted by hu...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
humbedooh pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-warble-node.git
commit 3bf383faed8396a9aac3132a75d475dba4118a6f
Author: Daniel Gruno <hu...@apache.org>
AuthorDate: Tue Jun 12 09:10:50 2018 -0500
add a brief readme
---
README.md | 9 +++++++++
1 file changed, 9 insertions(+)
diff --git a/README.md b/README.md
new file mode 100644
index 0000000..de0790d
--- /dev/null
+++ b/README.md
@@ -0,0 +1,9 @@
+# Apache Warble Monitoring Node Package
+This isn't really working yet, but anywho.
+
+To install:
+ - install python 3
+ - clone the git repo, cd to it
+ - run: `pip3 install -r requirements.txt`
+ - to test, run: `python3 node.py --test`
+
--
To stop receiving notification emails like this one, please contact
humbedooh@apache.org.
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@warble.apache.org
For additional commands, e-mail: commits-help@warble.apache.org
[incubator-warble-node] 04/05: add generic tcp test class
Posted by hu...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
humbedooh pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-warble-node.git
commit 0d5d510cf8be2bb06a11c15ae8e236d51d4e651b
Author: Daniel Gruno <hu...@apache.org>
AuthorDate: Tue Jun 12 09:09:01 2018 -0500
add generic tcp test class
---
plugins/basics/unittests.py | 10 ++++++++
plugins/tests/__init__.py | 2 ++
plugins/tests/tcp.py | 60 +++++++++++++++++++++++++++++++++++++++++++++
3 files changed, 72 insertions(+)
diff --git a/plugins/basics/unittests.py b/plugins/basics/unittests.py
index 4353ffa..b484d15 100644
--- a/plugins/basics/unittests.py
+++ b/plugins/basics/unittests.py
@@ -55,6 +55,16 @@ def uprint(t, params):
def run(gc):
gc['debug'] = True
+ # TCP test
+ params = {
+ 'host': 'www-us.apache.org',
+ 'port': 80
+ }
+ t = plugins.tests.tcp.test(gc)
+ uprint(t, params)
+ t.run(params)
+ spit(t)
+
# HTTP test
params = {
'host': 'www-eu.apache.org',
diff --git a/plugins/tests/__init__.py b/plugins/tests/__init__.py
index 6138558..6ca6524 100644
--- a/plugins/tests/__init__.py
+++ b/plugins/tests/__init__.py
@@ -1,7 +1,9 @@
+import plugins.tests.tcp
import plugins.tests.http
import plugins.tests.smtp
__all__ = [
+ 'tcp',
'http',
'smtp'
]
diff --git a/plugins/tests/tcp.py b/plugins/tests/tcp.py
new file mode 100644
index 0000000..1038191
--- /dev/null
+++ b/plugins/tests/tcp.py
@@ -0,0 +1,60 @@
+#!/usr/bin/env python3
+# -*- coding: utf-8 -*-
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""
+This is the generic TCP test suite for Apache Warble (incubating).
+It basically just connects to an port and disconnects.
+"""
+
+import plugins.basics
+import plugins.reports
+
+class test:
+ def __init__(self, globalConfig):
+ self.config = globalConfig
+
+ # Initialize a report object to store our findings
+ self.report = plugins.reports.generic.template(self.config)
+
+ def run(self, testParameters):
+
+
+ try:
+ # Open up a TCP socket, tie to the report object and pass test parameters (host, port etc)
+ request = plugins.basics.socket.tcp(testParameters, self.report)
+
+ # Connect to host
+ request.connect()
+
+ # If SSL, wrap the socket to OpenSSL via the built-in secure() call.
+ SSL = testParameters.get('SSL', False)
+ if SSL == True:
+ request.secure(SNI = testParameters.get('host'))
+
+ # We're connected, that's it! goodbye!
+ self.report.debug("Connected to host")
+ request.status_code = "Connection accepted"
+ status = True
+ self.report.timer('data')
+ self.report.debug("All went well, closing socket.")
+ self.report.timer('end')
+
+ except Exception as err:
+ print("Caught error:" + str(err))
+ if not self.report.error:
+ self.report.error('response', str(err))
+
--
To stop receiving notification emails like this one, please contact
humbedooh@apache.org.
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@warble.apache.org
For additional commands, e-mail: commits-help@warble.apache.org