You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@qpid.apache.org by ch...@apache.org on 2018/11/30 20:19:11 UTC
[7/7] qpid-dispatch git commit: DISPATCH-1199: move scraper tool to
tools/scraper directory
DISPATCH-1199: move scraper tool to tools/scraper directory
Project: http://git-wip-us.apache.org/repos/asf/qpid-dispatch/repo
Commit: http://git-wip-us.apache.org/repos/asf/qpid-dispatch/commit/b7ab3390
Tree: http://git-wip-us.apache.org/repos/asf/qpid-dispatch/tree/b7ab3390
Diff: http://git-wip-us.apache.org/repos/asf/qpid-dispatch/diff/b7ab3390
Branch: refs/heads/master
Commit: b7ab33906353b08d15aaee7413f166896368cd6d
Parents: d59ab14
Author: Chuck Rolke <cr...@redhat.com>
Authored: Fri Nov 30 15:18:52 2018 -0500
Committer: Chuck Rolke <cr...@redhat.com>
Committed: Fri Nov 30 15:18:52 2018 -0500
----------------------------------------------------------------------
bin/log_scraper/README.md | 179 ----
bin/log_scraper/amqp_detail.py | 633 ------------
bin/log_scraper/common.py | 142 ---
bin/log_scraper/log_splitter.py | 445 ---------
bin/log_scraper/nicknamer.py | 139 ---
bin/log_scraper/parser.py | 1033 --------------------
bin/log_scraper/router.py | 226 -----
bin/log_scraper/scraper.py | 825 ----------------
bin/log_scraper/test_data.py | 54 -
bin/log_scraper/test_data/A-two-instances.log | 113 ---
bin/log_scraper/test_data/test_data.txt | 30 -
bin/log_scraper/text.py | 137 ---
tools/scraper/README.md | 179 ++++
tools/scraper/amqp_detail.py | 633 ++++++++++++
tools/scraper/common.py | 142 +++
tools/scraper/log_splitter.py | 445 +++++++++
tools/scraper/nicknamer.py | 139 +++
tools/scraper/parser.py | 1033 ++++++++++++++++++++
tools/scraper/router.py | 226 +++++
tools/scraper/scraper.py | 825 ++++++++++++++++
tools/scraper/test_data.py | 54 +
tools/scraper/test_data/A-two-instances.log | 113 +++
tools/scraper/test_data/test_data.txt | 30 +
tools/scraper/text.py | 137 +++
24 files changed, 3956 insertions(+), 3956 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/qpid-dispatch/blob/b7ab3390/bin/log_scraper/README.md
----------------------------------------------------------------------
diff --git a/bin/log_scraper/README.md b/bin/log_scraper/README.md
deleted file mode 100644
index 64ec977..0000000
--- a/bin/log_scraper/README.md
+++ /dev/null
@@ -1,179 +0,0 @@
-# Scraper - Render qpid-dispatch log files
-
-Scraper is a spinoff of https://github.com/ChugR/Adverb that uses qpid-dispatch log
-files as the data source instead of pcap trace files. Scraper is a Python processing
-engine that does not require Wireshark or any network trace capture utilities.
-
-## Apache License, Version 2.0
-
-Licensed under the Apache License, Version 2.0 (the "License"); you may not use
-this file except in compliance with the License. You may obtain a copy of the
-License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software distributed
-under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR
-CONDITIONS OF ANY KIND, either express or implied. See the License for the
-specific language governing permissions and limitations under the License.
-
-
-## Concepts
-
-Scraper is a data scraping program. It reads qpid-dispatch router log files,
-categorizes and sorts the data, and produces an HTML summary.
-
-From each log file Scraper extracts:
- * Router version
- * Router container name
- * Router restart times. A single log file may contain data from several router
- reboot instances.
- * Router link state calculation reports
- * Interrouter and client connections
- * AMQP facts
- * Connection peers
- * Link pair establishment
- * Transfer traffic
- * Message disposition
- * Flow and credit propagation
-
- Scraper sorts these facts with microsecond precision using the log timestamps.
-
- Then Scraper merges the data from any number (as long as that number is less than 27!)
- of independent log files into a single view.
-
- Next Scraper performs some higher-level analysis.
-
- * Routers are identified by letter rather than by the container name: 'A', 'B', and
- so on. Log data in a file is grouped into instances and is identified by a number
- for that router instance: 'A0', 'A1', and so on.
- * Per router each AMQP data log entry is sorted into per-connection data lists.
- * Connection data lists are searched to discover router-to-router and router-to-client
- connection pairs.
- * Per connection data are subdivided into per-session and per-link lists, sorting
- the AMQP data into per-link-only views.
- * Bulk AMQP data may be shown or hidden on arbitrary per-connection selections.
- * Noteworthy AMQP frames are identified. By hand these are hard to find.
- * AMQP errors
- * Presettled transfers
- * Transfers with 'more' bit set
- * Resumed transfers
- * Aborted transfers
- * Flow with 'drain' set
- * Transfer messages are sorted by signature. Then a table is made showing where
- each message leaves or arrives over a connection.
- * Settlement state for each unsettled transfer is identified, displayed, and
- shown with delta and elapsed time values. See example in the Advanced section.
- * Link name propagation for each named link is shown in a table.
- * Router, peer, and link names can get really long. Nicknames for each are used
- with popups showing the full name.
- * Transfer data is identified with nicknames but without the popups. The popups
- were so big that Firefox refused to show them; so forget it and they weren't useful anyway.
- * Router link state cost calculations are merged with router restart records to
- create a comprehensive link state cost view. Routers may publish cost reports that
- do not include all other routers. In this case the other routers are identified
- visually to indicate that they are unreachable.
-
-### The basics
-
-* Enable router logging
-
-The routers need to generate proper logging for Scraper.
-The information classes are exposed by enabling log levels.
-
-| Log level | Information |
-|----------------|---------------------------|
-| ROUTER info | Router version |
-| SERVER info | Router restart discovery |
-| SERVER trace | AMQP control and data |
-| ROUTER_LS info | Router link state reports |
-
-
-* Run your tests to populate log files used as Scraper input.
-
-* Run Scraper to generate web content
-
- bin/scraper/main.py somefile.log > somefile.html
-
- bin/scraper/mina.py *.log > somefile.html
-
-* Profit
-
- firefox somefile.html
-
-### Advanced
-
-* Merging multiple qpid-dispatch log files
-
-Scraper accepts multiple log files names in the command line and
-merges the log data according to the router log timestamps.
-
- bin/scraper/main.py A.log B.log C.log > abc.html
-
-Note that the qpid-dispatch host system clocks for merged log files
-must be synchronized to within a few microseconds in order for the
-result to be useful. This is easiest to achieve when the routers are
-run on the same CPU core on a single system. Running Fedora 27 and 28
-on two hosts in a router network where the routers run _ntp_ to the same
-time provider produces perfectly acceptable results.
-
-Scraper does a decent job merging log files created within a
-qpid-dispatch self test.
-
-* Wow, that's a lot of data
-
-Indeed it is and good luck figuring it out. Sometimes, though, it's too much.
-The AMQP transfer data analysis is the worst offender in terms of CPU time,
-run-time memory usage, and monstrous html output files.
-Scraper provides one command line switch to
-turn off the data analysis:
-
- bin/scraper/main.py --no-data FILE [FILE ...]
-
-In no-data mode AMQP transfer, disposition, and flow frames in the log files are
-discarded. The resulting web page still includes lots of useful information with
-connection info, link name propagation, and link state analysis.
-
-* How to read the transfer analysis tables. Here's an instance:
-
-
-|Src |Time |Rtr|ConnId|Dir|ConnId|Peer |T delta |T elapsed |Settlement |S elapsed
-|-------|---------------|---|------|---|------|------|---------|----------|-----------------------------|---------
-|A0_2035|09:50:52.027975|A0 |A0_11 |<- | |peer_7|0.000000 |0.000000 |(accepted settled 0.005142 S)|0.005142
-|A0_2071|09:50:52.028556|A0 |A0_6 |-> |D0_4 |D |0.000581 |0.000581 |(accepted settled 0.004253 S)|0.004834
-|D0_1587|09:50:52.028696|D0 |D0_4 |<- |A0_6 |A |0.000140 |0.000721 |(accepted settled 0.003988 S)|0.004709
-|D0_1612|09:50:52.029260|D0 |D0_1 |-> |C0_6 |C |0.000564 |0.001285 |(accepted settled 0.003044 S)|0.004329
-|C0_1610|09:50:52.029350|C0 |C0_6 |<- |D0_1 |D |0.000090 |0.001375 |(accepted settled 0.002846 S)|0.004221
-|C0_1625|09:50:52.029672|C0 |C0_1 |-> |B0_5 |B |0.000322 |0.001697 |(accepted settled 0.002189 S)|0.003886
-|B0_1438|09:50:52.029760|B0 |B0_5 |<- |C0_1 |C |0.000088 |0.001785 |(accepted settled 0.002002 S)|0.003787
-|B0_1451|09:50:52.030117|B0 |B0_7 |-> | |peer_7|0.000357 |0.002142 |(accepted settled 0.001318 S)|0.003460
-
-Each row in this table represents the facts about when a single transfer and its corresponding settlement was seen entering or exiting a router.
-
-| Field | Contents |
-|--------------|----------|
-|Src | Router instance and file line number where the transfer was seen|
-| Time | timestamp
-| Rtr | Router letter id and instance
-| ConnId | Router connection id
-| Dir | transfer direction. _<-_ indicates into the router, _->_ indicates out of the router
-| ConnId | peer's connection id. Blank if the peer is a normal client and not a router.
-| Peer | Peer's name. _peer7_ whold show the peer's container name in a popup.
-| T delta | Time since previous row
-| T elapsed | Time since the message first entered the system
-| Settlement | Settlement state and time delta since message time in column 2 for this row. The settlement disposition log line is hyperlinked from the word _accepted_.
-| S elapsed | Settlement elapsed time. This is the difference between the accepted disposition log record and the time when the message first entered the system.
-
-Row-by-row it is easiest to read the each line from left to right
-* A0 connecton 11 received the transfer from peer_7.
-* A0 connection 6 sent the message to D0 connection 4.
-* D0 connection 4 received the message from A0 connection 6.
-
-and so on. This message came from a sender on peer_7, went through routers A, D, C, and B, and finally was
-returned to a listener on peer_7. The receiver received the message 0.002142 S after the sender sent it. The
-sender received the accepted disposition 0.005142 S after the sender sent the message.
-
-The transmit times are in order from top to bottom and the settlement times are in order from bottom to top.
-
-This table will morph a little if one of the router is missing from the analysis. If log file D.log was not
-presented to Scraper then the table would not make as much sense as when all logs are included.
http://git-wip-us.apache.org/repos/asf/qpid-dispatch/blob/b7ab3390/bin/log_scraper/amqp_detail.py
----------------------------------------------------------------------
diff --git a/bin/log_scraper/amqp_detail.py b/bin/log_scraper/amqp_detail.py
deleted file mode 100755
index 042a19b..0000000
--- a/bin/log_scraper/amqp_detail.py
+++ /dev/null
@@ -1,633 +0,0 @@
-#!/usr/bin/env python
-
-#
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing,
-# software distributed under the License is distributed on an
-# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-# KIND, either express or implied. See the License for the
-# specific language governing permissions and limitations
-# under the License.
-#
-from __future__ import unicode_literals
-from __future__ import division
-from __future__ import absolute_import
-from __future__ import print_function
-
-import sys
-import traceback
-
-import common
-import text
-
-"""
-Given a map of all connections with lists of the associated frames
-analyze and show per-connection, per-session, and per-link details.
-
-This is done in a two-step process:
- * Run through the frame lists and generates an intermediate structure
- with the the details for display.
- * Generate the html from the detail structure.
-This strategy allows for a third step that would allow more details
-to be gleaned from the static details. For instance, if router A
-sends a transfer to router B then router A's details could show
-how long it took for the transfer to reach router B. Similarly
-router B's details could show how long ago router A sent the transfer.
-"""
-
-
-class ConnectionDetail():
- """
- Holds facts about sessions over the connection's lifetime
- """
-
- def __init__(self, id):
- # id in form 'A_15':
- # A is the router logfile key
- # 15 is the log connection number [15]
- self.id = id
-
- # seq_no number differentiates items that otherwise have same identifiers.
- # Sessions, for example: a given connection may have N distinct session
- # with local channel 0.
- self.seq_no = 0
-
- # combined amqp_error frames on this connection
- self.amqp_errors = 0
-
- # session_list holds all SessionDetail records either active or retired
- # Sessions for a connection are identified by the local channel number.
- # There may be many sessions all using the same channel number.
- # This list holds all of them.
- self.session_list = []
-
- # this map indexed by the channel refers to the current item in the session_list
- self.chan_map = {}
-
- # count of AMQP performatives for this connection that are not accounted
- # properly in session and link processing.
- # Server Accepting, SASL mechs, init, outcome, AMQP, and so on
- self.unaccounted_frame_list = []
-
- def FindSession(self, channel):
- """
- Find the current session by channel number
- :param channel: the performative channel
- :return: the session or None
- """
- return self.chan_map[channel] if channel in self.chan_map else None
-
- def GetId(self):
- return self.id
-
- def GetSeqNo(self):
- self.seq_no += 1
- return str(self.seq_no)
-
- def EndChannel(self, channel):
- # take existing session out of connection chan map
- if channel in self.chan_map:
- del self.chan_map[channel]
-
- def GetLinkEventCount(self):
- c = 0
- for session in self.session_list:
- c += session.GetLinkEventCount()
- return c
-
-
-class SessionDetail:
- """
- Holds facts about a session
- """
-
- def __init__(self, conn_detail, conn_seq, start_time):
- # parent connection
- self.conn_detail = conn_detail
-
- # some seq number
- self.conn_epoch = conn_seq
-
- # Timing
- self.time_start = start_time
- self.time_end = start_time
-
- self.amqp_errors = 0
-
- self.channel = -1
- self.peer_chan = -1
-
- self.direction = ""
-
- # seq_no number differentiates items that otherwise have same identifiers.
- # links for example
- self.seq_no = 0
-
- self.log_line_list = []
-
- # link_list holds LinkDetail records
- # Links for a session are identified by a (handle, remote-handle) number pair.
- # There may be many links all using the same handle pairs.
- # This list holds all of them.
- self.link_list = []
-
- # link_list holds all links either active or retired
- # this map indexed by the handle refers to the current item in the link_list
- self.input_handle_link_map = {} # link created by peer
- self.output_handle_link_map = {} # link created locally
-
- # Link name in attach finds link details in link_list
- # This map contains the link handle to disambiguate the name
- self.link_name_to_detail_map = {}
- #
- # The map contains the pure link name and is used only to resolve name collisions
- self.link_name_conflict_map = {}
-
- # count of AMQP performatives for this connection that are not accounted
- # properly in link processing
- self.session_frame_list = []
-
- # Session dispositions
- # Sender/receiver dispositions may be sent or received
- self.rx_rcvr_disposition_map = {} # key=delivery id, val=disposition plf
- self.rx_sndr_disposition_map = {} # key=delivery id, val=disposition plf
- self.tx_rcvr_disposition_map = {} # key=delivery id, val=disposition plf
- self.tx_sndr_disposition_map = {} # key=delivery id, val=disposition plf
-
- def FrameCount(self):
- count = 0
- for link in self.link_list:
- count += len(link.frame_list)
- count += len(self.session_frame_list)
- return count
-
- def FindLinkByName(self, attach_name, link_name_unambiguous, parsed_log_line):
- # find conflicted name
- cnl = None
- if attach_name in self.link_name_conflict_map:
- cnl = self.link_name_conflict_map[attach_name]
- if cnl.input_handle == -1 and cnl.output_handle == -1:
- cnl = None
- # find non-conflicted name
- nl = None
- if link_name_unambiguous in self.link_name_to_detail_map:
- nl = self.link_name_to_detail_map[link_name_unambiguous]
- if nl.input_handle == -1 and nl.output_handle == -1:
- nl = None
- # report conflict
- # TODO: There's an issue with this logic generating false positives
- # if nl is None and (not cnl is None):
- # parsed_log_line.data.amqp_error = True
- # parsed_log_line.data.web_show_str += " <span style=\"background-color:yellow\">Link name conflict</span>"
- # return unambiguous link
- return nl
-
- def FindLinkByHandle(self, handle, find_remote):
- """
- Find the current link by handle number
- qualify lookup based on packet direction
- :param link: the performative channel
- :param dst_is_broker: packet direction
- :return: the session or None
- """
- if find_remote:
- return self.input_handle_link_map[handle] if handle in self.input_handle_link_map else None
- else:
- return self.output_handle_link_map[handle] if handle in self.output_handle_link_map else None
-
- def GetId(self):
- return self.conn_detail.GetId() + "_" + str(self.conn_epoch)
-
- def GetSeqNo(self):
- self.seq_no += 1
- return self.seq_no
-
- def DetachOutputHandle(self, handle):
- # take existing link out of session handle map
- if handle in self.output_handle_link_map:
- nl = self.output_handle_link_map[handle]
- del self.output_handle_link_map[handle]
- nl.output_handle = -1
-
- def DetachInputHandle(self, handle):
- # take existing link out of session remote handle map
- if handle in self.input_handle_link_map:
- nl = self.input_handle_link_map[handle]
- del self.input_handle_link_map[handle]
- nl.input_handle = -1
-
- def DetachHandle(self, handle, is_remote):
- if is_remote:
- self.DetachInputHandle(handle)
- else:
- self.DetachOutputHandle(handle)
-
- def GetLinkEventCount(self):
- c = 0
- for link in self.link_list:
- c += link.GetLinkEventCount()
- return c
-
-
-class LinkDetail():
- """
- Holds facts about a link endpoint
- This structure binds input and output links with same name
- """
-
- def __init__(self, session_detail, session_seq, link_name, start_time):
- # parent session
- self.session_detail = session_detail
-
- # some seq number
- self.session_seq = session_seq
-
- # link name
- self.name = link_name # plf.data.link_short_name
- self.display_name = link_name # show short name; hover to see long name
-
- # Timing
- self.time_start = start_time
- self.time_end = start_time
-
- self.amqp_errors = 0
-
- # paired handles
- self.output_handle = -1
- self.input_handle = -1
-
- # link originator
- self.direction = ""
- self.is_receiver = True
- self.first_address = ''
-
- # set by sender
- self.snd_settle_mode = ''
- self.sender_target_address = "none"
- self.sender_class = ''
-
- # set by receiver
- self.rcv_settle_mode = ''
- self.receiver_source_address = "none"
- self.receiver_class = ''
-
- self.frame_list = []
-
- def GetId(self):
- return self.session_detail.GetId() + "_" + str(self.session_seq)
-
- def FrameCount(self):
- return len(self.frame_list)
-
-
-class AllDetails():
- #
- #
- def format_errors(self, n_errors):
- return ("<span style=\"background-color:yellow\">%d</span>" % n_errors) if n_errors > 0 else ""
-
- def classify_connection(self, id):
- """
- Return probable connection class based on the kinds of links the connection uses.
- TODO: This assumes that the connection has one session and one
- :param id:
- :return:
- """
- return "oops"
-
- def time_offset(self, ttest, t0):
- """
- Return a string time delta between two datetime objects in seconds formatted
- to six significant decimal places.
- :param ttest:
- :param t0:
- :return:
- """
- delta = ttest - t0
- t = float(delta.seconds) + float(delta.microseconds) / 1000000.0
- return "%0.06f" % t
-
- def links_in_connection(self, id):
- conn_details = self.conn_details[id]
- n_links = 0
- for sess in conn_details.session_list:
- n_links += len(sess.link_list)
- return n_links
-
- def settlement_display(self, transfer, disposition):
- """
- Generate the details for a disposition settlement
- :param transfer: plf
- :param disposition: plf
- :return: display string
- """
- state = disposition.data.disposition_state # accept, reject, release, ...
- if state != "accepted":
- state = "<span style=\"background-color:orange\">%s</span>" % state
- l2disp = "<a href=\"#%s\">%s</a>" % (disposition.fid, state)
- sttld = "settled" if disposition.data.settled == "true" else "unsettled"
- delay = self.time_offset(disposition.datetime, transfer.datetime)
- return "(%s %s %s S)" % (l2disp, sttld, delay)
-
- def resolve_settlement(self, link, transfer, rcv_disposition, snd_disposition):
- """
- Generate the settlement display string for this transfer.
- :param link: linkDetails - holds settlement modes
- :param transfer: plf of the transfer frame
- :param rcv_disposition: plf of receiver role disposition
- :param snd_disposition: plf of sender role disposition
- :return: display string
- """
- if transfer.data.settled is not None and transfer.data.settled == "true":
- result = "transfer presettled"
- if rcv_disposition is not None:
- sys.stderr.write("WARING: Receiver disposition for presettled message. connid:%s, line:%s\n" %
- (rcv_disposition.data.conn_id, rcv_disposition.lineno))
- if snd_disposition is not None:
- sys.stderr.write("WARING: Sender disposition for presettled message. connid:%s, line:%s\n" %
- (snd_disposition.data.conn_id, snd_disposition.lineno))
- else:
- if "1" in link.snd_settle_mode:
- # link mode sends only settled transfers
- result = "link presettled"
- if rcv_disposition is not None:
- sys.stderr.write("WARING: Receiver disposition for presettled link. connid:%s, line:%s\n" %
- (rcv_disposition.data.conn_id, rcv_disposition.lineno))
- if snd_disposition is not None:
- sys.stderr.write("WARING: Sender disposition for presettled link. connid:%s, line:%s\n" %
- (snd_disposition.data.conn_id, snd_disposition.lineno))
- else:
- # transfer unsettled and link mode requires settlement
- if rcv_disposition is not None:
- rtext = self.settlement_display(transfer, rcv_disposition)
- transfer.data.final_disposition = rcv_disposition
- if snd_disposition is not None:
- stext = self.settlement_display(transfer, snd_disposition)
- transfer.data.final_disposition = snd_disposition
-
- if "0" in link.rcv_settle_mode:
- # one settlement expected
- if rcv_disposition is not None:
- result = rtext
- if snd_disposition is not None:
- sys.stderr.write("WARING: Sender disposition for single first(0) settlement link. "
- "connid:%s, line:%s\n" %
- (snd_disposition.data.conn_id, snd_disposition.lineno))
- else:
- result = "rcvr: absent"
- else:
- # two settlements expected
- if rcv_disposition is not None:
- result = "rcvr: " + rtext
- if snd_disposition is not None:
- result += ", sndr: " + stext
- else:
- result += ", sndr: absent"
- else:
- result = "rcvr: absent"
- if snd_disposition is not None:
- result += ", sndr: " + stext
- else:
- result += ", sndr: absent"
- return result
-
- def __init__(self, _router, _common):
- self.rtr = _router
- self.comn = _common
-
- # conn_details - AMQP analysis
- # key= connection id '1', '2'
- # val= ConnectionDetails
- # for each connection, for each session, for each link:
- # what happened
- self.conn_details = {}
-
- for conn in self.rtr.conn_list:
- id = self.rtr.conn_id(conn)
- self.conn_details[id] = ConnectionDetail(id)
- conn_details = self.conn_details[id]
- conn_frames = self.rtr.conn_to_frame_map[id]
- for plf in conn_frames:
- pname = plf.data.name
- if plf.data.amqp_error:
- conn_details.amqp_errors += 1
- if pname in ['', 'open', 'close']:
- conn_details.unaccounted_frame_list.append(plf)
- continue
- # session required
- channel = plf.data.channel
- sess_details = conn_details.FindSession(channel)
- if sess_details == None:
- sess_details = SessionDetail(conn_details, conn_details.GetSeqNo(), plf.datetime)
- conn_details.session_list.append(sess_details)
- conn_details.EndChannel(channel)
- conn_details.chan_map[channel] = sess_details
- sess_details.direction = plf.data.direction
- sess_details.channel = channel
- if plf.data.amqp_error:
- sess_details.amqp_errors += 1
-
- if pname in ['begin', 'end', 'disposition']:
- sess_details.session_frame_list.append(plf)
-
- elif pname in ['attach']:
- handle = plf.data.handle # proton local handle
- link_name = plf.data.link_short_name
- link_name_unambiguous = link_name + "_" + str(handle)
- error_was = plf.data.amqp_error
- nl = sess_details.FindLinkByName(link_name, link_name_unambiguous, plf)
- # if finding an ambiguous link name generated an error then propagate to session/connection
- if not error_was and plf.data.amqp_error:
- conn_details.amqp_errors += 1
- sess_details.amqp_errors += 1
- if nl is None:
- # Creating a new link from scratch resulting in a half attached link pair
- nl = LinkDetail(sess_details, sess_details.GetSeqNo(), link_name, plf.datetime)
- sess_details.link_list.append(nl)
- sess_details.link_name_to_detail_map[link_name_unambiguous] = nl
- sess_details.link_name_conflict_map[link_name] = nl
- nl.display_name = plf.data.link_short_name_popup
- nl.direction = plf.data.direction
- nl.is_receiver = plf.data.role == "receiver"
- nl.first_address = plf.data.source if nl.is_receiver else plf.data.target
- if plf.data.amqp_error:
- nl.amqp_errors += 1
-
- if plf.data.direction_is_in():
- # peer is creating link
- nl.input_handle = handle
- sess_details.DetachInputHandle(handle)
- sess_details.input_handle_link_map[handle] = nl
- else:
- # local is creating link
- nl.output_handle = handle
- sess_details.DetachOutputHandle(handle)
- sess_details.output_handle_link_map[handle] = nl
- if plf.data.is_receiver:
- nl.rcv_settle_mode = plf.data.rcv_settle_mode
- nl.receiver_source_address = plf.data.source
- nl.receiver_class = plf.data.link_class
- else:
- nl.snd_settle_mode = plf.data.snd_settle_mode
- nl.sender_target_address = plf.data.target
- nl.sender_class = plf.data.link_class
- nl.frame_list.append(plf)
-
- elif pname in ['detach']:
- ns = conn_details.FindSession(channel)
- if ns is None:
- conn_details.unaccounted_frame_list.append(plf)
- continue
- handle = plf.data.handle
- nl = ns.FindLinkByHandle(handle, plf.data.direction_is_in())
- ns.DetachHandle(handle, plf.data.direction_is_in())
- if nl is None:
- ns.session_frame_list.append(plf)
- else:
- if plf.data.amqp_error:
- nl.amqp_errors += 1
- nl.frame_list.append(plf)
-
- elif pname in ['transfer', 'flow']:
- ns = conn_details.FindSession(channel)
- if ns is None:
- conn_details.unaccounted_frame_list.append(plf)
- continue
- handle = plf.data.handle
- nl = ns.FindLinkByHandle(handle, plf.data.direction_is_in())
- if nl is None:
- ns.session_frame_list.append(plf)
- else:
- if plf.data.amqp_error:
- nl.amqp_errors += 1
- nl.frame_list.append(plf)
- # identify and index dispositions
- for conn in self.rtr.conn_list:
- id = self.rtr.conn_id(conn)
- conn_detail = self.conn_details[id]
- for sess in conn_detail.session_list:
- # for each disposition add state to disposition_map
- for splf in sess.session_frame_list:
- if splf.data.name == "disposition":
- if splf.data.direction == "<-":
- sdispmap = sess.rx_rcvr_disposition_map if splf.data.is_receiver else sess.rx_sndr_disposition_map
- else:
- sdispmap = sess.tx_rcvr_disposition_map if splf.data.is_receiver else sess.tx_sndr_disposition_map
- for sdid in range(int(splf.data.first), (int(splf.data.last) + 1)):
- did = str(sdid)
- if did in sdispmap:
- sys.stderr.write("ERROR: Delivery ID collision in disposition map. connid:%s, \n" %
- (splf.data.conn_id))
- sdispmap[did] = splf
-
- def show_html(self):
- for conn in self.rtr.conn_list:
- id = self.rtr.conn_id(conn)
- conn_detail = self.rtr.details.conn_details[id]
- conn_frames = self.rtr.conn_to_frame_map[id]
- print("<a name=\"cd_%s\"></a>" % id)
- # This lozenge shows/hides the connection's data
- print("<a href=\"javascript:toggle_node('%s_data')\">%s%s</a>" %
- (id, text.lozenge(), text.nbsp()))
- dir = self.rtr.conn_dir[id] if id in self.rtr.conn_dir else ""
- peer = self.rtr.conn_peer_display.get(id, "") # peer container id
- peerconnid = self.comn.conn_peers_connid.get(id, "")
- # show the connection title
- print("%s %s %s %s (nFrames=%d) %s<br>" % \
- (id, dir, peerconnid, peer, len(conn_frames), self.format_errors(conn_detail.amqp_errors)))
- # data div
- print("<div id=\"%s_data\" style=\"display:none; margin-bottom: 2px; margin-left: 10px\">" % id)
-
- # unaccounted frames
- print("<a href=\"javascript:toggle_node('%s_data_unacc')\">%s%s</a>" %
- (id, text.lozenge(), text.nbsp()))
- # show the connection-level frames
- errs = sum(1 for plf in conn_detail.unaccounted_frame_list if plf.data.amqp_error)
- print("Connection-based entries %s<br>" % self.format_errors(errs))
- print("<div id=\"%s_data_unacc\" style=\"display:none; margin-bottom: 2px; margin-left: 10px\">" % id)
- for plf in conn_detail.unaccounted_frame_list:
- print(plf.adverbl_link_to(), plf.datetime, plf.data.direction, peer, plf.data.web_show_str, "<br>")
- print("</div>") # end unaccounted frames
-
- # loop to print session details
- for sess in conn_detail.session_list:
- # show the session toggle and title
- print("<a href=\"javascript:toggle_node('%s_sess_%s')\">%s%s</a>" %
- (id, sess.conn_epoch, text.lozenge(), text.nbsp()))
- print("Session %s: channel: %s, peer channel: %s; Time: start %s, Counts: frames: %d %s<br>" % \
- (sess.conn_epoch, sess.channel, sess.peer_chan, sess.time_start, \
- sess.FrameCount(), self.format_errors(sess.amqp_errors)))
- print("<div id=\"%s_sess_%s\" style=\"display:none; margin-bottom: 2px; margin-left: 10px\">" %
- (id, sess.conn_epoch))
- # show the session-level frames
- errs = sum(1 for plf in sess.session_frame_list if plf.data.amqp_error)
- print("<a href=\"javascript:toggle_node('%s_sess_%s_unacc')\">%s%s</a>" %
- (id, sess.conn_epoch, text.lozenge(), text.nbsp()))
- print("Session-based entries %s<br>" % self.format_errors(errs))
- print("<div id=\"%s_sess_%s_unacc\" style=\"display:none; margin-bottom: 2px; margin-left: 10px\">" %
- (id, sess.conn_epoch))
- for plf in sess.session_frame_list:
- print(plf.adverbl_link_to(), plf.datetime, plf.data.direction, peer, plf.data.web_show_str, "<br>")
- print("</div>") # end <id>_sess_<conn_epoch>_unacc
- # loops to print session link details
- # first loop prints link table
- print("<table")
- print("<tr><th>Link</th> <th>Dir</th> <th>Role</th> <th>Address</th> <th>Class</th> "
- "<th>snd-settle-mode</th> <th>rcv-settle-mode</th> <th>Start time</th> <th>Frames</th> "
- "<th>AMQP errors</tr>")
- for link in sess.link_list:
- # show the link toggle and title
- showthis = ("<a href=\"javascript:toggle_node('%s_sess_%s_link_%s')\">%s</a>" %
- (id, sess.conn_epoch, link.session_seq, link.display_name))
- role = "receiver" if link.is_receiver else "sender"
- print("<tr><td>%s</td><td>%s</td><td>%s</td><td>%s</td><td>%s</td><td>%s</td><td>%s</td>"
- "<td>%s</td><td>%d</td><td>%s</td></tr>" % \
- (showthis, link.direction, role, link.first_address,
- (link.sender_class + '-' + link.receiver_class), link.snd_settle_mode,
- link.rcv_settle_mode, link.time_start, link.FrameCount(),
- self.format_errors(link.amqp_errors)))
- print("</table>")
- # second loop prints the link's frames
- for link in sess.link_list:
- print(
- "<div id=\"%s_sess_%s_link_%s\" style=\"display:none; margin-top: 2px; margin-bottom: 2px; margin-left: 10px\">" %
- (id, sess.conn_epoch, link.session_seq))
- print("<h4>Connection %s Session %s Link %s</h4>" %
- (id, sess.conn_epoch, link.display_name))
- for plf in link.frame_list:
- if plf.data.name == "transfer":
- tdid = plf.data.delivery_id
- if plf.data.direction == "->":
- rmap = sess.rx_rcvr_disposition_map
- tmap = sess.rx_sndr_disposition_map
- else:
- rmap = sess.tx_rcvr_disposition_map
- tmap = sess.tx_sndr_disposition_map
- plf.data.disposition_display = self.resolve_settlement(link, plf,
- rmap.get(tdid),
- tmap.get(tdid))
- print(plf.adverbl_link_to(), plf.datetime, plf.data.direction, peer, plf.data.web_show_str,
- plf.data.disposition_display, "<br>")
- print("</div>") # end link <id>_sess_<conn_epoch>_link_<sess_seq>
-
- print("</div>") # end session <id>_sess_<conn_epoch>
-
- print("</div>") # end current connection data
-
-
-if __name__ == "__main__":
-
- try:
- pass
- except:
- traceback.print_exc(file=sys.stdout)
- pass
http://git-wip-us.apache.org/repos/asf/qpid-dispatch/blob/b7ab3390/bin/log_scraper/common.py
----------------------------------------------------------------------
diff --git a/bin/log_scraper/common.py b/bin/log_scraper/common.py
deleted file mode 100755
index 0a74f3c..0000000
--- a/bin/log_scraper/common.py
+++ /dev/null
@@ -1,142 +0,0 @@
-#!/usr/bin/env python
-
-#
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing,
-# software distributed under the License is distributed on an
-# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-# KIND, either express or implied. See the License for the
-# specific language governing permissions and limitations
-# under the License.
-#
-
-# Common data storage and utilities
-
-import sys
-
-import nicknamer
-
-IS_PY2 = sys.version_info[0] == 2
-
-if IS_PY2:
- def dict_iteritems(d):
- return d.iteritems()
- def dict_iterkeys(d):
- return d.iterkeys()
-else:
- def dict_iteritems(d):
- return iter(d.items())
- def dict_iterkeys(d):
- return iter(d.keys())
-
-class Common():
-
- # analysis_level_ludicrous
- # Adverbl tries too hard to cross reference data
- # Use these switchs to turn some of the biggest offenders off
- per_link_detail = True
- message_progress_tables = False
-
- # returned from argparse.parse_args()
- args = None
-
- # first letter of the connection names
- log_char_base = 'A'
-
- # number of logs processed
- n_logs = 0
-
- # array of file name strings from command line
- # len=n_logs
- log_fns = []
-
- # discovered router container names
- # len=n_logs
- router_ids = [] # raw long names
-
- # router display names shortened with popups
- router_display_names = []
-
- # router modes in plain text
- router_modes = []
-
- # list of router-instance lists
- # [[A0, A1], [B0], [C0, C1, C2]]
- routers = []
-
- # ordered list of connection names across all routers
- all_conn_names = []
-
- # conn_details_map -
- # key=conn_id, val=ConnectionDetail for that connection
- conn_details_map = {}
-
- # mapping of connected routers by connection id
- # A0_1 is connected to B3_2
- # key = full conn_id 'A0_5'
- # val = full conn_id 'B0_8'
- # note names[key]=val and names[val]=key mutual reference
- conn_peers_connid = {}
-
- # short display name for peer indexed by connection id
- # A0_1 maps to B's container_name nickname
- conn_peers_display = {}
-
- # conn_to_frame_map - global list for easier iteration in main
- # key = conn_id full A0_3
- # val = list of plf lines
- conn_to_frame_map = {}
-
- shorteners = nicknamer.Shorteners()
-
- # when --no-data is in effect, how many log lines were skipped?
- data_skipped = 0
-
- def router_id_index(self, id):
- """
- Given a router full container name, return the index in router_ids table
- Throw value error if not found
- :param id:
- :return:
- """
- return self.router_ids.index(id)
-
-
-def log_letter_of(idx):
- '''
- Return the letter A, B, C, ... from the index 0..n
- :param idx:
- :return: A..Z
- '''
- if idx >= 26:
- sys.exit('ERROR: too many log files')
- return "ABCDEFGHIJKLMNOPQRSTUVWXYZ"[idx]
-
-def index_of_log_letter(letter):
- '''
- Return the index 0..25 of the firster letter of the 'letter' string
- Raise error if out of range
- :param letter:
- :return:
- '''
- val = "ABCDEFGHIJKLMNOPQRSTUVWXYZ".find(letter[0].upper())
- if val < 0 or val > 25:
- raise ValueError("index_of_log_letter Invalid log letter: %s", letter)
- return val
-
-class RestartRec():
- def __init__(self, _id, _router, _event, _datetime):
- self.id = _id
- self.router = _router
- self.event = _event
- self.datetime = _datetime
-
http://git-wip-us.apache.org/repos/asf/qpid-dispatch/blob/b7ab3390/bin/log_scraper/log_splitter.py
----------------------------------------------------------------------
diff --git a/bin/log_scraper/log_splitter.py b/bin/log_scraper/log_splitter.py
deleted file mode 100755
index cc5664e..0000000
--- a/bin/log_scraper/log_splitter.py
+++ /dev/null
@@ -1,445 +0,0 @@
-#!/usr/bin/env python
-
-#
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing,
-# software distributed under the License is distributed on an
-# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-# KIND, either express or implied. See the License for the
-# specific language governing permissions and limitations
-# under the License.
-#
-
-# Split a gigantic (or not) log file into files of traffic for each connection.
-# Identify probable router and broker connections, QpidJMS client connections,
-# and AMQP errors. Create lists of connections sorted by log line and by transfer counts.
-# Emit a web page summarizing the results.
-
-from __future__ import unicode_literals
-from __future__ import division
-from __future__ import absolute_import
-from __future__ import print_function
-
-import cgi
-import os
-import sys
-import traceback
-from collections import defaultdict
-
-
-class connection():
- def __init__(self, instance, conn_id, logfile):
- self.instance = instance
- self.conn_id = conn_id
- self.logfile = logfile
- self.lines = []
- self.key_name = connection.keyname(instance, conn_id)
- self.transfers = 0
- self.peer_open = ""
- self.peer_type = ""
- self.log_n_lines = 0
- self.log_n_dir = ""
- self.file_name = ""
- self.path_name = ""
-
- @staticmethod
- def keyname(instance, conn_id):
- tmp = "0000000" + str(conn_id)
- return str(instance) + "." + tmp[-8:]
-
- def disp_name(self):
- return str(self.instance) + "_" + str(self.conn_id)
-
- def generate_paths(self):
- self.log_n_dir = "10e%d" % self.log_n_lines
- self.file_name = self.disp_name() + ".log"
- self.path_name = self.log_n_dir + "/" + self.file_name
-
-
-class LogFile:
- def __init__(self, fn, top_n=24):
- """
- Represent connections in a file
- :param fn: file name
- :param
- """
- self.log_fn = fn # file name
- self.top_n = top_n # how many to report
- self.instance = 0 # incremented when router restarts in log file
- self.amqp_lines = 0 # server trace lines
- self.transfers = 0 # server transfers
-
- # restarts
- self.restarts = []
-
- # connections
- # dictionary of connection data
- # key = connection id: <instance>.<conn_id> "0.3"
- # val = connection class object
- self.connections = {}
-
- # router_connections
- # list of received opens that suggest a router at the other end
- self.router_connections = []
-
- # broker connections
- # list of received opens that suggest a broker at the other end
- self.broker_connections = []
-
- # errors
- # amqp errors in time order
- self.errors = []
-
- # conns_by_size_transfer
- # all connections in transfer size descending order
- self.conns_by_size_transfer = []
-
- # conns_by_size_loglines
- # all connections in log_lines size descending order
- self.conns_by_size_loglines = []
-
- # histogram - count of connections with N logs < 10^index
- # [0] = N < 10^0
- # [1] = N < 10^1
- self.histogram = [0,0,0,0,0,0,0,0,0,0]
- self.hist_max = len(self.histogram) - 1
-
- def parse_identify(self, text, line, before_col=70):
- """
- Look for text in line but make sure it's not in the body of some message,
- :param text:
- :param line:
- :param before_col: limit on how far to search into line
- """
- st = line.find(text, 0, (before_col + len(text)))
- if st < 0:
- return False
- return st < 70
-
- def parse_line(self, line):
- """
- Do minimum parsing on line.
- If container name then bump instance value
- If server trace then get conn_id and add line to connections data
- :param line:
- :return:
- """
- key_sstart = "SERVER (info) Container Name:" # Normal 'router is starting' restart discovery line
- key_strace = "SERVER (trace) [" # AMQP traffic
- key_error = "@error(29)"
- key_openin = "<- @open(16)"
- key_xfer = "@transfer"
- key_prod_dispatch = ':product="qpid-dispatch-router"'
- key_prod_aartemis = ':product="apache-activemq-artemis"'
- key_prod_aqpidcpp = ':product="qpid-cpp"'
- key_prod_aqpidjms = ':product="QpidJMS"'
-
- if self.parse_identify(key_sstart, line):
- self.instance += 1
- self.restarts.append(line)
- else:
- if self.parse_identify(key_strace, line):
- self.amqp_lines += 1
- idx = line.find(key_strace)
- idx += len(key_strace)
- eidx = line.find("]", idx + 1)
- conn_id = line[idx:eidx]
- keyname = connection.keyname(self.instance, conn_id)
- if keyname not in self.connections:
- self.connections[keyname] = connection(self.instance, conn_id, self)
- curr_conn = self.connections[keyname]
- curr_conn.lines.append(line)
- # router hint
- if key_openin in line:
- # inbound open
- if key_prod_dispatch in line:
- self.router_connections.append(curr_conn)
- curr_conn.peer_open = line
- curr_conn.peer_type = key_prod_dispatch
- elif key_prod_aqpidjms in line:
- curr_conn.peer_type = key_prod_aqpidjms
- else:
- for k in [key_prod_aartemis, key_prod_aqpidcpp]:
- if k in line:
- self.broker_connections.append(curr_conn)
- curr_conn.peer_open = line
- curr_conn.peer_type = k
- elif self.parse_identify(key_xfer, line):
- self.transfers += 1
- curr_conn.transfers += 1
- if key_error in line:
- self.errors.append(line)
-
- def log_of(self, x):
- """
- calculate nearest power of 10 > x
- :param x:
- :return:
- """
- for i in range(self.hist_max):
- if x < 10 ** i:
- return i
- return self.hist_max
-
- def sort_sizes(self, sortfunc1, sortfunc2):
- smap = defaultdict(list)
- conns_by_size = []
- # create size map. index is size, list holds all connections of that many transfers
- for k, v in dict_iteritems(self.connections):
- smap[str(sortfunc1(v))].append(v)
- # create a sorted list of sizes in sizemap
- sl = list(dict_iterkeys(smap))
- sli = [int(k) for k in sl]
- slist = sorted(sli, reverse=True)
- # create grand list of all connections
- for cursize in slist:
- lsm = smap[str(cursize)]
- lsm = sorted(lsm, key = sortfunc2, reverse=True)
- #lsm = sorted(lsm, key = lambda x: int(x.conn_id))
- for ls in lsm:
- conns_by_size.append(ls)
- return conns_by_size
-
-
- def summarize_connections(self):
- # sort connections based on transfer count and on n log lines
- self.conns_by_size_transfer = self.sort_sizes(lambda x: x.transfers, lambda x: len(x.lines))
- self.conns_by_size_loglines = self.sort_sizes(lambda x: len(x.lines), lambda x: x.transfers)
-
- # compute log_n and file name facts for all connections
- for k, v in dict_iteritems(self.connections):
- v.log_n_lines = self.log_of(len(v.lines))
- v.generate_paths()
-
- # Write the web doc to stdout
- print ("""<!DOCTYPE html>
- <html>
- <head>
- <title>%s qpid-dispatch log split</title>
-
- <style>
- * {
- font-family: sans-serif;
- }
- table {
- border-collapse: collapse;
- }
- table, td, th {
- border: 1px solid black;
- padding: 3px;
- }
- </style>
-""" % self.log_fn)
-
- print("""
-<h3>Contents</h3>
-<table>
-<tr> <th>Section</th> <th>Description</th> </tr>
-<tr><td><a href=\"#c_summary\" >Summary</a></td> <td>Summary</td></tr>
-<tr><td><a href=\"#c_restarts\" >Router restarts</a></td> <td>Router reboot records</td></tr>
-<tr><td><a href=\"#c_router_conn\" >Interrouter connections</a></td> <td>Probable interrouter connections</td></tr>
-<tr><td><a href=\"#c_broker_conn\" >Broker connections</a></td> <td>Probable broker connections</td></tr>
-<tr><td><a href=\"#c_errors\" >AMQP errors</a></td> <td>AMQP errors</td></tr>
-<tr><td><a href=\"#c_conn_xfersize\" >Conn by N transfers</a></td> <td>Connections sorted by transfer log count</td></tr>
-<tr><td><a href=\"#c_conn_xfer0\" >Conn with no transfers</a></td> <td>Connections with no transfers</td></tr>
-<tr><td><a href=\"#c_conn_logsize\" >Conn by N log lines</a></td> <td>Connections sorted by total log line count</td></tr>
-</table>
-<hr>
-""")
- print("<a name=\"c_summary\"></a>")
- print("<table>")
- print("<tr><th>Statistic</th> <th>Value</th></tr>")
- print("<tr><td>File</td> <td>%s</td></tr>" % self.log_fn)
- print("<tr><td>Router starts</td> <td>%s</td></tr>" % str(self.instance))
- print("<tr><td>Connections</td> <td>%s</td></tr>" % str(len(self.connections)))
- print("<tr><td>Router connections</td> <td>%s</td></tr>" % str(len(self.router_connections)))
- print("<tr><td>AMQP log lines</td> <td>%s</td></tr>" % str(self.amqp_lines))
- print("<tr><td>AMQP errors</td> <td>%s</td></tr>" % str(len(self.errors)))
- print("<tr><td>AMQP transfers</td> <td>%s</td></tr>" % str(self.transfers))
- print("</table>")
- print("<hr>")
-
- # Restarts
- print("<a name=\"c_restarts\"></a>")
- print("<h3>Restarts</h3>")
- for i in range(1, (self.instance + 1)):
- rr = self.restarts[i-1]
- print("(%d) - %s<br>" % (i, rr), end='')
- print("<hr>")
-
- # interrouter connections
- print("<a name=\"c_router_conn\"></a>")
- print("<h3>Probable inter-router connections (N=%d)</h3>" % (len(self.router_connections)))
- print("<table>")
- print("<tr><th>Connection</th> <th>Transfers</th> <th>Log lines</th> <th>AMQP Open<th></tr>")
- for rc in self.router_connections:
- print("<tr><td><a href=\"%s/%s\">%s</a></td><td>%d</td><td>%d</td><td>%s</td></tr>" %
- (rc.logfile.odir(), rc.path_name, rc.disp_name(), rc.transfers, len(rc.lines),
- cgi.escape(rc.peer_open)))
- print("</table>")
- print("<hr>")
-
- # broker connections
- print("<a name=\"c_broker_conn\"></a>")
- print("<h3>Probable broker connections (N=%d)</h3>" % (len(self.broker_connections)))
- print("<table>")
- print("<tr><th>Connection</th> <th>Transfers</th> <th>Log lines</th> <th>AMQP Open<th></tr>")
- for rc in self.broker_connections:
- print("<tr><td><a href=\"%s/%s\">%s</a></td><td>%d</td><td>%d</td><td>%s</td></tr>" %
- (rc.logfile.odir(), rc.path_name, rc.disp_name(), rc.transfers, len(rc.lines),
- cgi.escape(rc.peer_open)))
- print("</table>")
- print("<hr>")
-
- ## histogram
- #for cursize in self.sizelist:
- # self.histogram[self.log_of(cursize)] += len(self.sizemap[str(cursize)])
- #print()
- #print("Log lines per connection distribution")
- #for i in range(1, self.hist_max):
- # print("N < 10e%d : %d" %(i, self.histogram[i]))
- #print("N >= 10e%d : %d" % ((self.hist_max - 1), self.histogram[self.hist_max]))
-
- # errors
- print("<a name=\"c_errors\"></a>")
- print("<h3>AMQP errors (N=%d)</h3>" % (len(self.errors)))
- print("<table>")
- print("<tr><th>N</th> <th>AMQP error</th></tr>")
- for i in range(len(self.errors)):
- print("<tr><td>%d</td> <td>%s</td></tr>" % (i, cgi.escape(self.errors[i].strip())))
- print("</table>")
- print("<hr>")
-
- def odir(self):
- return os.path.join(os.getcwd(), (self.log_fn + ".splits"))
-
- def write_subfiles(self):
- # Q: Where to put the generated files? A: odir
- odir = self.odir()
- odirs = ['dummy'] # dirs indexed by log of n-lines
-
- os.makedirs(odir)
- for i in range(1, self.hist_max):
- nrange = ("10e%d" % (i))
- ndir = os.path.join(odir, nrange)
- os.makedirs(ndir)
- odirs.append(ndir)
-
- for k, c in dict_iteritems(self.connections):
- cdir = odirs[self.log_of(len(c.lines))]
- opath = os.path.join(cdir, (c.disp_name() + ".log"))
- with open(opath, 'w') as f:
- for l in c.lines:
- f.write(l)
-
- xfer0 = 0
- for rc in self.conns_by_size_transfer:
- if rc.transfers == 0:
- xfer0 += 1
- print("<a name=\"c_conn_xfersize\"></a>")
- print("<h3>Connections by transfer count (N=%d)</h3>" % (len(self.conns_by_size_transfer) - xfer0))
- print("<table>")
- n = 1
- print("<tr><th>N</th><th>Connection</th> <th>Transfers</th> <th>Log lines</th> <th>Type</th> <th>AMQP detail<th></tr>")
- for rc in self.conns_by_size_transfer:
- if rc.transfers > 0:
- print("<tr><td>%d</td><td><a href=\"%s/%s\">%s</a></td> <td>%d</td> <td>%d</td> <td>%s</td> <td>%s</td></tr>" %
- (n, rc.logfile.odir(), rc.path_name, rc.disp_name(), rc.transfers, len(rc.lines),
- rc.peer_type, cgi.escape(rc.peer_open)))
- n += 1
- print("</table>")
- print("<hr>")
-
- print("<a name=\"c_conn_xfer0\"></a>")
- print("<h3>Connections with no AMQP transfers (N=%d)</h3>" % (xfer0))
- print("<table>")
- n = 1
- print("<tr><th>N</th><th>Connection</th> <th>Transfers</th> <th>Log lines</th> <th>Type</th> <th>AMQP detail<th></tr>")
- for rc in self.conns_by_size_transfer:
- if rc.transfers == 0:
- print("<tr><td>%d</td><td><a href=\"%s/%s\">%s</a></td> <td>%d</td> <td>%d</td> <td>%s</td> <td>%s</td></tr>" %
- (n, rc.logfile.odir(), rc.path_name, rc.disp_name(), rc.transfers, len(rc.lines),
- rc.peer_type, cgi.escape(rc.peer_open)))
- n += 1
- print("</table>")
- print("<hr>")
-
- print("<a name=\"c_conn_logsize\"></a>")
- print("<h3>Connections by total log line count (N=%d)</h3>" % (len(self.conns_by_size_loglines)))
- print("<table>")
- n = 1
- print("<tr><th>N</th><th>Connection</th> <th>Transfers</th> <th>Log lines</th> <th>Type</th> <th>AMQP detail<th></tr>")
- for rc in self.conns_by_size_loglines:
- print("<tr><td>%d</td><td><a href=\"%s/%s\">%s</a></td> <td>%d</td> <td>%d</td> <td>%s</td> <td>%s</td></tr>" %
- (n, rc.logfile.odir(), rc.path_name, rc.disp_name(), rc.transfers, len(rc.lines),
- rc.peer_type, cgi.escape(rc.peer_open)))
- n += 1
- print("</table>")
- print("<hr>")
-
-
-# py 2-3 compat
-
-IS_PY2 = sys.version_info[0] == 2
-
-if IS_PY2:
- def dict_iteritems(d):
- return d.iteritems()
- def dict_iterkeys(d):
- return d.iterkeys()
-else:
- def dict_iteritems(d):
- return iter(d.items())
- def dict_iterkeys(d):
- return iter(d.keys())
-
-
-#
-#
-def main_except(log_fn):
- """
- Given a log file name, split the file into per-connection sub files
- """
- log_files = []
-
- if not os.path.exists(log_fn):
- sys.exit('ERROR: log file %s was not found!' % log_fn)
-
- # parse the log file
- with open(log_fn, 'r') as infile:
- lf = LogFile(log_fn)
- odir = lf.odir()
- if os.path.exists(odir):
- sys.exit('ERROR: output directory %s exists' % odir)
- log_files.append(lf)
- for line in infile:
- lf.parse_line(line)
-
- # write output
- for lf in log_files:
- lf.summarize_connections() # prints web page to console
- lf.write_subfiles() # generates split files one-per-connection
- pass
-
-def main(argv):
- try:
- if len(argv) != 2:
- sys.exit('Usage: %s log-file-name' % argv[0])
- main_except(argv[1])
- return 0
- except Exception as e:
- traceback.print_exc()
- return 1
-
-
-if __name__ == "__main__":
- sys.exit(main(sys.argv))
http://git-wip-us.apache.org/repos/asf/qpid-dispatch/blob/b7ab3390/bin/log_scraper/nicknamer.py
----------------------------------------------------------------------
diff --git a/bin/log_scraper/nicknamer.py b/bin/log_scraper/nicknamer.py
deleted file mode 100755
index f198270..0000000
--- a/bin/log_scraper/nicknamer.py
+++ /dev/null
@@ -1,139 +0,0 @@
-#!/usr/bin/env python
-
-#
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing,
-# software distributed under the License is distributed on an
-# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-# KIND, either express or implied. See the License for the
-# specific language governing permissions and limitations
-# under the License.
-#
-
-from collections import defaultdict
-import common
-import cgi
-
-class ShortNames():
- '''
- Name shortener.
- The short name for display is "name_" + index(longName)
- Embellish the display name with an html popup
- Link and endpoint names, and data are tracked separately
- Names longer than threshold are shortened
- Each class has a prefix used when the table is dumped as HTML
- '''
- def __init__(self, prefixText, _threshold=25):
- self.longnames = []
- self.prefix = prefixText
- self.threshold = _threshold
- self.customer_dict = defaultdict(list)
-
- def translate(self, lname, show_popup=False, customer=None):
- '''
- Translate a long name into a short name, maybe.
- Memorize all names, translated or not
- Strip leading/trailing double quotes
- :param lname: the name
- :return: If shortened HTML string of shortened name with popup containing long name else
- not-so-long name.
- '''
- if lname.startswith("\"") and lname.endswith("\""):
- lname = lname[1:-1]
- try:
- idx = self.longnames.index(lname)
- except:
- self.longnames.append(lname)
- idx = self.longnames.index(lname)
- # return as-given if short enough
- if customer is not None:
- self.customer_dict[lname].append(customer)
- if len(lname) < self.threshold:
- return lname
- sname = self.prefix + "_" + str(idx)
- if customer is not None:
- self.customer_dict[sname].append(customer)
- if show_popup:
- return "<span title=\"" + cgi.escape(lname) + "\">" + sname + "</span>"
- else:
- return sname
-
- def len(self):
- return len(self.longnames)
-
- def prefix(self):
- return self.prefix
-
- def shortname(self, idx):
- name = self.longnames[idx]
- if len(name) < self.threshold:
- return name
- return self.prefix + "_" + str(idx)
-
- def prefixname(self, idx):
- return self.prefix + "_" + str(idx)
-
- def sname_to_popup(self, sname):
- if not sname.startswith(self.prefix):
- raise ValueError("Short name '%s' does not start with prefix '%s'" % (sname, self.prefix))
- try:
- lname = self.longnames[ int(sname[ (len(self.prefix) + 1): ])]
- except:
- raise ValueError("Short name '%s' did not translate to a long name" % (sname))
- return "<span title=\"" + cgi.escape(lname) + sname + "</span>"
-
- def longname(self, idx, cgi_escape=False):
- '''
- Get the cgi.escape'd long name
- :param idx:
- :param cgi_escape: true if caller wants the string for html display
- :return:
- '''
- return cgi.escape(self.longnames[idx]) if cgi_escape else self.longnames[idx]
-
- def htmlDump(self, with_link=False):
- '''
- Print the name table as an unnumbered list to stdout
- long names are cgi.escape'd
- :param with_link: true if link name link name is hyperlinked targeting itself
- :return: null
- '''
- if len(self.longnames) > 0:
- print ("<h3>" + self.prefix + " Name Index</h3>")
- print ("<ul>")
- for i in range(0, len(self.longnames)):
- name = self.prefix + "_" + str(i)
- dump_anchor = "<a name=\"%s_dump\"></a>" % (name)
- if with_link:
- name = "<a href=\"#%s\">%s</a>" % (name, name)
- print ("<li> " + dump_anchor + name + " - " + cgi.escape(self.longnames[i]) + "</li>")
- print ("</ul>")
-
- def sort_customers(self):
- for c in common.dict_iterkeys(self.customer_dict):
- l = self.customer_dict[c]
- self.customer_dict[c] = sorted(l, key=lambda lfl: lfl.datetime)
-
- def customers(self, sname):
- return self.customer_dict[sname]
-
-class Shorteners():
- def __init__(self):
- self.short_link_names = ShortNames("link", 15)
- self.short_addr_names = ShortNames("address")
- self.short_data_names = ShortNames("transfer", 2)
- self.short_peer_names = ShortNames("peer")
- self.short_rtr_names = ShortNames("router")
-
-
-if __name__ == "__main__":
- pass
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@qpid.apache.org
For additional commands, e-mail: commits-help@qpid.apache.org