You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@heron.apache.org by jo...@apache.org on 2020/04/16 05:42:51 UTC
[incubator-heron] branch asf-site updated: git-site-role commit
from publish_site.sh
This is an automated email from the ASF dual-hosted git repository.
joshfischer pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/incubator-heron.git
The following commit(s) were added to refs/heads/asf-site by this push:
new a83bfab git-site-role commit from publish_site.sh
a83bfab is described below
commit a83bfab13aa170ef6e1e80be4aa0dada83686f52
Author: jenkins <bu...@apache.org>
AuthorDate: Thu Apr 16 05:42:39 2020 +0000
git-site-role commit from publish_site.sh
---
content/api/python/api/bolt/base_bolt.m.html | 1581 ++++++++
content/api/python/api/bolt/bolt.m.html | 1617 ++++++++
content/api/python/api/bolt/index.html | 29 +
content/api/python/api/bolt/window_bolt.m.html | 2425 ++++++++++++
.../api/python/api/component/component_spec.m.html | 2066 ++++++++++
content/api/python/api/component/index.html | 8 +
content/api/python/api/index.html | 16 +
.../index.html => api/spout/base_spout.m.html} | 456 ++-
content/api/python/api/spout/index.html | 21 +
content/api/python/api/spout/spout.m.html | 1738 +++++++++
content/api/python/api/stream.m.html | 1723 +++++++++
content/api/python/api/topology.m.html | 2452 ++++++++++++
content/api/python/proto/ckptmgr_pb2.m.html | 4045 ++++++++++++++++++++
.../{api/index.html => proto/common_pb2.m.html} | 424 +-
.../execution_state_pb2.m.html} | 469 ++-
content/api/python/proto/index.html | 89 +
content/api/python/proto/metrics_pb2.m.html | 1993 ++++++++++
content/api/python/proto/packing_plan_pb2.m.html | 1600 ++++++++
content/api/python/proto/physical_plan_pb2.m.html | 1703 ++++++++
content/api/python/proto/scheduler_pb2.m.html | 1609 ++++++++
.../impl/index.html => proto/stats_pb2.m.html} | 156 +-
content/api/python/proto/stmgr_pb2.m.html | 2154 +++++++++++
content/api/python/proto/tmaster_pb2.m.html | 3338 ++++++++++++++++
content/api/python/proto/topology_pb2.m.html | 2902 ++++++++++++++
content/api/python/proto/tuple_pb2.m.html | 2049 ++++++++++
.../{api/index.html => streamlet/builder.m.html} | 304 +-
.../api/python/streamlet/impl/consumebolt.m.html | 2212 +++++++++++
.../api/python/streamlet/impl/filterbolt.m.html | 2234 +++++++++++
.../api/python/streamlet/impl/flatmapbolt.m.html | 2250 +++++++++++
.../python/streamlet/impl/generatorspout.m.html | 2243 +++++++++++
content/api/python/streamlet/impl/index.html | 120 +
content/api/python/streamlet/impl/joinbolt.m.html | 2764 +++++++++++++
content/api/python/streamlet/impl/logbolt.m.html | 2188 +++++++++++
content/api/python/streamlet/impl/mapbolt.m.html | 2234 +++++++++++
.../streamlet/impl/reducebykeyandwindowbolt.m.html | 2441 ++++++++++++
.../streamlet/impl/reducebywindowbolt.m.html | 2399 ++++++++++++
.../python/streamlet/impl/repartitionbolt.m.html | 2370 ++++++++++++
.../impl/{index.html => streamletboltbase.m.html} | 100 +-
.../impl/{index.html => streamletspoutbase.m.html} | 100 +-
.../api/python/streamlet/impl/supplierspout.m.html | 2227 +++++++++++
.../api/python/streamlet/impl/transformbolt.m.html | 2244 +++++++++++
content/api/python/streamlet/impl/unionbolt.m.html | 2211 +++++++++++
content/api/python/streamlet/index.html | 24 +
.../{api/index.html => streamlet/runner.m.html} | 266 +-
content/api/python/streamlet/streamlet.m.html | 2098 ++++++++++
45 files changed, 69114 insertions(+), 578 deletions(-)
diff --git a/content/api/python/api/bolt/base_bolt.m.html b/content/api/python/api/bolt/base_bolt.m.html
new file mode 100644
index 0000000..a7c204f
--- /dev/null
+++ b/content/api/python/api/bolt/base_bolt.m.html
@@ -0,0 +1,1581 @@
+<!doctype html>
+<head>
+ <meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
+ <meta name="viewport" content="width=device-width, initial-scale=1, minimum-scale=1" />
+
+ <title>heronpy.api.bolt.base_bolt API documentation</title>
+ <meta name="description" content="base_bolt.py" />
+
+ <link href='http://fonts.googleapis.com/css?family=Source+Sans+Pro:400,300' rel='stylesheet' type='text/css'>
+
+ <style type="text/css">
+
+* {
+ box-sizing: border-box;
+}
+/*! normalize.css v1.1.1 | MIT License | git.io/normalize */
+
+/* ==========================================================================
+ HTML5 display definitions
+ ========================================================================== */
+
+/**
+ * Correct `block` display not defined in IE 6/7/8/9 and Firefox 3.
+ */
+
+article,
+aside,
+details,
+figcaption,
+figure,
+footer,
+header,
+hgroup,
+main,
+nav,
+section,
+summary {
+ display: block;
+}
+
+/**
+ * Correct `inline-block` display not defined in IE 6/7/8/9 and Firefox 3.
+ */
+
+audio,
+canvas,
+video {
+ display: inline-block;
+ *display: inline;
+ *zoom: 1;
+}
+
+/**
+ * Prevent modern browsers from displaying `audio` without controls.
+ * Remove excess height in iOS 5 devices.
+ */
+
+audio:not([controls]) {
+ display: none;
+ height: 0;
+}
+
+/**
+ * Address styling not present in IE 7/8/9, Firefox 3, and Safari 4.
+ * Known issue: no IE 6 support.
+ */
+
+[hidden] {
+ display: none;
+}
+
+/* ==========================================================================
+ Base
+ ========================================================================== */
+
+/**
+ * 1. Prevent system color scheme's background color being used in Firefox, IE,
+ * and Opera.
+ * 2. Prevent system color scheme's text color being used in Firefox, IE, and
+ * Opera.
+ * 3. Correct text resizing oddly in IE 6/7 when body `font-size` is set using
+ * `em` units.
+ * 4. Prevent iOS text size adjust after orientation change, without disabling
+ * user zoom.
+ */
+
+html {
+ background: #fff; /* 1 */
+ color: #000; /* 2 */
+ font-size: 100%; /* 3 */
+ -webkit-text-size-adjust: 100%; /* 4 */
+ -ms-text-size-adjust: 100%; /* 4 */
+}
+
+/**
+ * Address `font-family` inconsistency between `textarea` and other form
+ * elements.
+ */
+
+html,
+button,
+input,
+select,
+textarea {
+ font-family: sans-serif;
+}
+
+/**
+ * Address margins handled incorrectly in IE 6/7.
+ */
+
+body {
+ margin: 0;
+}
+
+/* ==========================================================================
+ Links
+ ========================================================================== */
+
+/**
+ * Address `outline` inconsistency between Chrome and other browsers.
+ */
+
+a:focus {
+ outline: thin dotted;
+}
+
+/**
+ * Improve readability when focused and also mouse hovered in all browsers.
+ */
+
+a:active,
+a:hover {
+ outline: 0;
+}
+
+/* ==========================================================================
+ Typography
+ ========================================================================== */
+
+/**
+ * Address font sizes and margins set differently in IE 6/7.
+ * Address font sizes within `section` and `article` in Firefox 4+, Safari 5,
+ * and Chrome.
+ */
+
+h1 {
+ font-size: 2em;
+ margin: 0.67em 0;
+}
+
+h2 {
+ font-size: 1.5em;
+ margin: 0.83em 0;
+}
+
+h3 {
+ font-size: 1.17em;
+ margin: 1em 0;
+}
+
+h4 {
+ font-size: 1em;
+ margin: 1.33em 0;
+}
+
+h5 {
+ font-size: 0.83em;
+ margin: 1.67em 0;
+}
+
+h6 {
+ font-size: 0.67em;
+ margin: 2.33em 0;
+}
+
+/**
+ * Address styling not present in IE 7/8/9, Safari 5, and Chrome.
+ */
+
+abbr[title] {
+ border-bottom: 1px dotted;
+}
+
+/**
+ * Address style set to `bolder` in Firefox 3+, Safari 4/5, and Chrome.
+ */
+
+b,
+strong {
+ font-weight: bold;
+}
+
+blockquote {
+ margin: 1em 40px;
+}
+
+/**
+ * Address styling not present in Safari 5 and Chrome.
+ */
+
+dfn {
+ font-style: italic;
+}
+
+/**
+ * Address differences between Firefox and other browsers.
+ * Known issue: no IE 6/7 normalization.
+ */
+
+hr {
+ -moz-box-sizing: content-box;
+ box-sizing: content-box;
+ height: 0;
+}
+
+/**
+ * Address styling not present in IE 6/7/8/9.
+ */
+
+mark {
+ background: #ff0;
+ color: #000;
+}
+
+/**
+ * Address margins set differently in IE 6/7.
+ */
+
+p,
+pre {
+ margin: 1em 0;
+}
+
+/**
+ * Correct font family set oddly in IE 6, Safari 4/5, and Chrome.
+ */
+
+code,
+kbd,
+pre,
+samp {
+ font-family: monospace, serif;
+ _font-family: 'courier new', monospace;
+ font-size: 1em;
+}
+
+/**
+ * Improve readability of pre-formatted text in all browsers.
+ */
+
+pre {
+ white-space: pre;
+ white-space: pre-wrap;
+ word-wrap: break-word;
+}
+
+/**
+ * Address CSS quotes not supported in IE 6/7.
+ */
+
+q {
+ quotes: none;
+}
+
+/**
+ * Address `quotes` property not supported in Safari 4.
+ */
+
+q:before,
+q:after {
+ content: '';
+ content: none;
+}
+
+/**
+ * Address inconsistent and variable font size in all browsers.
+ */
+
+small {
+ font-size: 80%;
+}
+
+/**
+ * Prevent `sub` and `sup` affecting `line-height` in all browsers.
+ */
+
+sub,
+sup {
+ font-size: 75%;
+ line-height: 0;
+ position: relative;
+ vertical-align: baseline;
+}
+
+sup {
+ top: -0.5em;
+}
+
+sub {
+ bottom: -0.25em;
+}
+
+/* ==========================================================================
+ Lists
+ ========================================================================== */
+
+/**
+ * Address margins set differently in IE 6/7.
+ */
+
+dl,
+menu,
+ol,
+ul {
+ margin: 1em 0;
+}
+
+dd {
+ margin: 0 0 0 40px;
+}
+
+/**
+ * Address paddings set differently in IE 6/7.
+ */
+
+menu,
+ol,
+ul {
+ padding: 0 0 0 40px;
+}
+
+/**
+ * Correct list images handled incorrectly in IE 7.
+ */
+
+nav ul,
+nav ol {
+ list-style: none;
+ list-style-image: none;
+}
+
+/* ==========================================================================
+ Embedded content
+ ========================================================================== */
+
+/**
+ * 1. Remove border when inside `a` element in IE 6/7/8/9 and Firefox 3.
+ * 2. Improve image quality when scaled in IE 7.
+ */
+
+img {
+ border: 0; /* 1 */
+ -ms-interpolation-mode: bicubic; /* 2 */
+}
+
+/**
+ * Correct overflow displayed oddly in IE 9.
+ */
+
+svg:not(:root) {
+ overflow: hidden;
+}
+
+/* ==========================================================================
+ Figures
+ ========================================================================== */
+
+/**
+ * Address margin not present in IE 6/7/8/9, Safari 5, and Opera 11.
+ */
+
+figure {
+ margin: 0;
+}
+
+/* ==========================================================================
+ Forms
+ ========================================================================== */
+
+/**
+ * Correct margin displayed oddly in IE 6/7.
+ */
+
+form {
+ margin: 0;
+}
+
+/**
+ * Define consistent border, margin, and padding.
+ */
+
+fieldset {
+ border: 1px solid #c0c0c0;
+ margin: 0 2px;
+ padding: 0.35em 0.625em 0.75em;
+}
+
+/**
+ * 1. Correct color not being inherited in IE 6/7/8/9.
+ * 2. Correct text not wrapping in Firefox 3.
+ * 3. Correct alignment displayed oddly in IE 6/7.
+ */
+
+legend {
+ border: 0; /* 1 */
+ padding: 0;
+ white-space: normal; /* 2 */
+ *margin-left: -7px; /* 3 */
+}
+
+/**
+ * 1. Correct font size not being inherited in all browsers.
+ * 2. Address margins set differently in IE 6/7, Firefox 3+, Safari 5,
+ * and Chrome.
+ * 3. Improve appearance and consistency in all browsers.
+ */
+
+button,
+input,
+select,
+textarea {
+ font-size: 100%; /* 1 */
+ margin: 0; /* 2 */
+ vertical-align: baseline; /* 3 */
+ *vertical-align: middle; /* 3 */
+}
+
+/**
+ * Address Firefox 3+ setting `line-height` on `input` using `!important` in
+ * the UA stylesheet.
+ */
+
+button,
+input {
+ line-height: normal;
+}
+
+/**
+ * Address inconsistent `text-transform` inheritance for `button` and `select`.
+ * All other form control elements do not inherit `text-transform` values.
+ * Correct `button` style inheritance in Chrome, Safari 5+, and IE 6+.
+ * Correct `select` style inheritance in Firefox 4+ and Opera.
+ */
+
+button,
+select {
+ text-transform: none;
+}
+
+/**
+ * 1. Avoid the WebKit bug in Android 4.0.* where (2) destroys native `audio`
+ * and `video` controls.
+ * 2. Correct inability to style clickable `input` types in iOS.
+ * 3. Improve usability and consistency of cursor style between image-type
+ * `input` and others.
+ * 4. Remove inner spacing in IE 7 without affecting normal text inputs.
+ * Known issue: inner spacing remains in IE 6.
+ */
+
+button,
+html input[type="button"], /* 1 */
+input[type="reset"],
+input[type="submit"] {
+ -webkit-appearance: button; /* 2 */
+ cursor: pointer; /* 3 */
+ *overflow: visible; /* 4 */
+}
+
+/**
+ * Re-set default cursor for disabled elements.
+ */
+
+button[disabled],
+html input[disabled] {
+ cursor: default;
+}
+
+/**
+ * 1. Address box sizing set to content-box in IE 8/9.
+ * 2. Remove excess padding in IE 8/9.
+ * 3. Remove excess padding in IE 7.
+ * Known issue: excess padding remains in IE 6.
+ */
+
+input[type="checkbox"],
+input[type="radio"] {
+ box-sizing: border-box; /* 1 */
+ padding: 0; /* 2 */
+ *height: 13px; /* 3 */
+ *width: 13px; /* 3 */
+}
+
+/**
+ * 1. Address `appearance` set to `searchfield` in Safari 5 and Chrome.
+ * 2. Address `box-sizing` set to `border-box` in Safari 5 and Chrome
+ * (include `-moz` to future-proof).
+ */
+
+input[type="search"] {
+ -webkit-appearance: textfield; /* 1 */
+ -moz-box-sizing: content-box;
+ -webkit-box-sizing: content-box; /* 2 */
+ box-sizing: content-box;
+}
+
+/**
+ * Remove inner padding and search cancel button in Safari 5 and Chrome
+ * on OS X.
+ */
+
+input[type="search"]::-webkit-search-cancel-button,
+input[type="search"]::-webkit-search-decoration {
+ -webkit-appearance: none;
+}
+
+/**
+ * Remove inner padding and border in Firefox 3+.
+ */
+
+button::-moz-focus-inner,
+input::-moz-focus-inner {
+ border: 0;
+ padding: 0;
+}
+
+/**
+ * 1. Remove default vertical scrollbar in IE 6/7/8/9.
+ * 2. Improve readability and alignment in all browsers.
+ */
+
+textarea {
+ overflow: auto; /* 1 */
+ vertical-align: top; /* 2 */
+}
+
+/* ==========================================================================
+ Tables
+ ========================================================================== */
+
+/**
+ * Remove most spacing between table cells.
+ */
+
+table {
+ border-collapse: collapse;
+ border-spacing: 0;
+}
+
+ </style>
+
+ <style type="text/css">
+
+ html, body {
+ margin: 0;
+ padding: 0;
+ min-height: 100%;
+ }
+ body {
+ background: #fff;
+ font-family: "Source Sans Pro", "Helvetica Neueue", Helvetica, sans;
+ font-weight: 300;
+ font-size: 16px;
+ line-height: 1.6em;
+ }
+ #content {
+ width: 70%;
+ max-width: 850px;
+ float: left;
+ padding: 30px 60px;
+ border-left: 1px solid #ddd;
+ }
+ #sidebar {
+ width: 25%;
+ float: left;
+ padding: 30px;
+ overflow: hidden;
+ }
+ #nav {
+ font-size: 130%;
+ margin: 0 0 15px 0;
+ }
+
+ #top {
+ display: block;
+ position: fixed;
+ bottom: 5px;
+ left: 5px;
+ font-size: .85em;
+ text-transform: uppercase;
+ }
+
+ #footer {
+ font-size: .75em;
+ padding: 5px 30px;
+ border-top: 1px solid #ddd;
+ text-align: right;
+ }
+ #footer p {
+ margin: 0 0 0 30px;
+ display: inline-block;
+ }
+
+ h1, h2, h3, h4, h5 {
+ font-weight: 300;
+ }
+ h1 {
+ font-size: 2.5em;
+ line-height: 1.1em;
+ margin: 0 0 .50em 0;
+ }
+
+ h2 {
+ font-size: 1.75em;
+ margin: 1em 0 .50em 0;
+ }
+
+ h3 {
+ margin: 25px 0 10px 0;
+ }
+
+ h4 {
+ margin: 0;
+ font-size: 105%;
+ }
+
+ a {
+ color: #058;
+ text-decoration: none;
+ transition: color .3s ease-in-out;
+ }
+
+ a:hover {
+ color: #e08524;
+ transition: color .3s ease-in-out;
+ }
+
+ pre, code, .mono, .name {
+ font-family: "Ubuntu Mono", "Cousine", "DejaVu Sans Mono", monospace;
+ }
+
+ .title .name {
+ font-weight: bold;
+ }
+ .section-title {
+ margin-top: 2em;
+ }
+ .ident {
+ color: #900;
+ }
+
+ code {
+ background: #f9f9f9;
+ }
+
+ pre {
+ background: #fefefe;
+ border: 1px solid #ddd;
+ box-shadow: 2px 2px 0 #f3f3f3;
+ margin: 0 30px;
+ padding: 15px 30px;
+ }
+
+ .codehilite {
+ margin: 0 30px 10px 30px;
+ }
+
+ .codehilite pre {
+ margin: 0;
+ }
+ .codehilite .err { background: #ff3300; color: #fff !important; }
+
+ table#module-list {
+ font-size: 110%;
+ }
+
+ table#module-list tr td:first-child {
+ padding-right: 10px;
+ white-space: nowrap;
+ }
+
+ table#module-list td {
+ vertical-align: top;
+ padding-bottom: 8px;
+ }
+
+ table#module-list td p {
+ margin: 0 0 7px 0;
+ }
+
+ .def {
+ display: table;
+ }
+
+ .def p {
+ display: table-cell;
+ vertical-align: top;
+ text-align: left;
+ }
+
+ .def p:first-child {
+ white-space: nowrap;
+ }
+
+ .def p:last-child {
+ width: 100%;
+ }
+
+
+ #index {
+ list-style-type: none;
+ margin: 0;
+ padding: 0;
+ }
+ ul#index .class_name {
+ /* font-size: 110%; */
+ font-weight: bold;
+ }
+ #index ul {
+ margin: 0;
+ }
+
+ .item {
+ margin: 0 0 15px 0;
+ }
+
+ .item .class {
+ margin: 0 0 25px 30px;
+ }
+
+ .item .class ul.class_list {
+ margin: 0 0 20px 0;
+ }
+
+ .item .name {
+ background: #fafafa;
+ margin: 0;
+ font-weight: bold;
+ padding: 5px 10px;
+ border-radius: 3px;
+ display: inline-block;
+ min-width: 40%;
+ }
+ .item .name:hover {
+ background: #f6f6f6;
+ }
+
+ .item .empty_desc {
+ margin: 0 0 5px 0;
+ padding: 0;
+ }
+
+ .item .inheritance {
+ margin: 3px 0 0 30px;
+ }
+
+ .item .inherited {
+ color: #666;
+ }
+
+ .item .desc {
+ padding: 0 8px;
+ margin: 0;
+ }
+
+ .item .desc p {
+ margin: 0 0 10px 0;
+ }
+
+ .source_cont {
+ margin: 0;
+ padding: 0;
+ }
+
+ .source_link a {
+ background: #ffc300;
+ font-weight: 400;
+ font-size: .75em;
+ text-transform: uppercase;
+ color: #fff;
+ text-shadow: 1px 1px 0 #f4b700;
+
+ padding: 3px 8px;
+ border-radius: 2px;
+ transition: background .3s ease-in-out;
+ }
+ .source_link a:hover {
+ background: #FF7200;
+ text-shadow: none;
+ transition: background .3s ease-in-out;
+ }
+
+ .source {
+ display: none;
+ max-height: 600px;
+ overflow-y: scroll;
+ margin-bottom: 15px;
+ }
+
+ .source .codehilite {
+ margin: 0;
+ }
+
+ .desc h1, .desc h2, .desc h3 {
+ font-size: 100% !important;
+ }
+ .clear {
+ clear: both;
+ }
+
+ @media all and (max-width: 950px) {
+ #sidebar {
+ width: 35%;
+ }
+ #content {
+ width: 65%;
+ }
+ }
+ @media all and (max-width: 650px) {
+ #top {
+ display: none;
+ }
+ #sidebar {
+ float: none;
+ width: auto;
+ }
+ #content {
+ float: none;
+ width: auto;
+ padding: 30px;
+ }
+
+ #index ul {
+ padding: 0;
+ margin-bottom: 15px;
+ }
+ #index ul li {
+ display: inline-block;
+ margin-right: 30px;
+ }
+ #footer {
+ text-align: left;
+ }
+ #footer p {
+ display: block;
+ margin: inherit;
+ }
+ }
+
+ /*****************************/
+
+ </style>
+
+
+ <style type="text/css">
+
+/* ==========================================================================
+ EXAMPLE Media Queries for Responsive Design.
+ These examples override the primary ('mobile first') styles.
+ Modify as content requires.
+ ========================================================================== */
+
+@media only screen and (min-width: 35em) {
+ /* Style adjustments for viewports that meet the condition */
+}
+
+@media print,
+ (-o-min-device-pixel-ratio: 5/4),
+ (-webkit-min-device-pixel-ratio: 1.25),
+ (min-resolution: 120dpi) {
+ /* Style adjustments for high resolution devices */
+}
+
+/* ==========================================================================
+ Print styles.
+ Inlined to avoid required HTTP connection: h5bp.com/r
+ ========================================================================== */
+
+@media print {
+ * {
+ background: transparent !important;
+ color: #000 !important; /* Black prints faster: h5bp.com/s */
+ box-shadow: none !important;
+ text-shadow: none !important;
+ }
+
+ a,
+ a:visited {
+ text-decoration: underline;
+ }
+
+ a[href]:after {
+ content: " (" attr(href) ")";
+ }
+
+ abbr[title]:after {
+ content: " (" attr(title) ")";
+ }
+
+ /*
+ * Don't show links for images, or javascript/internal links
+ */
+
+ .ir a:after,
+ a[href^="javascript:"]:after,
+ a[href^="#"]:after {
+ content: "";
+ }
+
+ pre,
+ blockquote {
+ border: 1px solid #999;
+ page-break-inside: avoid;
+ }
+
+ thead {
+ display: table-header-group; /* h5bp.com/t */
+ }
+
+ tr,
+ img {
+ page-break-inside: avoid;
+ }
+
+ img {
+ max-width: 100% !important;
+ }
+
+ @page {
+ margin: 0.5cm;
+ }
+
+ p,
+ h2,
+ h3 {
+ orphans: 3;
+ widows: 3;
+ }
+
+ h2,
+ h3 {
+ page-break-after: avoid;
+ }
+}
+
+ </style>
+
+ <script type="text/javascript">
+ function toggle(id, $link) {
+ $node = document.getElementById(id);
+ if (!$node)
+ return;
+ if (!$node.style.display || $node.style.display == 'none') {
+ $node.style.display = 'block';
+ $link.innerHTML = 'Hide source ≢';
+ } else {
+ $node.style.display = 'none';
+ $link.innerHTML = 'Show source ≡';
+ }
+ }
+ </script>
+</head>
+<body>
+<a href="#" id="top">Top</a>
+
+<div id="container">
+
+
+ <div id="sidebar">
+ <h1>Index</h1>
+ <ul id="index">
+
+
+ <li class="set"><h3><a href="#header-classes">Classes</a></h3>
+ <ul>
+ <li class="mono">
+ <span class="class_name"><a href="#heronpy.api.bolt.base_bolt.BaseBolt">BaseBolt</a></span>
+
+
+ <ul>
+ <li class="mono"><a href="#heronpy.api.bolt.base_bolt.BaseBolt.is_tick">is_tick</a></li>
+ <li class="mono"><a href="#heronpy.api.bolt.base_bolt.BaseBolt.__init__">__init__</a></li>
+ <li class="mono"><a href="#heronpy.api.bolt.base_bolt.BaseBolt.ack">ack</a></li>
+ <li class="mono"><a href="#heronpy.api.bolt.base_bolt.BaseBolt.emit">emit</a></li>
+ <li class="mono"><a href="#heronpy.api.bolt.base_bolt.BaseBolt.fail">fail</a></li>
+ <li class="mono"><a href="#heronpy.api.bolt.base_bolt.BaseBolt.log">log</a></li>
+ <li class="mono"><a href="#heronpy.api.bolt.base_bolt.BaseBolt.spec">spec</a></li>
+ </ul>
+
+ </li>
+ </ul>
+ </li>
+
+ </ul>
+ </div>
+
+ <article id="content">
+
+
+
+
+
+
+ <header id="section-intro">
+ <h1 class="title"><span class="name">heronpy.api.bolt.base_bolt</span> module</h1>
+ <p>base_bolt.py</p>
+
+ <p class="source_link"><a href="javascript:void(0);" onclick="toggle('source-heronpy.api.bolt.base_bolt', this);">Show source ≡</a></p>
+ <div id="source-heronpy.api.bolt.base_bolt" class="source">
+ <pre><code># Copyright 2016 - Parsely, Inc. (d/b/a Parse.ly)
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+'''base_bolt.py'''
+import copy
+
+from heronpy.api.tuple import TupleHelper
+
+from heronpy.api.component.component_spec import HeronComponentSpec
+from heronpy.api.component.base_component import BaseComponent
+from heronpy.api.stream import Stream
+
+class BaseBolt(BaseComponent):
+ """BaseBolt class
+
+ This is the base for heron bolt, which wraps the implementation of publicly available methods.
+ This includes:
+ - <classmethod> spec()
+ - emit()
+ - <staticmethod> is_tick()
+ - ack()
+ - fail()
+
+ They are compatible with StreamParse API.
+ """
+ # pylint: disable=no-member
+ @classmethod
+ def spec(cls, name=None, inputs=None, par=1, config=None, optional_outputs=None):
+ """Register this bolt to the topology and create ``HeronComponentSpec``
+
+ This method takes an optional ``outputs`` argument for supporting dynamic output fields
+ declaration. However, it is recommended that ``outputs`` should be declared as
+ an attribute of your ``Bolt`` subclass. Also, some ways of declaring inputs is not supported
+ in this implementation; please read the documentation below.
+
+ :type name: str
+ :param name: Name of this bolt.
+ :type inputs: dict or list
+ :param inputs: Streams that feed into this Bolt.
+
+ Two forms of this are acceptable:
+
+ 1. A `dict` mapping from ``HeronComponentSpec`` to ``Grouping``.
+ In this case, default stream is used.
+ 2. A `dict` mapping from ``GlobalStreamId`` to ``Grouping``.
+ This ``GlobalStreamId`` object itself is different from StreamParse, because
+ Heron does not use thrift, although its constructor method is compatible.
+ 3. A `list` of ``HeronComponentSpec``. In this case, default stream with
+ SHUFFLE grouping is used.
+ 4. A `list` of ``GlobalStreamId``. In this case, SHUFFLE grouping is used.
+ :type par: int
+ :param par: Parallelism hint for this spout.
+ :type config: dict
+ :param config: Component-specific config settings.
+ :type optional_outputs: list of (str or Stream) or tuple of (str or Stream)
+ :param optional_outputs: Additional output fields for this bolt. These fields are added to
+ existing ``outputs`` class attributes of your bolt. This is an optional
+ argument, and exists only for supporting dynamic output field
+ declaration.
+ """
+ python_class_path = "%s.%s" % (cls.__module__, cls.__name__)
+
+ if hasattr(cls, 'outputs'):
+ # avoid modification to cls.outputs
+ _outputs = copy.copy(cls.outputs)
+ else:
+ _outputs = []
+
+ if optional_outputs is not None:
+ assert isinstance(optional_outputs, (list, tuple))
+ for out in optional_outputs:
+ assert isinstance(out, (str, Stream))
+ _outputs.append(out)
+
+ return HeronComponentSpec(name, python_class_path, is_spout=False, par=par,
+ inputs=inputs, outputs=_outputs, config=config)
+
+ def emit(self, tup, stream=Stream.DEFAULT_STREAM_ID,
+ anchors=None, direct_task=None, need_task_ids=False):
+ """Emits a new tuple from this Bolt
+
+ It is compatible with StreamParse API.
+
+ :type tup: list or tuple
+ :param tup: the new output Tuple to send from this bolt,
+ which should contain only serializable data.
+ :type stream: str
+ :param stream: the ID of the stream to emit this Tuple to.
+ Leave empty to emit to the default stream.
+ :type anchors: list
+ :param anchors: a list of HeronTuples to which the emitted Tuples should be anchored.
+ :type direct_task: int
+ :param direct_task: the task to send the Tuple to if performing a direct emit.
+ :type need_task_ids: bool
+ :param need_task_ids: indicate whether or not you would like the task IDs the Tuple was emitted.
+ """
+ self.delegate.emit(tup, stream, anchors, direct_task, need_task_ids)
+
+ @staticmethod
+ def is_tick(tup):
+ """Returns whether or not a given HeronTuple is a tick Tuple
+
+ It is compatible with StreamParse API.
+ """
+ return tup.stream == TupleHelper.TICK_TUPLE_ID
+
+ def ack(self, tup):
+ """Indicate that processing of a Tuple has succeeded
+
+ It is compatible with StreamParse API.
+ """
+ self.delegate.ack(tup)
+
+ def fail(self, tup):
+ """Indicate that processing of a Tuple has failed
+
+ It is compatible with StreamParse API.
+ """
+ self.delegate.fail(tup)
+</code></pre>
+ </div>
+
+ </header>
+
+ <section id="section-items">
+
+
+ <h2 class="section-title" id="header-classes">Classes</h2>
+
+ <div class="item">
+ <p id="heronpy.api.bolt.base_bolt.BaseBolt" class="name">class <span class="ident">BaseBolt</span></p>
+
+
+ <div class="desc"><p>BaseBolt class</p>
+<p>This is the base for heron bolt, which wraps the implementation of publicly available methods.
+This includes:
+ - <classmethod> spec()
+ - emit()
+ - <staticmethod> is_tick()
+ - ack()
+ - fail()</p>
+<p>They are compatible with StreamParse API.</p></div>
+ <div class="source_cont">
+ <p class="source_link"><a href="javascript:void(0);" onclick="toggle('source-heronpy.api.bolt.base_bolt.BaseBolt', this);">Show source ≡</a></p>
+ <div id="source-heronpy.api.bolt.base_bolt.BaseBolt" class="source">
+ <pre><code>class BaseBolt(BaseComponent):
+ """BaseBolt class
+
+ This is the base for heron bolt, which wraps the implementation of publicly available methods.
+ This includes:
+ - <classmethod> spec()
+ - emit()
+ - <staticmethod> is_tick()
+ - ack()
+ - fail()
+
+ They are compatible with StreamParse API.
+ """
+ # pylint: disable=no-member
+ @classmethod
+ def spec(cls, name=None, inputs=None, par=1, config=None, optional_outputs=None):
+ """Register this bolt to the topology and create ``HeronComponentSpec``
+
+ This method takes an optional ``outputs`` argument for supporting dynamic output fields
+ declaration. However, it is recommended that ``outputs`` should be declared as
+ an attribute of your ``Bolt`` subclass. Also, some ways of declaring inputs is not supported
+ in this implementation; please read the documentation below.
+
+ :type name: str
+ :param name: Name of this bolt.
+ :type inputs: dict or list
+ :param inputs: Streams that feed into this Bolt.
+
+ Two forms of this are acceptable:
+
+ 1. A `dict` mapping from ``HeronComponentSpec`` to ``Grouping``.
+ In this case, default stream is used.
+ 2. A `dict` mapping from ``GlobalStreamId`` to ``Grouping``.
+ This ``GlobalStreamId`` object itself is different from StreamParse, because
+ Heron does not use thrift, although its constructor method is compatible.
+ 3. A `list` of ``HeronComponentSpec``. In this case, default stream with
+ SHUFFLE grouping is used.
+ 4. A `list` of ``GlobalStreamId``. In this case, SHUFFLE grouping is used.
+ :type par: int
+ :param par: Parallelism hint for this spout.
+ :type config: dict
+ :param config: Component-specific config settings.
+ :type optional_outputs: list of (str or Stream) or tuple of (str or Stream)
+ :param optional_outputs: Additional output fields for this bolt. These fields are added to
+ existing ``outputs`` class attributes of your bolt. This is an optional
+ argument, and exists only for supporting dynamic output field
+ declaration.
+ """
+ python_class_path = "%s.%s" % (cls.__module__, cls.__name__)
+
+ if hasattr(cls, 'outputs'):
+ # avoid modification to cls.outputs
+ _outputs = copy.copy(cls.outputs)
+ else:
+ _outputs = []
+
+ if optional_outputs is not None:
+ assert isinstance(optional_outputs, (list, tuple))
+ for out in optional_outputs:
+ assert isinstance(out, (str, Stream))
+ _outputs.append(out)
+
+ return HeronComponentSpec(name, python_class_path, is_spout=False, par=par,
+ inputs=inputs, outputs=_outputs, config=config)
+
+ def emit(self, tup, stream=Stream.DEFAULT_STREAM_ID,
+ anchors=None, direct_task=None, need_task_ids=False):
+ """Emits a new tuple from this Bolt
+
+ It is compatible with StreamParse API.
+
+ :type tup: list or tuple
+ :param tup: the new output Tuple to send from this bolt,
+ which should contain only serializable data.
+ :type stream: str
+ :param stream: the ID of the stream to emit this Tuple to.
+ Leave empty to emit to the default stream.
+ :type anchors: list
+ :param anchors: a list of HeronTuples to which the emitted Tuples should be anchored.
+ :type direct_task: int
+ :param direct_task: the task to send the Tuple to if performing a direct emit.
+ :type need_task_ids: bool
+ :param need_task_ids: indicate whether or not you would like the task IDs the Tuple was emitted.
+ """
+ self.delegate.emit(tup, stream, anchors, direct_task, need_task_ids)
+
+ @staticmethod
+ def is_tick(tup):
+ """Returns whether or not a given HeronTuple is a tick Tuple
+
+ It is compatible with StreamParse API.
+ """
+ return tup.stream == TupleHelper.TICK_TUPLE_ID
+
+ def ack(self, tup):
+ """Indicate that processing of a Tuple has succeeded
+
+ It is compatible with StreamParse API.
+ """
+ self.delegate.ack(tup)
+
+ def fail(self, tup):
+ """Indicate that processing of a Tuple has failed
+
+ It is compatible with StreamParse API.
+ """
+ self.delegate.fail(tup)
+</code></pre>
+ </div>
+</div>
+
+
+ <div class="class">
+ <h3>Ancestors (in MRO)</h3>
+ <ul class="class_list">
+ <li><a href="#heronpy.api.bolt.base_bolt.BaseBolt">BaseBolt</a></li>
+ <li>heronpy.api.component.base_component.BaseComponent</li>
+ <li>__builtin__.object</li>
+ </ul>
+ <h3>Static methods</h3>
+
+ <div class="item">
+ <div class="name def" id="heronpy.api.bolt.base_bolt.BaseBolt.is_tick">
+ <p>def <span class="ident">is_tick</span>(</p><p>tup)</p>
+ </div>
+
+
+
+
+ <div class="desc"><p>Returns whether or not a given HeronTuple is a tick Tuple</p>
+<p>It is compatible with StreamParse API.</p></div>
+ <div class="source_cont">
+ <p class="source_link"><a href="javascript:void(0);" onclick="toggle('source-heronpy.api.bolt.base_bolt.BaseBolt.is_tick', this);">Show source ≡</a></p>
+ <div id="source-heronpy.api.bolt.base_bolt.BaseBolt.is_tick" class="source">
+ <pre><code>@staticmethod
+def is_tick(tup):
+ """Returns whether or not a given HeronTuple is a tick Tuple
+ It is compatible with StreamParse API.
+ """
+ return tup.stream == TupleHelper.TICK_TUPLE_ID
+</code></pre>
+ </div>
+</div>
+
+ </div>
+
+ <h3>Methods</h3>
+
+ <div class="item">
+ <div class="name def" id="heronpy.api.bolt.base_bolt.BaseBolt.__init__">
+ <p>def <span class="ident">__init__</span>(</p><p>self, delegate)</p>
+ </div>
+
+
+
+
+ <div class="desc"><p>Initializes BaseComponent</p>
+<p>:param delegate: SpoutInstance or BoltInstance</p></div>
+ <div class="source_cont">
+ <p class="source_link"><a href="javascript:void(0);" onclick="toggle('source-heronpy.api.bolt.base_bolt.BaseBolt.__init__', this);">Show source ≡</a></p>
+ <div id="source-heronpy.api.bolt.base_bolt.BaseBolt.__init__" class="source">
+ <pre><code>def __init__(self, delegate):
+ """Initializes BaseComponent
+ :param delegate: SpoutInstance or BoltInstance
+ """
+ self.delegate = delegate
+ self.logger = self.delegate.logger
+</code></pre>
+ </div>
+</div>
+
+ </div>
+
+
+ <div class="item">
+ <div class="name def" id="heronpy.api.bolt.base_bolt.BaseBolt.ack">
+ <p>def <span class="ident">ack</span>(</p><p>self, tup)</p>
+ </div>
+
+
+
+
+ <div class="desc"><p>Indicate that processing of a Tuple has succeeded</p>
+<p>It is compatible with StreamParse API.</p></div>
+ <div class="source_cont">
+ <p class="source_link"><a href="javascript:void(0);" onclick="toggle('source-heronpy.api.bolt.base_bolt.BaseBolt.ack', this);">Show source ≡</a></p>
+ <div id="source-heronpy.api.bolt.base_bolt.BaseBolt.ack" class="source">
+ <pre><code>def ack(self, tup):
+ """Indicate that processing of a Tuple has succeeded
+ It is compatible with StreamParse API.
+ """
+ self.delegate.ack(tup)
+</code></pre>
+ </div>
+</div>
+
+ </div>
+
+
+ <div class="item">
+ <div class="name def" id="heronpy.api.bolt.base_bolt.BaseBolt.emit">
+ <p>def <span class="ident">emit</span>(</p><p>self, tup, stream='default', anchors=None, direct_task=None, need_task_ids=False)</p>
+ </div>
+
+
+
+
+ <div class="desc"><p>Emits a new tuple from this Bolt</p>
+<p>It is compatible with StreamParse API.</p>
+<p>:type tup: list or tuple
+:param tup: the new output Tuple to send from this bolt,
+ which should contain only serializable data.
+:type stream: str
+:param stream: the ID of the stream to emit this Tuple to.
+ Leave empty to emit to the default stream.
+:type anchors: list
+:param anchors: a list of HeronTuples to which the emitted Tuples should be anchored.
+:type direct_task: int
+:param direct_task: the task to send the Tuple to if performing a direct emit.
+:type need_task_ids: bool
+:param need_task_ids: indicate whether or not you would like the task IDs the Tuple was emitted.</p></div>
+ <div class="source_cont">
+ <p class="source_link"><a href="javascript:void(0);" onclick="toggle('source-heronpy.api.bolt.base_bolt.BaseBolt.emit', this);">Show source ≡</a></p>
+ <div id="source-heronpy.api.bolt.base_bolt.BaseBolt.emit" class="source">
+ <pre><code>def emit(self, tup, stream=Stream.DEFAULT_STREAM_ID,
+ anchors=None, direct_task=None, need_task_ids=False):
+ """Emits a new tuple from this Bolt
+ It is compatible with StreamParse API.
+ :type tup: list or tuple
+ :param tup: the new output Tuple to send from this bolt,
+ which should contain only serializable data.
+ :type stream: str
+ :param stream: the ID of the stream to emit this Tuple to.
+ Leave empty to emit to the default stream.
+ :type anchors: list
+ :param anchors: a list of HeronTuples to which the emitted Tuples should be anchored.
+ :type direct_task: int
+ :param direct_task: the task to send the Tuple to if performing a direct emit.
+ :type need_task_ids: bool
+ :param need_task_ids: indicate whether or not you would like the task IDs the Tuple was emitted.
+ """
+ self.delegate.emit(tup, stream, anchors, direct_task, need_task_ids)
+</code></pre>
+ </div>
+</div>
+
+ </div>
+
+
+ <div class="item">
+ <div class="name def" id="heronpy.api.bolt.base_bolt.BaseBolt.fail">
+ <p>def <span class="ident">fail</span>(</p><p>self, tup)</p>
+ </div>
+
+
+
+
+ <div class="desc"><p>Indicate that processing of a Tuple has failed</p>
+<p>It is compatible with StreamParse API.</p></div>
+ <div class="source_cont">
+ <p class="source_link"><a href="javascript:void(0);" onclick="toggle('source-heronpy.api.bolt.base_bolt.BaseBolt.fail', this);">Show source ≡</a></p>
+ <div id="source-heronpy.api.bolt.base_bolt.BaseBolt.fail" class="source">
+ <pre><code>def fail(self, tup):
+ """Indicate that processing of a Tuple has failed
+ It is compatible with StreamParse API.
+ """
+ self.delegate.fail(tup)
+</code></pre>
+ </div>
+</div>
+
+ </div>
+
+
+ <div class="item">
+ <div class="name def" id="heronpy.api.bolt.base_bolt.BaseBolt.log">
+ <p>def <span class="ident">log</span>(</p><p>self, message, level=None)</p>
+ </div>
+
+
+
+
+ <div class="desc"><p>Log message, optionally providing a logging level</p>
+<p>:type message: str
+:param message: the log message to send
+:type level: str
+:param level: the logging level,
+ one of: trace (=debug), debug, info, warn or error (default: info)</p></div>
+ <div class="source_cont">
+ <p class="source_link"><a href="javascript:void(0);" onclick="toggle('source-heronpy.api.bolt.base_bolt.BaseBolt.log', this);">Show source ≡</a></p>
+ <div id="source-heronpy.api.bolt.base_bolt.BaseBolt.log" class="source">
+ <pre><code>def log(self, message, level=None):
+ """Log message, optionally providing a logging level
+ :type message: str
+ :param message: the log message to send
+ :type level: str
+ :param level: the logging level,
+ one of: trace (=debug), debug, info, warn or error (default: info)
+ """
+ self.delegate.log(message, level)
+</code></pre>
+ </div>
+</div>
+
+ </div>
+
+
+ <div class="item">
+ <div class="name def" id="heronpy.api.bolt.base_bolt.BaseBolt.spec">
+ <p>def <span class="ident">spec</span>(</p><p>cls, name=None, inputs=None, par=1, config=None, optional_outputs=None)</p>
+ </div>
+
+
+
+
+ <div class="desc"><p>Register this bolt to the topology and create <code>HeronComponentSpec</code></p>
+<p>This method takes an optional <code>outputs</code> argument for supporting dynamic output fields
+declaration. However, it is recommended that <code>outputs</code> should be declared as
+an attribute of your <code>Bolt</code> subclass. Also, some ways of declaring inputs is not supported
+in this implementation; please read the documentation below.</p>
+<p>:type name: str
+:param name: Name of this bolt.
+:type inputs: dict or list
+:param inputs: Streams that feed into this Bolt.</p>
+<pre><code> Two forms of this are acceptable:
+
+ 1. A `dict` mapping from ``HeronComponentSpec`` to ``Grouping``.
+ In this case, default stream is used.
+ 2. A `dict` mapping from ``GlobalStreamId`` to ``Grouping``.
+ This ``GlobalStreamId`` object itself is different from StreamParse, because
+ Heron does not use thrift, although its constructor method is compatible.
+ 3. A `list` of ``HeronComponentSpec``. In this case, default stream with
+ SHUFFLE grouping is used.
+ 4. A `list` of ``GlobalStreamId``. In this case, SHUFFLE grouping is used.
+</code></pre>
+<p>:type par: int
+:param par: Parallelism hint for this spout.
+:type config: dict
+:param config: Component-specific config settings.
+:type optional_outputs: list of (str or Stream) or tuple of (str or Stream)
+:param optional_outputs: Additional output fields for this bolt. These fields are added to
+ existing <code>outputs</code> class attributes of your bolt. This is an optional
+ argument, and exists only for supporting dynamic output field
+ declaration.</p></div>
+ <div class="source_cont">
+ <p class="source_link"><a href="javascript:void(0);" onclick="toggle('source-heronpy.api.bolt.base_bolt.BaseBolt.spec', this);">Show source ≡</a></p>
+ <div id="source-heronpy.api.bolt.base_bolt.BaseBolt.spec" class="source">
+ <pre><code>@classmethod
+def spec(cls, name=None, inputs=None, par=1, config=None, optional_outputs=None):
+ """Register this bolt to the topology and create ``HeronComponentSpec``
+ This method takes an optional ``outputs`` argument for supporting dynamic output fields
+ declaration. However, it is recommended that ``outputs`` should be declared as
+ an attribute of your ``Bolt`` subclass. Also, some ways of declaring inputs is not supported
+ in this implementation; please read the documentation below.
+ :type name: str
+ :param name: Name of this bolt.
+ :type inputs: dict or list
+ :param inputs: Streams that feed into this Bolt.
+ Two forms of this are acceptable:
+ 1. A `dict` mapping from ``HeronComponentSpec`` to ``Grouping``.
+ In this case, default stream is used.
+ 2. A `dict` mapping from ``GlobalStreamId`` to ``Grouping``.
+ This ``GlobalStreamId`` object itself is different from StreamParse, because
+ Heron does not use thrift, although its constructor method is compatible.
+ 3. A `list` of ``HeronComponentSpec``. In this case, default stream with
+ SHUFFLE grouping is used.
+ 4. A `list` of ``GlobalStreamId``. In this case, SHUFFLE grouping is used.
+ :type par: int
+ :param par: Parallelism hint for this spout.
+ :type config: dict
+ :param config: Component-specific config settings.
+ :type optional_outputs: list of (str or Stream) or tuple of (str or Stream)
+ :param optional_outputs: Additional output fields for this bolt. These fields are added to
+ existing ``outputs`` class attributes of your bolt. This is an optional
+ argument, and exists only for supporting dynamic output field
+ declaration.
+ """
+ python_class_path = "%s.%s" % (cls.__module__, cls.__name__)
+ if hasattr(cls, 'outputs'):
+ # avoid modification to cls.outputs
+ _outputs = copy.copy(cls.outputs)
+ else:
+ _outputs = []
+ if optional_outputs is not None:
+ assert isinstance(optional_outputs, (list, tuple))
+ for out in optional_outputs:
+ assert isinstance(out, (str, Stream))
+ _outputs.append(out)
+ return HeronComponentSpec(name, python_class_path, is_spout=False, par=par,
+ inputs=inputs, outputs=_outputs, config=config)
+</code></pre>
+ </div>
+</div>
+
+ </div>
+
+ </div>
+ </div>
+
+ </section>
+
+ </article>
+ <div class="clear"> </div>
+ <footer id="footer">
+ <p>
+ Documentation generated by
+ <a href="https://github.com/BurntSushi/pdoc">pdoc 0.3.2</a>
+ </p>
+
+ <p>pdoc is in the public domain with the
+ <a href="http://unlicense.org">UNLICENSE</a></p>
+
+ <p>Design by <a href="http://nadh.in">Kailash Nadh</a></p>
+ </footer>
+</div>
+</body>
+</html>
diff --git a/content/api/python/api/bolt/bolt.m.html b/content/api/python/api/bolt/bolt.m.html
new file mode 100644
index 0000000..fa3a56e
--- /dev/null
+++ b/content/api/python/api/bolt/bolt.m.html
@@ -0,0 +1,1617 @@
+<!doctype html>
+<head>
+ <meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
+ <meta name="viewport" content="width=device-width, initial-scale=1, minimum-scale=1" />
+
+ <title>heronpy.api.bolt.bolt API documentation</title>
+ <meta name="description" content="bolt.py: API for defining bolt in python" />
+
+ <link href='http://fonts.googleapis.com/css?family=Source+Sans+Pro:400,300' rel='stylesheet' type='text/css'>
+
+ <style type="text/css">
+
+* {
+ box-sizing: border-box;
+}
+/*! normalize.css v1.1.1 | MIT License | git.io/normalize */
+
+/* ==========================================================================
+ HTML5 display definitions
+ ========================================================================== */
+
+/**
+ * Correct `block` display not defined in IE 6/7/8/9 and Firefox 3.
+ */
+
+article,
+aside,
+details,
+figcaption,
+figure,
+footer,
+header,
+hgroup,
+main,
+nav,
+section,
+summary {
+ display: block;
+}
+
+/**
+ * Correct `inline-block` display not defined in IE 6/7/8/9 and Firefox 3.
+ */
+
+audio,
+canvas,
+video {
+ display: inline-block;
+ *display: inline;
+ *zoom: 1;
+}
+
+/**
+ * Prevent modern browsers from displaying `audio` without controls.
+ * Remove excess height in iOS 5 devices.
+ */
+
+audio:not([controls]) {
+ display: none;
+ height: 0;
+}
+
+/**
+ * Address styling not present in IE 7/8/9, Firefox 3, and Safari 4.
+ * Known issue: no IE 6 support.
+ */
+
+[hidden] {
+ display: none;
+}
+
+/* ==========================================================================
+ Base
+ ========================================================================== */
+
+/**
+ * 1. Prevent system color scheme's background color being used in Firefox, IE,
+ * and Opera.
+ * 2. Prevent system color scheme's text color being used in Firefox, IE, and
+ * Opera.
+ * 3. Correct text resizing oddly in IE 6/7 when body `font-size` is set using
+ * `em` units.
+ * 4. Prevent iOS text size adjust after orientation change, without disabling
+ * user zoom.
+ */
+
+html {
+ background: #fff; /* 1 */
+ color: #000; /* 2 */
+ font-size: 100%; /* 3 */
+ -webkit-text-size-adjust: 100%; /* 4 */
+ -ms-text-size-adjust: 100%; /* 4 */
+}
+
+/**
+ * Address `font-family` inconsistency between `textarea` and other form
+ * elements.
+ */
+
+html,
+button,
+input,
+select,
+textarea {
+ font-family: sans-serif;
+}
+
+/**
+ * Address margins handled incorrectly in IE 6/7.
+ */
+
+body {
+ margin: 0;
+}
+
+/* ==========================================================================
+ Links
+ ========================================================================== */
+
+/**
+ * Address `outline` inconsistency between Chrome and other browsers.
+ */
+
+a:focus {
+ outline: thin dotted;
+}
+
+/**
+ * Improve readability when focused and also mouse hovered in all browsers.
+ */
+
+a:active,
+a:hover {
+ outline: 0;
+}
+
+/* ==========================================================================
+ Typography
+ ========================================================================== */
+
+/**
+ * Address font sizes and margins set differently in IE 6/7.
+ * Address font sizes within `section` and `article` in Firefox 4+, Safari 5,
+ * and Chrome.
+ */
+
+h1 {
+ font-size: 2em;
+ margin: 0.67em 0;
+}
+
+h2 {
+ font-size: 1.5em;
+ margin: 0.83em 0;
+}
+
+h3 {
+ font-size: 1.17em;
+ margin: 1em 0;
+}
+
+h4 {
+ font-size: 1em;
+ margin: 1.33em 0;
+}
+
+h5 {
+ font-size: 0.83em;
+ margin: 1.67em 0;
+}
+
+h6 {
+ font-size: 0.67em;
+ margin: 2.33em 0;
+}
+
+/**
+ * Address styling not present in IE 7/8/9, Safari 5, and Chrome.
+ */
+
+abbr[title] {
+ border-bottom: 1px dotted;
+}
+
+/**
+ * Address style set to `bolder` in Firefox 3+, Safari 4/5, and Chrome.
+ */
+
+b,
+strong {
+ font-weight: bold;
+}
+
+blockquote {
+ margin: 1em 40px;
+}
+
+/**
+ * Address styling not present in Safari 5 and Chrome.
+ */
+
+dfn {
+ font-style: italic;
+}
+
+/**
+ * Address differences between Firefox and other browsers.
+ * Known issue: no IE 6/7 normalization.
+ */
+
+hr {
+ -moz-box-sizing: content-box;
+ box-sizing: content-box;
+ height: 0;
+}
+
+/**
+ * Address styling not present in IE 6/7/8/9.
+ */
+
+mark {
+ background: #ff0;
+ color: #000;
+}
+
+/**
+ * Address margins set differently in IE 6/7.
+ */
+
+p,
+pre {
+ margin: 1em 0;
+}
+
+/**
+ * Correct font family set oddly in IE 6, Safari 4/5, and Chrome.
+ */
+
+code,
+kbd,
+pre,
+samp {
+ font-family: monospace, serif;
+ _font-family: 'courier new', monospace;
+ font-size: 1em;
+}
+
+/**
+ * Improve readability of pre-formatted text in all browsers.
+ */
+
+pre {
+ white-space: pre;
+ white-space: pre-wrap;
+ word-wrap: break-word;
+}
+
+/**
+ * Address CSS quotes not supported in IE 6/7.
+ */
+
+q {
+ quotes: none;
+}
+
+/**
+ * Address `quotes` property not supported in Safari 4.
+ */
+
+q:before,
+q:after {
+ content: '';
+ content: none;
+}
+
+/**
+ * Address inconsistent and variable font size in all browsers.
+ */
+
+small {
+ font-size: 80%;
+}
+
+/**
+ * Prevent `sub` and `sup` affecting `line-height` in all browsers.
+ */
+
+sub,
+sup {
+ font-size: 75%;
+ line-height: 0;
+ position: relative;
+ vertical-align: baseline;
+}
+
+sup {
+ top: -0.5em;
+}
+
+sub {
+ bottom: -0.25em;
+}
+
+/* ==========================================================================
+ Lists
+ ========================================================================== */
+
+/**
+ * Address margins set differently in IE 6/7.
+ */
+
+dl,
+menu,
+ol,
+ul {
+ margin: 1em 0;
+}
+
+dd {
+ margin: 0 0 0 40px;
+}
+
+/**
+ * Address paddings set differently in IE 6/7.
+ */
+
+menu,
+ol,
+ul {
+ padding: 0 0 0 40px;
+}
+
+/**
+ * Correct list images handled incorrectly in IE 7.
+ */
+
+nav ul,
+nav ol {
+ list-style: none;
+ list-style-image: none;
+}
+
+/* ==========================================================================
+ Embedded content
+ ========================================================================== */
+
+/**
+ * 1. Remove border when inside `a` element in IE 6/7/8/9 and Firefox 3.
+ * 2. Improve image quality when scaled in IE 7.
+ */
+
+img {
+ border: 0; /* 1 */
+ -ms-interpolation-mode: bicubic; /* 2 */
+}
+
+/**
+ * Correct overflow displayed oddly in IE 9.
+ */
+
+svg:not(:root) {
+ overflow: hidden;
+}
+
+/* ==========================================================================
+ Figures
+ ========================================================================== */
+
+/**
+ * Address margin not present in IE 6/7/8/9, Safari 5, and Opera 11.
+ */
+
+figure {
+ margin: 0;
+}
+
+/* ==========================================================================
+ Forms
+ ========================================================================== */
+
+/**
+ * Correct margin displayed oddly in IE 6/7.
+ */
+
+form {
+ margin: 0;
+}
+
+/**
+ * Define consistent border, margin, and padding.
+ */
+
+fieldset {
+ border: 1px solid #c0c0c0;
+ margin: 0 2px;
+ padding: 0.35em 0.625em 0.75em;
+}
+
+/**
+ * 1. Correct color not being inherited in IE 6/7/8/9.
+ * 2. Correct text not wrapping in Firefox 3.
+ * 3. Correct alignment displayed oddly in IE 6/7.
+ */
+
+legend {
+ border: 0; /* 1 */
+ padding: 0;
+ white-space: normal; /* 2 */
+ *margin-left: -7px; /* 3 */
+}
+
+/**
+ * 1. Correct font size not being inherited in all browsers.
+ * 2. Address margins set differently in IE 6/7, Firefox 3+, Safari 5,
+ * and Chrome.
+ * 3. Improve appearance and consistency in all browsers.
+ */
+
+button,
+input,
+select,
+textarea {
+ font-size: 100%; /* 1 */
+ margin: 0; /* 2 */
+ vertical-align: baseline; /* 3 */
+ *vertical-align: middle; /* 3 */
+}
+
+/**
+ * Address Firefox 3+ setting `line-height` on `input` using `!important` in
+ * the UA stylesheet.
+ */
+
+button,
+input {
+ line-height: normal;
+}
+
+/**
+ * Address inconsistent `text-transform` inheritance for `button` and `select`.
+ * All other form control elements do not inherit `text-transform` values.
+ * Correct `button` style inheritance in Chrome, Safari 5+, and IE 6+.
+ * Correct `select` style inheritance in Firefox 4+ and Opera.
+ */
+
+button,
+select {
+ text-transform: none;
+}
+
+/**
+ * 1. Avoid the WebKit bug in Android 4.0.* where (2) destroys native `audio`
+ * and `video` controls.
+ * 2. Correct inability to style clickable `input` types in iOS.
+ * 3. Improve usability and consistency of cursor style between image-type
+ * `input` and others.
+ * 4. Remove inner spacing in IE 7 without affecting normal text inputs.
+ * Known issue: inner spacing remains in IE 6.
+ */
+
+button,
+html input[type="button"], /* 1 */
+input[type="reset"],
+input[type="submit"] {
+ -webkit-appearance: button; /* 2 */
+ cursor: pointer; /* 3 */
+ *overflow: visible; /* 4 */
+}
+
+/**
+ * Re-set default cursor for disabled elements.
+ */
+
+button[disabled],
+html input[disabled] {
+ cursor: default;
+}
+
+/**
+ * 1. Address box sizing set to content-box in IE 8/9.
+ * 2. Remove excess padding in IE 8/9.
+ * 3. Remove excess padding in IE 7.
+ * Known issue: excess padding remains in IE 6.
+ */
+
+input[type="checkbox"],
+input[type="radio"] {
+ box-sizing: border-box; /* 1 */
+ padding: 0; /* 2 */
+ *height: 13px; /* 3 */
+ *width: 13px; /* 3 */
+}
+
+/**
+ * 1. Address `appearance` set to `searchfield` in Safari 5 and Chrome.
+ * 2. Address `box-sizing` set to `border-box` in Safari 5 and Chrome
+ * (include `-moz` to future-proof).
+ */
+
+input[type="search"] {
+ -webkit-appearance: textfield; /* 1 */
+ -moz-box-sizing: content-box;
+ -webkit-box-sizing: content-box; /* 2 */
+ box-sizing: content-box;
+}
+
+/**
+ * Remove inner padding and search cancel button in Safari 5 and Chrome
+ * on OS X.
+ */
+
+input[type="search"]::-webkit-search-cancel-button,
+input[type="search"]::-webkit-search-decoration {
+ -webkit-appearance: none;
+}
+
+/**
+ * Remove inner padding and border in Firefox 3+.
+ */
+
+button::-moz-focus-inner,
+input::-moz-focus-inner {
+ border: 0;
+ padding: 0;
+}
+
+/**
+ * 1. Remove default vertical scrollbar in IE 6/7/8/9.
+ * 2. Improve readability and alignment in all browsers.
+ */
+
+textarea {
+ overflow: auto; /* 1 */
+ vertical-align: top; /* 2 */
+}
+
+/* ==========================================================================
+ Tables
+ ========================================================================== */
+
+/**
+ * Remove most spacing between table cells.
+ */
+
+table {
+ border-collapse: collapse;
+ border-spacing: 0;
+}
+
+ </style>
+
+ <style type="text/css">
+
+ html, body {
+ margin: 0;
+ padding: 0;
+ min-height: 100%;
+ }
+ body {
+ background: #fff;
+ font-family: "Source Sans Pro", "Helvetica Neueue", Helvetica, sans;
+ font-weight: 300;
+ font-size: 16px;
+ line-height: 1.6em;
+ }
+ #content {
+ width: 70%;
+ max-width: 850px;
+ float: left;
+ padding: 30px 60px;
+ border-left: 1px solid #ddd;
+ }
+ #sidebar {
+ width: 25%;
+ float: left;
+ padding: 30px;
+ overflow: hidden;
+ }
+ #nav {
+ font-size: 130%;
+ margin: 0 0 15px 0;
+ }
+
+ #top {
+ display: block;
+ position: fixed;
+ bottom: 5px;
+ left: 5px;
+ font-size: .85em;
+ text-transform: uppercase;
+ }
+
+ #footer {
+ font-size: .75em;
+ padding: 5px 30px;
+ border-top: 1px solid #ddd;
+ text-align: right;
+ }
+ #footer p {
+ margin: 0 0 0 30px;
+ display: inline-block;
+ }
+
+ h1, h2, h3, h4, h5 {
+ font-weight: 300;
+ }
+ h1 {
+ font-size: 2.5em;
+ line-height: 1.1em;
+ margin: 0 0 .50em 0;
+ }
+
+ h2 {
+ font-size: 1.75em;
+ margin: 1em 0 .50em 0;
+ }
+
+ h3 {
+ margin: 25px 0 10px 0;
+ }
+
+ h4 {
+ margin: 0;
+ font-size: 105%;
+ }
+
+ a {
+ color: #058;
+ text-decoration: none;
+ transition: color .3s ease-in-out;
+ }
+
+ a:hover {
+ color: #e08524;
+ transition: color .3s ease-in-out;
+ }
+
+ pre, code, .mono, .name {
+ font-family: "Ubuntu Mono", "Cousine", "DejaVu Sans Mono", monospace;
+ }
+
+ .title .name {
+ font-weight: bold;
+ }
+ .section-title {
+ margin-top: 2em;
+ }
+ .ident {
+ color: #900;
+ }
+
+ code {
+ background: #f9f9f9;
+ }
+
+ pre {
+ background: #fefefe;
+ border: 1px solid #ddd;
+ box-shadow: 2px 2px 0 #f3f3f3;
+ margin: 0 30px;
+ padding: 15px 30px;
+ }
+
+ .codehilite {
+ margin: 0 30px 10px 30px;
+ }
+
+ .codehilite pre {
+ margin: 0;
+ }
+ .codehilite .err { background: #ff3300; color: #fff !important; }
+
+ table#module-list {
+ font-size: 110%;
+ }
+
+ table#module-list tr td:first-child {
+ padding-right: 10px;
+ white-space: nowrap;
+ }
+
+ table#module-list td {
+ vertical-align: top;
+ padding-bottom: 8px;
+ }
+
+ table#module-list td p {
+ margin: 0 0 7px 0;
+ }
+
+ .def {
+ display: table;
+ }
+
+ .def p {
+ display: table-cell;
+ vertical-align: top;
+ text-align: left;
+ }
+
+ .def p:first-child {
+ white-space: nowrap;
+ }
+
+ .def p:last-child {
+ width: 100%;
+ }
+
+
+ #index {
+ list-style-type: none;
+ margin: 0;
+ padding: 0;
+ }
+ ul#index .class_name {
+ /* font-size: 110%; */
+ font-weight: bold;
+ }
+ #index ul {
+ margin: 0;
+ }
+
+ .item {
+ margin: 0 0 15px 0;
+ }
+
+ .item .class {
+ margin: 0 0 25px 30px;
+ }
+
+ .item .class ul.class_list {
+ margin: 0 0 20px 0;
+ }
+
+ .item .name {
+ background: #fafafa;
+ margin: 0;
+ font-weight: bold;
+ padding: 5px 10px;
+ border-radius: 3px;
+ display: inline-block;
+ min-width: 40%;
+ }
+ .item .name:hover {
+ background: #f6f6f6;
+ }
+
+ .item .empty_desc {
+ margin: 0 0 5px 0;
+ padding: 0;
+ }
+
+ .item .inheritance {
+ margin: 3px 0 0 30px;
+ }
+
+ .item .inherited {
+ color: #666;
+ }
+
+ .item .desc {
+ padding: 0 8px;
+ margin: 0;
+ }
+
+ .item .desc p {
+ margin: 0 0 10px 0;
+ }
+
+ .source_cont {
+ margin: 0;
+ padding: 0;
+ }
+
+ .source_link a {
+ background: #ffc300;
+ font-weight: 400;
+ font-size: .75em;
+ text-transform: uppercase;
+ color: #fff;
+ text-shadow: 1px 1px 0 #f4b700;
+
+ padding: 3px 8px;
+ border-radius: 2px;
+ transition: background .3s ease-in-out;
+ }
+ .source_link a:hover {
+ background: #FF7200;
+ text-shadow: none;
+ transition: background .3s ease-in-out;
+ }
+
+ .source {
+ display: none;
+ max-height: 600px;
+ overflow-y: scroll;
+ margin-bottom: 15px;
+ }
+
+ .source .codehilite {
+ margin: 0;
+ }
+
+ .desc h1, .desc h2, .desc h3 {
+ font-size: 100% !important;
+ }
+ .clear {
+ clear: both;
+ }
+
+ @media all and (max-width: 950px) {
+ #sidebar {
+ width: 35%;
+ }
+ #content {
+ width: 65%;
+ }
+ }
+ @media all and (max-width: 650px) {
+ #top {
+ display: none;
+ }
+ #sidebar {
+ float: none;
+ width: auto;
+ }
+ #content {
+ float: none;
+ width: auto;
+ padding: 30px;
+ }
+
+ #index ul {
+ padding: 0;
+ margin-bottom: 15px;
+ }
+ #index ul li {
+ display: inline-block;
+ margin-right: 30px;
+ }
+ #footer {
+ text-align: left;
+ }
+ #footer p {
+ display: block;
+ margin: inherit;
+ }
+ }
+
+ /*****************************/
+
+ </style>
+
+
+ <style type="text/css">
+
+/* ==========================================================================
+ EXAMPLE Media Queries for Responsive Design.
+ These examples override the primary ('mobile first') styles.
+ Modify as content requires.
+ ========================================================================== */
+
+@media only screen and (min-width: 35em) {
+ /* Style adjustments for viewports that meet the condition */
+}
+
+@media print,
+ (-o-min-device-pixel-ratio: 5/4),
+ (-webkit-min-device-pixel-ratio: 1.25),
+ (min-resolution: 120dpi) {
+ /* Style adjustments for high resolution devices */
+}
+
+/* ==========================================================================
+ Print styles.
+ Inlined to avoid required HTTP connection: h5bp.com/r
+ ========================================================================== */
+
+@media print {
+ * {
+ background: transparent !important;
+ color: #000 !important; /* Black prints faster: h5bp.com/s */
+ box-shadow: none !important;
+ text-shadow: none !important;
+ }
+
+ a,
+ a:visited {
+ text-decoration: underline;
+ }
+
+ a[href]:after {
+ content: " (" attr(href) ")";
+ }
+
+ abbr[title]:after {
+ content: " (" attr(title) ")";
+ }
+
+ /*
+ * Don't show links for images, or javascript/internal links
+ */
+
+ .ir a:after,
+ a[href^="javascript:"]:after,
+ a[href^="#"]:after {
+ content: "";
+ }
+
+ pre,
+ blockquote {
+ border: 1px solid #999;
+ page-break-inside: avoid;
+ }
+
+ thead {
+ display: table-header-group; /* h5bp.com/t */
+ }
+
+ tr,
+ img {
+ page-break-inside: avoid;
+ }
+
+ img {
+ max-width: 100% !important;
+ }
+
+ @page {
+ margin: 0.5cm;
+ }
+
+ p,
+ h2,
+ h3 {
+ orphans: 3;
+ widows: 3;
+ }
+
+ h2,
+ h3 {
+ page-break-after: avoid;
+ }
+}
+
+ </style>
+
+ <script type="text/javascript">
+ function toggle(id, $link) {
+ $node = document.getElementById(id);
+ if (!$node)
+ return;
+ if (!$node.style.display || $node.style.display == 'none') {
+ $node.style.display = 'block';
+ $link.innerHTML = 'Hide source ≢';
+ } else {
+ $node.style.display = 'none';
+ $link.innerHTML = 'Show source ≡';
+ }
+ }
+ </script>
+</head>
+<body>
+<a href="#" id="top">Top</a>
+
+<div id="container">
+
+
+ <div id="sidebar">
+ <h1>Index</h1>
+ <ul id="index">
+
+
+ <li class="set"><h3><a href="#header-classes">Classes</a></h3>
+ <ul>
+ <li class="mono">
+ <span class="class_name"><a href="#heronpy.api.bolt.bolt.Bolt">Bolt</a></span>
+
+
+ <ul>
+ <li class="mono"><a href="#heronpy.api.bolt.bolt.Bolt.is_tick">is_tick</a></li>
+ <li class="mono"><a href="#heronpy.api.bolt.bolt.Bolt.__init__">__init__</a></li>
+ <li class="mono"><a href="#heronpy.api.bolt.bolt.Bolt.ack">ack</a></li>
+ <li class="mono"><a href="#heronpy.api.bolt.bolt.Bolt.emit">emit</a></li>
+ <li class="mono"><a href="#heronpy.api.bolt.bolt.Bolt.fail">fail</a></li>
+ <li class="mono"><a href="#heronpy.api.bolt.bolt.Bolt.initialize">initialize</a></li>
+ <li class="mono"><a href="#heronpy.api.bolt.bolt.Bolt.log">log</a></li>
+ <li class="mono"><a href="#heronpy.api.bolt.bolt.Bolt.process">process</a></li>
+ <li class="mono"><a href="#heronpy.api.bolt.bolt.Bolt.process_tick">process_tick</a></li>
+ <li class="mono"><a href="#heronpy.api.bolt.bolt.Bolt.spec">spec</a></li>
+ </ul>
+
+ </li>
+ </ul>
+ </li>
+
+ </ul>
+ </div>
+
+ <article id="content">
+
+
+
+
+
+
+ <header id="section-intro">
+ <h1 class="title"><span class="name">heronpy.api.bolt.bolt</span> module</h1>
+ <p>bolt.py: API for defining bolt in python</p>
+
+ <p class="source_link"><a href="javascript:void(0);" onclick="toggle('source-heronpy.api.bolt.bolt', this);">Show source ≡</a></p>
+ <div id="source-heronpy.api.bolt.bolt" class="source">
+ <pre><code># Copyright 2016 - Parsely, Inc. (d/b/a Parse.ly)
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+'''bolt.py: API for defining bolt in python'''
+from abc import abstractmethod
+from heronpy.api.bolt.base_bolt import BaseBolt
+
+class Bolt(BaseBolt):
+ """API for defining a bolt for Heron in Python
+
+ Topology writers need to inherit this ``Bolt`` class to define their own custom bolt, by
+ implementing ``initialize()`` and ``process()`` methods.
+ """
+
+ @abstractmethod
+ def initialize(self, config, context):
+ """Called when a task for this component is initialized within a worker on the cluster
+
+ It provides the bolt with the environment in which the bolt executes.
+ Note that ``__init__()`` should not be overriden for initialization of a bolt, as it is used
+ internally by BaseBolt; instead, ``initialize()`` should be used to initialize any custom
+ variables or connection to databases.
+
+ :type config: dict
+ :param config: The Heron configuration for this bolt. This is the configuration provided to
+ the topology merged in with cluster configuration on this machine.
+ Note that types of string values in the config have been automatically converted,
+ meaning that number strings and boolean strings are converted to the appropriate
+ types.
+ :type context: :class:`TopologyContext`
+ :param context: This object can be used to get information about this task's place within the
+ topology, including the task id and component id of this task, input and output
+ information, etc.
+ """
+ pass
+
+ @abstractmethod
+ def process(self, tup):
+ """Process a single tuple of input
+
+ The Tuple object contains metadata on it about which component/stream/task it came from.
+ To emit a tuple, call ``self.emit(tuple)``.
+ Note that tick tuples are not passed to this method, as the ``process_tick()`` method is
+ responsible for processing them.
+
+ **Must be implemented by a subclass, otherwise NotImplementedError is raised.**
+
+ :type tup: :class:`Tuple`
+ :param tup: Tuple to process
+ """
+ raise NotImplementedError("Bolt not implementing process() method.")
+
+ @abstractmethod
+ def process_tick(self, tup):
+ """Process special tick tuple
+
+ It is compatible with StreamParse API.
+
+ Tick tuples allow time-based behavior to be included in bolts. They will be sent to the bolts
+ for which `topology.tick.tuple.freq.secs``
+ (or, ``.api_constants.TOPOLOGY_TICK_TUPLE_FREQ_SECS`` key) is set to an integer value, the
+ number of seconds.
+
+ Default behavior is to ignore tick tuples. This method should be overridden by subclasses
+ if you want to react to timer events via tick tuples.
+
+ :type tup: :class:`Tuple`
+ :param tup: the tick tuple to be processed
+ """
+ pass
+</code></pre>
+ </div>
+
+ </header>
+
+ <section id="section-items">
+
+
+ <h2 class="section-title" id="header-classes">Classes</h2>
+
+ <div class="item">
+ <p id="heronpy.api.bolt.bolt.Bolt" class="name">class <span class="ident">Bolt</span></p>
+
+
+ <div class="desc"><p>API for defining a bolt for Heron in Python</p>
+<p>Topology writers need to inherit this <code>Bolt</code> class to define their own custom bolt, by
+implementing <code>initialize()</code> and <code>process()</code> methods.</p></div>
+ <div class="source_cont">
+ <p class="source_link"><a href="javascript:void(0);" onclick="toggle('source-heronpy.api.bolt.bolt.Bolt', this);">Show source ≡</a></p>
+ <div id="source-heronpy.api.bolt.bolt.Bolt" class="source">
+ <pre><code>class Bolt(BaseBolt):
+ """API for defining a bolt for Heron in Python
+
+ Topology writers need to inherit this ``Bolt`` class to define their own custom bolt, by
+ implementing ``initialize()`` and ``process()`` methods.
+ """
+
+ @abstractmethod
+ def initialize(self, config, context):
+ """Called when a task for this component is initialized within a worker on the cluster
+
+ It provides the bolt with the environment in which the bolt executes.
+ Note that ``__init__()`` should not be overriden for initialization of a bolt, as it is used
+ internally by BaseBolt; instead, ``initialize()`` should be used to initialize any custom
+ variables or connection to databases.
+
+ :type config: dict
+ :param config: The Heron configuration for this bolt. This is the configuration provided to
+ the topology merged in with cluster configuration on this machine.
+ Note that types of string values in the config have been automatically converted,
+ meaning that number strings and boolean strings are converted to the appropriate
+ types.
+ :type context: :class:`TopologyContext`
+ :param context: This object can be used to get information about this task's place within the
+ topology, including the task id and component id of this task, input and output
+ information, etc.
+ """
+ pass
+
+ @abstractmethod
+ def process(self, tup):
+ """Process a single tuple of input
+
+ The Tuple object contains metadata on it about which component/stream/task it came from.
+ To emit a tuple, call ``self.emit(tuple)``.
+ Note that tick tuples are not passed to this method, as the ``process_tick()`` method is
+ responsible for processing them.
+
+ **Must be implemented by a subclass, otherwise NotImplementedError is raised.**
+
+ :type tup: :class:`Tuple`
+ :param tup: Tuple to process
+ """
+ raise NotImplementedError("Bolt not implementing process() method.")
+
+ @abstractmethod
+ def process_tick(self, tup):
+ """Process special tick tuple
+
+ It is compatible with StreamParse API.
+
+ Tick tuples allow time-based behavior to be included in bolts. They will be sent to the bolts
+ for which `topology.tick.tuple.freq.secs``
+ (or, ``.api_constants.TOPOLOGY_TICK_TUPLE_FREQ_SECS`` key) is set to an integer value, the
+ number of seconds.
+
+ Default behavior is to ignore tick tuples. This method should be overridden by subclasses
+ if you want to react to timer events via tick tuples.
+
+ :type tup: :class:`Tuple`
+ :param tup: the tick tuple to be processed
+ """
+ pass
+</code></pre>
+ </div>
+</div>
+
+
+ <div class="class">
+ <h3>Ancestors (in MRO)</h3>
+ <ul class="class_list">
+ <li><a href="#heronpy.api.bolt.bolt.Bolt">Bolt</a></li>
+ <li>heronpy.api.bolt.base_bolt.BaseBolt</li>
+ <li>heronpy.api.component.base_component.BaseComponent</li>
+ <li>__builtin__.object</li>
+ </ul>
+ <h3>Static methods</h3>
+
+ <div class="item">
+ <div class="name def" id="heronpy.api.bolt.bolt.Bolt.is_tick">
+ <p>def <span class="ident">is_tick</span>(</p><p>tup)</p>
+ </div>
+
+
+
+
+ <div class="desc"><p>Returns whether or not a given HeronTuple is a tick Tuple</p>
+<p>It is compatible with StreamParse API.</p></div>
+ <div class="source_cont">
+ <p class="source_link"><a href="javascript:void(0);" onclick="toggle('source-heronpy.api.bolt.bolt.Bolt.is_tick', this);">Show source ≡</a></p>
+ <div id="source-heronpy.api.bolt.bolt.Bolt.is_tick" class="source">
+ <pre><code>@staticmethod
+def is_tick(tup):
+ """Returns whether or not a given HeronTuple is a tick Tuple
+ It is compatible with StreamParse API.
+ """
+ return tup.stream == TupleHelper.TICK_TUPLE_ID
+</code></pre>
+ </div>
+</div>
+
+ </div>
+
+ <h3>Methods</h3>
+
+ <div class="item">
+ <div class="name def" id="heronpy.api.bolt.bolt.Bolt.__init__">
+ <p>def <span class="ident">__init__</span>(</p><p>self, delegate)</p>
+ </div>
+
+
+
+
+ <div class="desc"><p>Initializes BaseComponent</p>
+<p>:param delegate: SpoutInstance or BoltInstance</p></div>
+ <div class="source_cont">
+ <p class="source_link"><a href="javascript:void(0);" onclick="toggle('source-heronpy.api.bolt.bolt.Bolt.__init__', this);">Show source ≡</a></p>
+ <div id="source-heronpy.api.bolt.bolt.Bolt.__init__" class="source">
+ <pre><code>def __init__(self, delegate):
+ """Initializes BaseComponent
+ :param delegate: SpoutInstance or BoltInstance
+ """
+ self.delegate = delegate
+ self.logger = self.delegate.logger
+</code></pre>
+ </div>
+</div>
+
+ </div>
+
+
+ <div class="item">
+ <div class="name def" id="heronpy.api.bolt.bolt.Bolt.ack">
+ <p>def <span class="ident">ack</span>(</p><p>self, tup)</p>
+ </div>
+
+
+
+
+ <div class="desc"><p>Indicate that processing of a Tuple has succeeded</p>
+<p>It is compatible with StreamParse API.</p></div>
+ <div class="source_cont">
+ <p class="source_link"><a href="javascript:void(0);" onclick="toggle('source-heronpy.api.bolt.bolt.Bolt.ack', this);">Show source ≡</a></p>
+ <div id="source-heronpy.api.bolt.bolt.Bolt.ack" class="source">
+ <pre><code>def ack(self, tup):
+ """Indicate that processing of a Tuple has succeeded
+ It is compatible with StreamParse API.
+ """
+ self.delegate.ack(tup)
+</code></pre>
+ </div>
+</div>
+
+ </div>
+
+
+ <div class="item">
+ <div class="name def" id="heronpy.api.bolt.bolt.Bolt.emit">
+ <p>def <span class="ident">emit</span>(</p><p>self, tup, stream='default', anchors=None, direct_task=None, need_task_ids=False)</p>
+ </div>
+
+
+
+
+ <div class="desc"><p>Emits a new tuple from this Bolt</p>
+<p>It is compatible with StreamParse API.</p>
+<p>:type tup: list or tuple
+:param tup: the new output Tuple to send from this bolt,
+ which should contain only serializable data.
+:type stream: str
+:param stream: the ID of the stream to emit this Tuple to.
+ Leave empty to emit to the default stream.
+:type anchors: list
+:param anchors: a list of HeronTuples to which the emitted Tuples should be anchored.
+:type direct_task: int
+:param direct_task: the task to send the Tuple to if performing a direct emit.
+:type need_task_ids: bool
+:param need_task_ids: indicate whether or not you would like the task IDs the Tuple was emitted.</p></div>
+ <div class="source_cont">
+ <p class="source_link"><a href="javascript:void(0);" onclick="toggle('source-heronpy.api.bolt.bolt.Bolt.emit', this);">Show source ≡</a></p>
+ <div id="source-heronpy.api.bolt.bolt.Bolt.emit" class="source">
+ <pre><code>def emit(self, tup, stream=Stream.DEFAULT_STREAM_ID,
+ anchors=None, direct_task=None, need_task_ids=False):
+ """Emits a new tuple from this Bolt
+ It is compatible with StreamParse API.
+ :type tup: list or tuple
+ :param tup: the new output Tuple to send from this bolt,
+ which should contain only serializable data.
+ :type stream: str
+ :param stream: the ID of the stream to emit this Tuple to.
+ Leave empty to emit to the default stream.
+ :type anchors: list
+ :param anchors: a list of HeronTuples to which the emitted Tuples should be anchored.
+ :type direct_task: int
+ :param direct_task: the task to send the Tuple to if performing a direct emit.
+ :type need_task_ids: bool
+ :param need_task_ids: indicate whether or not you would like the task IDs the Tuple was emitted.
+ """
+ self.delegate.emit(tup, stream, anchors, direct_task, need_task_ids)
+</code></pre>
+ </div>
+</div>
+
+ </div>
+
+
+ <div class="item">
+ <div class="name def" id="heronpy.api.bolt.bolt.Bolt.fail">
+ <p>def <span class="ident">fail</span>(</p><p>self, tup)</p>
+ </div>
+
+
+
+
+ <div class="desc"><p>Indicate that processing of a Tuple has failed</p>
+<p>It is compatible with StreamParse API.</p></div>
+ <div class="source_cont">
+ <p class="source_link"><a href="javascript:void(0);" onclick="toggle('source-heronpy.api.bolt.bolt.Bolt.fail', this);">Show source ≡</a></p>
+ <div id="source-heronpy.api.bolt.bolt.Bolt.fail" class="source">
+ <pre><code>def fail(self, tup):
+ """Indicate that processing of a Tuple has failed
+ It is compatible with StreamParse API.
+ """
+ self.delegate.fail(tup)
+</code></pre>
+ </div>
+</div>
+
+ </div>
+
+
+ <div class="item">
+ <div class="name def" id="heronpy.api.bolt.bolt.Bolt.initialize">
+ <p>def <span class="ident">initialize</span>(</p><p>self, config, context)</p>
+ </div>
+
+
+
+
+ <div class="desc"><p>Called when a task for this component is initialized within a worker on the cluster</p>
+<p>It provides the bolt with the environment in which the bolt executes.
+Note that <code>__init__()</code> should not be overriden for initialization of a bolt, as it is used
+internally by BaseBolt; instead, <code>initialize()</code> should be used to initialize any custom
+variables or connection to databases.</p>
+<p>:type config: dict
+:param config: The Heron configuration for this bolt. This is the configuration provided to
+ the topology merged in with cluster configuration on this machine.
+ Note that types of string values in the config have been automatically converted,
+ meaning that number strings and boolean strings are converted to the appropriate
+ types.
+:type context: :class:<code>TopologyContext</code>
+:param context: This object can be used to get information about this task's place within the
+ topology, including the task id and component id of this task, input and output
+ information, etc.</p></div>
+ <div class="source_cont">
+ <p class="source_link"><a href="javascript:void(0);" onclick="toggle('source-heronpy.api.bolt.bolt.Bolt.initialize', this);">Show source ≡</a></p>
+ <div id="source-heronpy.api.bolt.bolt.Bolt.initialize" class="source">
+ <pre><code>@abstractmethod
+def initialize(self, config, context):
+ """Called when a task for this component is initialized within a worker on the cluster
+ It provides the bolt with the environment in which the bolt executes.
+ Note that ``__init__()`` should not be overriden for initialization of a bolt, as it is used
+ internally by BaseBolt; instead, ``initialize()`` should be used to initialize any custom
+ variables or connection to databases.
+ :type config: dict
+ :param config: The Heron configuration for this bolt. This is the configuration provided to
+ the topology merged in with cluster configuration on this machine.
+ Note that types of string values in the config have been automatically converted,
+ meaning that number strings and boolean strings are converted to the appropriate
+ types.
+ :type context: :class:`TopologyContext`
+ :param context: This object can be used to get information about this task's place within the
+ topology, including the task id and component id of this task, input and output
+ information, etc.
+ """
+ pass
+</code></pre>
+ </div>
+</div>
+
+ </div>
+
+
+ <div class="item">
+ <div class="name def" id="heronpy.api.bolt.bolt.Bolt.log">
+ <p>def <span class="ident">log</span>(</p><p>self, message, level=None)</p>
+ </div>
+
+
+
+
+ <div class="desc"><p>Log message, optionally providing a logging level</p>
+<p>:type message: str
+:param message: the log message to send
+:type level: str
+:param level: the logging level,
+ one of: trace (=debug), debug, info, warn or error (default: info)</p></div>
+ <div class="source_cont">
+ <p class="source_link"><a href="javascript:void(0);" onclick="toggle('source-heronpy.api.bolt.bolt.Bolt.log', this);">Show source ≡</a></p>
+ <div id="source-heronpy.api.bolt.bolt.Bolt.log" class="source">
+ <pre><code>def log(self, message, level=None):
+ """Log message, optionally providing a logging level
+ :type message: str
+ :param message: the log message to send
+ :type level: str
+ :param level: the logging level,
+ one of: trace (=debug), debug, info, warn or error (default: info)
+ """
+ self.delegate.log(message, level)
+</code></pre>
+ </div>
+</div>
+
+ </div>
+
+
+ <div class="item">
+ <div class="name def" id="heronpy.api.bolt.bolt.Bolt.process">
+ <p>def <span class="ident">process</span>(</p><p>self, tup)</p>
+ </div>
+
+
+
+
+ <div class="desc"><p>Process a single tuple of input</p>
+<p>The Tuple object contains metadata on it about which component/stream/task it came from.
+To emit a tuple, call <code>self.emit(tuple)</code>.
+Note that tick tuples are not passed to this method, as the <code>process_tick()</code> method is
+responsible for processing them.</p>
+<p><strong>Must be implemented by a subclass, otherwise NotImplementedError is raised.</strong></p>
+<p>:type tup: :class:<code>Tuple</code>
+:param tup: Tuple to process</p></div>
+ <div class="source_cont">
+ <p class="source_link"><a href="javascript:void(0);" onclick="toggle('source-heronpy.api.bolt.bolt.Bolt.process', this);">Show source ≡</a></p>
+ <div id="source-heronpy.api.bolt.bolt.Bolt.process" class="source">
+ <pre><code>@abstractmethod
+def process(self, tup):
+ """Process a single tuple of input
+ The Tuple object contains metadata on it about which component/stream/task it came from.
+ To emit a tuple, call ``self.emit(tuple)``.
+ Note that tick tuples are not passed to this method, as the ``process_tick()`` method is
+ responsible for processing them.
+ **Must be implemented by a subclass, otherwise NotImplementedError is raised.**
+ :type tup: :class:`Tuple`
+ :param tup: Tuple to process
+ """
+ raise NotImplementedError("Bolt not implementing process() method.")
+</code></pre>
+ </div>
+</div>
+
+ </div>
+
+
+ <div class="item">
+ <div class="name def" id="heronpy.api.bolt.bolt.Bolt.process_tick">
+ <p>def <span class="ident">process_tick</span>(</p><p>self, tup)</p>
+ </div>
+
+
+
+
+ <div class="desc"><p>Process special tick tuple</p>
+<p>It is compatible with StreamParse API.</p>
+<p>Tick tuples allow time-based behavior to be included in bolts. They will be sent to the bolts
+for which `topology.tick.tuple.freq.secs<code>(or,</code>.api_constants.TOPOLOGY_TICK_TUPLE_FREQ_SECS`` key) is set to an integer value, the
+number of seconds.</p>
+<p>Default behavior is to ignore tick tuples. This method should be overridden by subclasses
+if you want to react to timer events via tick tuples.</p>
+<p>:type tup: :class:<code>Tuple</code>
+:param tup: the tick tuple to be processed</p></div>
+ <div class="source_cont">
+ <p class="source_link"><a href="javascript:void(0);" onclick="toggle('source-heronpy.api.bolt.bolt.Bolt.process_tick', this);">Show source ≡</a></p>
+ <div id="source-heronpy.api.bolt.bolt.Bolt.process_tick" class="source">
+ <pre><code>@abstractmethod
+def process_tick(self, tup):
+ """Process special tick tuple
+ It is compatible with StreamParse API.
+ Tick tuples allow time-based behavior to be included in bolts. They will be sent to the bolts
+ for which `topology.tick.tuple.freq.secs``
+ (or, ``.api_constants.TOPOLOGY_TICK_TUPLE_FREQ_SECS`` key) is set to an integer value, the
+ number of seconds.
+ Default behavior is to ignore tick tuples. This method should be overridden by subclasses
+ if you want to react to timer events via tick tuples.
+ :type tup: :class:`Tuple`
+ :param tup: the tick tuple to be processed
+ """
+ pass
+</code></pre>
+ </div>
+</div>
+
+ </div>
+
+
+ <div class="item">
+ <div class="name def" id="heronpy.api.bolt.bolt.Bolt.spec">
+ <p>def <span class="ident">spec</span>(</p><p>cls, name=None, inputs=None, par=1, config=None, optional_outputs=None)</p>
+ </div>
+
+
+
+
+ <div class="desc"><p>Register this bolt to the topology and create <code>HeronComponentSpec</code></p>
+<p>This method takes an optional <code>outputs</code> argument for supporting dynamic output fields
+declaration. However, it is recommended that <code>outputs</code> should be declared as
+an attribute of your <code>Bolt</code> subclass. Also, some ways of declaring inputs is not supported
+in this implementation; please read the documentation below.</p>
+<p>:type name: str
+:param name: Name of this bolt.
+:type inputs: dict or list
+:param inputs: Streams that feed into this Bolt.</p>
+<pre><code> Two forms of this are acceptable:
+
+ 1. A `dict` mapping from ``HeronComponentSpec`` to ``Grouping``.
+ In this case, default stream is used.
+ 2. A `dict` mapping from ``GlobalStreamId`` to ``Grouping``.
+ This ``GlobalStreamId`` object itself is different from StreamParse, because
+ Heron does not use thrift, although its constructor method is compatible.
+ 3. A `list` of ``HeronComponentSpec``. In this case, default stream with
+ SHUFFLE grouping is used.
+ 4. A `list` of ``GlobalStreamId``. In this case, SHUFFLE grouping is used.
+</code></pre>
+<p>:type par: int
+:param par: Parallelism hint for this spout.
+:type config: dict
+:param config: Component-specific config settings.
+:type optional_outputs: list of (str or Stream) or tuple of (str or Stream)
+:param optional_outputs: Additional output fields for this bolt. These fields are added to
+ existing <code>outputs</code> class attributes of your bolt. This is an optional
+ argument, and exists only for supporting dynamic output field
+ declaration.</p></div>
+ <div class="source_cont">
+ <p class="source_link"><a href="javascript:void(0);" onclick="toggle('source-heronpy.api.bolt.bolt.Bolt.spec', this);">Show source ≡</a></p>
+ <div id="source-heronpy.api.bolt.bolt.Bolt.spec" class="source">
+ <pre><code>@classmethod
+def spec(cls, name=None, inputs=None, par=1, config=None, optional_outputs=None):
+ """Register this bolt to the topology and create ``HeronComponentSpec``
+ This method takes an optional ``outputs`` argument for supporting dynamic output fields
+ declaration. However, it is recommended that ``outputs`` should be declared as
+ an attribute of your ``Bolt`` subclass. Also, some ways of declaring inputs is not supported
+ in this implementation; please read the documentation below.
+ :type name: str
+ :param name: Name of this bolt.
+ :type inputs: dict or list
+ :param inputs: Streams that feed into this Bolt.
+ Two forms of this are acceptable:
+ 1. A `dict` mapping from ``HeronComponentSpec`` to ``Grouping``.
+ In this case, default stream is used.
+ 2. A `dict` mapping from ``GlobalStreamId`` to ``Grouping``.
+ This ``GlobalStreamId`` object itself is different from StreamParse, because
+ Heron does not use thrift, although its constructor method is compatible.
+ 3. A `list` of ``HeronComponentSpec``. In this case, default stream with
+ SHUFFLE grouping is used.
+ 4. A `list` of ``GlobalStreamId``. In this case, SHUFFLE grouping is used.
+ :type par: int
+ :param par: Parallelism hint for this spout.
+ :type config: dict
+ :param config: Component-specific config settings.
+ :type optional_outputs: list of (str or Stream) or tuple of (str or Stream)
+ :param optional_outputs: Additional output fields for this bolt. These fields are added to
+ existing ``outputs`` class attributes of your bolt. This is an optional
+ argument, and exists only for supporting dynamic output field
+ declaration.
+ """
+ python_class_path = "%s.%s" % (cls.__module__, cls.__name__)
+ if hasattr(cls, 'outputs'):
+ # avoid modification to cls.outputs
+ _outputs = copy.copy(cls.outputs)
+ else:
+ _outputs = []
+ if optional_outputs is not None:
+ assert isinstance(optional_outputs, (list, tuple))
+ for out in optional_outputs:
+ assert isinstance(out, (str, Stream))
+ _outputs.append(out)
+ return HeronComponentSpec(name, python_class_path, is_spout=False, par=par,
+ inputs=inputs, outputs=_outputs, config=config)
+</code></pre>
+ </div>
+</div>
+
+ </div>
+
+ </div>
+ </div>
+
+ </section>
+
+ </article>
+ <div class="clear"> </div>
+ <footer id="footer">
+ <p>
+ Documentation generated by
+ <a href="https://github.com/BurntSushi/pdoc">pdoc 0.3.2</a>
+ </p>
+
+ <p>pdoc is in the public domain with the
+ <a href="http://unlicense.org">UNLICENSE</a></p>
+
+ <p>Design by <a href="http://nadh.in">Kailash Nadh</a></p>
+ </footer>
+</div>
+</body>
+</html>
diff --git a/content/api/python/api/bolt/index.html b/content/api/python/api/bolt/index.html
index db300f4..8531d56 100644
--- a/content/api/python/api/bolt/index.html
+++ b/content/api/python/api/bolt/index.html
@@ -972,6 +972,13 @@ table {
+ <li class="set"><h3><a href="#header-submodules">Sub-modules</a></h3>
+ <ul>
+ <li class="mono"><a href="base_bolt.m.html">heronpy.api.bolt.base_bolt</a></li>
+ <li class="mono"><a href="bolt.m.html">heronpy.api.bolt.bolt</a></li>
+ <li class="mono"><a href="window_bolt.m.html">heronpy.api.bolt.window_bolt</a></li>
+ </ul>
+ </li>
</ul>
</div>
@@ -1015,6 +1022,28 @@ __import__('pkg_resources').declare_namespace(__name__)
+ <h2 class="section-title" id="header-submodules">Sub-modules</h2>
+ <div class="item">
+ <p class="name"><a href="base_bolt.m.html">heronpy.api.bolt.base_bolt</a></p>
+
+
+ <div class="desc"><p>base_bolt.py</p></div>
+
+ </div>
+ <div class="item">
+ <p class="name"><a href="bolt.m.html">heronpy.api.bolt.bolt</a></p>
+
+
+ <div class="desc"><p>bolt.py: API for defining bolt in python</p></div>
+
+ </div>
+ <div class="item">
+ <p class="name"><a href="window_bolt.m.html">heronpy.api.bolt.window_bolt</a></p>
+
+
+ <div class="desc"><p>window_bolt.py: API for defining windowed bolts in Heron</p></div>
+
+ </div>
</section>
</article>
diff --git a/content/api/python/api/bolt/window_bolt.m.html b/content/api/python/api/bolt/window_bolt.m.html
new file mode 100644
index 0000000..a960c3f
--- /dev/null
+++ b/content/api/python/api/bolt/window_bolt.m.html
@@ -0,0 +1,2425 @@
+<!doctype html>
+<head>
+ <meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
+ <meta name="viewport" content="width=device-width, initial-scale=1, minimum-scale=1" />
+
+ <title>heronpy.api.bolt.window_bolt API documentation</title>
+ <meta name="description" content="window_bolt.py: API for defining windowed bolts in Heron" />
+
+ <link href='http://fonts.googleapis.com/css?family=Source+Sans+Pro:400,300' rel='stylesheet' type='text/css'>
+
+ <style type="text/css">
+
+* {
+ box-sizing: border-box;
+}
+/*! normalize.css v1.1.1 | MIT License | git.io/normalize */
+
+/* ==========================================================================
+ HTML5 display definitions
+ ========================================================================== */
+
+/**
+ * Correct `block` display not defined in IE 6/7/8/9 and Firefox 3.
+ */
+
+article,
+aside,
+details,
+figcaption,
+figure,
+footer,
+header,
+hgroup,
+main,
+nav,
+section,
+summary {
+ display: block;
+}
+
+/**
+ * Correct `inline-block` display not defined in IE 6/7/8/9 and Firefox 3.
+ */
+
+audio,
+canvas,
+video {
+ display: inline-block;
+ *display: inline;
+ *zoom: 1;
+}
+
+/**
+ * Prevent modern browsers from displaying `audio` without controls.
+ * Remove excess height in iOS 5 devices.
+ */
+
+audio:not([controls]) {
+ display: none;
+ height: 0;
+}
+
+/**
+ * Address styling not present in IE 7/8/9, Firefox 3, and Safari 4.
+ * Known issue: no IE 6 support.
+ */
+
+[hidden] {
+ display: none;
+}
+
+/* ==========================================================================
+ Base
+ ========================================================================== */
+
+/**
+ * 1. Prevent system color scheme's background color being used in Firefox, IE,
+ * and Opera.
+ * 2. Prevent system color scheme's text color being used in Firefox, IE, and
+ * Opera.
+ * 3. Correct text resizing oddly in IE 6/7 when body `font-size` is set using
+ * `em` units.
+ * 4. Prevent iOS text size adjust after orientation change, without disabling
+ * user zoom.
+ */
+
+html {
+ background: #fff; /* 1 */
+ color: #000; /* 2 */
+ font-size: 100%; /* 3 */
+ -webkit-text-size-adjust: 100%; /* 4 */
+ -ms-text-size-adjust: 100%; /* 4 */
+}
+
+/**
+ * Address `font-family` inconsistency between `textarea` and other form
+ * elements.
+ */
+
+html,
+button,
+input,
+select,
+textarea {
+ font-family: sans-serif;
+}
+
+/**
+ * Address margins handled incorrectly in IE 6/7.
+ */
+
+body {
+ margin: 0;
+}
+
+/* ==========================================================================
+ Links
+ ========================================================================== */
+
+/**
+ * Address `outline` inconsistency between Chrome and other browsers.
+ */
+
+a:focus {
+ outline: thin dotted;
+}
+
+/**
+ * Improve readability when focused and also mouse hovered in all browsers.
+ */
+
+a:active,
+a:hover {
+ outline: 0;
+}
+
+/* ==========================================================================
+ Typography
+ ========================================================================== */
+
+/**
+ * Address font sizes and margins set differently in IE 6/7.
+ * Address font sizes within `section` and `article` in Firefox 4+, Safari 5,
+ * and Chrome.
+ */
+
+h1 {
+ font-size: 2em;
+ margin: 0.67em 0;
+}
+
+h2 {
+ font-size: 1.5em;
+ margin: 0.83em 0;
+}
+
+h3 {
+ font-size: 1.17em;
+ margin: 1em 0;
+}
+
+h4 {
+ font-size: 1em;
+ margin: 1.33em 0;
+}
+
+h5 {
+ font-size: 0.83em;
+ margin: 1.67em 0;
+}
+
+h6 {
+ font-size: 0.67em;
+ margin: 2.33em 0;
+}
+
+/**
+ * Address styling not present in IE 7/8/9, Safari 5, and Chrome.
+ */
+
+abbr[title] {
+ border-bottom: 1px dotted;
+}
+
+/**
+ * Address style set to `bolder` in Firefox 3+, Safari 4/5, and Chrome.
+ */
+
+b,
+strong {
+ font-weight: bold;
+}
+
+blockquote {
+ margin: 1em 40px;
+}
+
+/**
+ * Address styling not present in Safari 5 and Chrome.
+ */
+
+dfn {
+ font-style: italic;
+}
+
+/**
+ * Address differences between Firefox and other browsers.
+ * Known issue: no IE 6/7 normalization.
+ */
+
+hr {
+ -moz-box-sizing: content-box;
+ box-sizing: content-box;
+ height: 0;
+}
+
+/**
+ * Address styling not present in IE 6/7/8/9.
+ */
+
+mark {
+ background: #ff0;
+ color: #000;
+}
+
+/**
+ * Address margins set differently in IE 6/7.
+ */
+
+p,
+pre {
+ margin: 1em 0;
+}
+
+/**
+ * Correct font family set oddly in IE 6, Safari 4/5, and Chrome.
+ */
+
+code,
+kbd,
+pre,
+samp {
+ font-family: monospace, serif;
+ _font-family: 'courier new', monospace;
+ font-size: 1em;
+}
+
+/**
+ * Improve readability of pre-formatted text in all browsers.
+ */
+
+pre {
+ white-space: pre;
+ white-space: pre-wrap;
+ word-wrap: break-word;
+}
+
+/**
+ * Address CSS quotes not supported in IE 6/7.
+ */
+
+q {
+ quotes: none;
+}
+
+/**
+ * Address `quotes` property not supported in Safari 4.
+ */
+
+q:before,
+q:after {
+ content: '';
+ content: none;
+}
+
+/**
+ * Address inconsistent and variable font size in all browsers.
+ */
+
+small {
+ font-size: 80%;
+}
+
+/**
+ * Prevent `sub` and `sup` affecting `line-height` in all browsers.
+ */
+
+sub,
+sup {
+ font-size: 75%;
+ line-height: 0;
+ position: relative;
+ vertical-align: baseline;
+}
+
+sup {
+ top: -0.5em;
+}
+
+sub {
+ bottom: -0.25em;
+}
+
+/* ==========================================================================
+ Lists
+ ========================================================================== */
+
+/**
+ * Address margins set differently in IE 6/7.
+ */
+
+dl,
+menu,
+ol,
+ul {
+ margin: 1em 0;
+}
+
+dd {
+ margin: 0 0 0 40px;
+}
+
+/**
+ * Address paddings set differently in IE 6/7.
+ */
+
+menu,
+ol,
+ul {
+ padding: 0 0 0 40px;
+}
+
+/**
+ * Correct list images handled incorrectly in IE 7.
+ */
+
+nav ul,
+nav ol {
+ list-style: none;
+ list-style-image: none;
+}
+
+/* ==========================================================================
+ Embedded content
+ ========================================================================== */
+
+/**
+ * 1. Remove border when inside `a` element in IE 6/7/8/9 and Firefox 3.
+ * 2. Improve image quality when scaled in IE 7.
+ */
+
+img {
+ border: 0; /* 1 */
+ -ms-interpolation-mode: bicubic; /* 2 */
+}
+
+/**
+ * Correct overflow displayed oddly in IE 9.
+ */
+
+svg:not(:root) {
+ overflow: hidden;
+}
+
+/* ==========================================================================
+ Figures
+ ========================================================================== */
+
+/**
+ * Address margin not present in IE 6/7/8/9, Safari 5, and Opera 11.
+ */
+
+figure {
+ margin: 0;
+}
+
+/* ==========================================================================
+ Forms
+ ========================================================================== */
+
+/**
+ * Correct margin displayed oddly in IE 6/7.
+ */
+
+form {
+ margin: 0;
+}
+
+/**
+ * Define consistent border, margin, and padding.
+ */
+
+fieldset {
+ border: 1px solid #c0c0c0;
+ margin: 0 2px;
+ padding: 0.35em 0.625em 0.75em;
+}
+
+/**
+ * 1. Correct color not being inherited in IE 6/7/8/9.
+ * 2. Correct text not wrapping in Firefox 3.
+ * 3. Correct alignment displayed oddly in IE 6/7.
+ */
+
+legend {
+ border: 0; /* 1 */
+ padding: 0;
+ white-space: normal; /* 2 */
+ *margin-left: -7px; /* 3 */
+}
+
+/**
+ * 1. Correct font size not being inherited in all browsers.
+ * 2. Address margins set differently in IE 6/7, Firefox 3+, Safari 5,
+ * and Chrome.
+ * 3. Improve appearance and consistency in all browsers.
+ */
+
+button,
+input,
+select,
+textarea {
+ font-size: 100%; /* 1 */
+ margin: 0; /* 2 */
+ vertical-align: baseline; /* 3 */
+ *vertical-align: middle; /* 3 */
+}
+
+/**
+ * Address Firefox 3+ setting `line-height` on `input` using `!important` in
+ * the UA stylesheet.
+ */
+
+button,
+input {
+ line-height: normal;
+}
+
+/**
+ * Address inconsistent `text-transform` inheritance for `button` and `select`.
+ * All other form control elements do not inherit `text-transform` values.
+ * Correct `button` style inheritance in Chrome, Safari 5+, and IE 6+.
+ * Correct `select` style inheritance in Firefox 4+ and Opera.
+ */
+
+button,
+select {
+ text-transform: none;
+}
+
+/**
+ * 1. Avoid the WebKit bug in Android 4.0.* where (2) destroys native `audio`
+ * and `video` controls.
+ * 2. Correct inability to style clickable `input` types in iOS.
+ * 3. Improve usability and consistency of cursor style between image-type
+ * `input` and others.
+ * 4. Remove inner spacing in IE 7 without affecting normal text inputs.
+ * Known issue: inner spacing remains in IE 6.
+ */
+
+button,
+html input[type="button"], /* 1 */
+input[type="reset"],
+input[type="submit"] {
+ -webkit-appearance: button; /* 2 */
+ cursor: pointer; /* 3 */
+ *overflow: visible; /* 4 */
+}
+
+/**
+ * Re-set default cursor for disabled elements.
+ */
+
+button[disabled],
+html input[disabled] {
+ cursor: default;
+}
+
+/**
+ * 1. Address box sizing set to content-box in IE 8/9.
+ * 2. Remove excess padding in IE 8/9.
+ * 3. Remove excess padding in IE 7.
+ * Known issue: excess padding remains in IE 6.
+ */
+
+input[type="checkbox"],
+input[type="radio"] {
+ box-sizing: border-box; /* 1 */
+ padding: 0; /* 2 */
+ *height: 13px; /* 3 */
+ *width: 13px; /* 3 */
+}
+
+/**
+ * 1. Address `appearance` set to `searchfield` in Safari 5 and Chrome.
+ * 2. Address `box-sizing` set to `border-box` in Safari 5 and Chrome
+ * (include `-moz` to future-proof).
+ */
+
+input[type="search"] {
+ -webkit-appearance: textfield; /* 1 */
+ -moz-box-sizing: content-box;
+ -webkit-box-sizing: content-box; /* 2 */
+ box-sizing: content-box;
+}
+
+/**
+ * Remove inner padding and search cancel button in Safari 5 and Chrome
+ * on OS X.
+ */
+
+input[type="search"]::-webkit-search-cancel-button,
+input[type="search"]::-webkit-search-decoration {
+ -webkit-appearance: none;
+}
+
+/**
+ * Remove inner padding and border in Firefox 3+.
+ */
+
+button::-moz-focus-inner,
+input::-moz-focus-inner {
+ border: 0;
+ padding: 0;
+}
+
+/**
+ * 1. Remove default vertical scrollbar in IE 6/7/8/9.
+ * 2. Improve readability and alignment in all browsers.
+ */
+
+textarea {
+ overflow: auto; /* 1 */
+ vertical-align: top; /* 2 */
+}
+
+/* ==========================================================================
+ Tables
+ ========================================================================== */
+
+/**
+ * Remove most spacing between table cells.
+ */
+
+table {
+ border-collapse: collapse;
+ border-spacing: 0;
+}
+
+ </style>
+
+ <style type="text/css">
+
+ html, body {
+ margin: 0;
+ padding: 0;
+ min-height: 100%;
+ }
+ body {
+ background: #fff;
+ font-family: "Source Sans Pro", "Helvetica Neueue", Helvetica, sans;
+ font-weight: 300;
+ font-size: 16px;
+ line-height: 1.6em;
+ }
+ #content {
+ width: 70%;
+ max-width: 850px;
+ float: left;
+ padding: 30px 60px;
+ border-left: 1px solid #ddd;
+ }
+ #sidebar {
+ width: 25%;
+ float: left;
+ padding: 30px;
+ overflow: hidden;
+ }
+ #nav {
+ font-size: 130%;
+ margin: 0 0 15px 0;
+ }
+
+ #top {
+ display: block;
+ position: fixed;
+ bottom: 5px;
+ left: 5px;
+ font-size: .85em;
+ text-transform: uppercase;
+ }
+
+ #footer {
+ font-size: .75em;
+ padding: 5px 30px;
+ border-top: 1px solid #ddd;
+ text-align: right;
+ }
+ #footer p {
+ margin: 0 0 0 30px;
+ display: inline-block;
+ }
+
+ h1, h2, h3, h4, h5 {
+ font-weight: 300;
+ }
+ h1 {
+ font-size: 2.5em;
+ line-height: 1.1em;
+ margin: 0 0 .50em 0;
+ }
+
+ h2 {
+ font-size: 1.75em;
+ margin: 1em 0 .50em 0;
+ }
+
+ h3 {
+ margin: 25px 0 10px 0;
+ }
+
+ h4 {
+ margin: 0;
+ font-size: 105%;
+ }
+
+ a {
+ color: #058;
+ text-decoration: none;
+ transition: color .3s ease-in-out;
+ }
+
+ a:hover {
+ color: #e08524;
+ transition: color .3s ease-in-out;
+ }
+
+ pre, code, .mono, .name {
+ font-family: "Ubuntu Mono", "Cousine", "DejaVu Sans Mono", monospace;
+ }
+
+ .title .name {
+ font-weight: bold;
+ }
+ .section-title {
+ margin-top: 2em;
+ }
+ .ident {
+ color: #900;
+ }
+
+ code {
+ background: #f9f9f9;
+ }
+
+ pre {
+ background: #fefefe;
+ border: 1px solid #ddd;
+ box-shadow: 2px 2px 0 #f3f3f3;
+ margin: 0 30px;
+ padding: 15px 30px;
+ }
+
+ .codehilite {
+ margin: 0 30px 10px 30px;
+ }
+
+ .codehilite pre {
+ margin: 0;
+ }
+ .codehilite .err { background: #ff3300; color: #fff !important; }
+
+ table#module-list {
+ font-size: 110%;
+ }
+
+ table#module-list tr td:first-child {
+ padding-right: 10px;
+ white-space: nowrap;
+ }
+
+ table#module-list td {
+ vertical-align: top;
+ padding-bottom: 8px;
+ }
+
+ table#module-list td p {
+ margin: 0 0 7px 0;
+ }
+
+ .def {
+ display: table;
+ }
+
+ .def p {
+ display: table-cell;
+ vertical-align: top;
+ text-align: left;
+ }
+
+ .def p:first-child {
+ white-space: nowrap;
+ }
+
+ .def p:last-child {
+ width: 100%;
+ }
+
+
+ #index {
+ list-style-type: none;
+ margin: 0;
+ padding: 0;
+ }
+ ul#index .class_name {
+ /* font-size: 110%; */
+ font-weight: bold;
+ }
+ #index ul {
+ margin: 0;
+ }
+
+ .item {
+ margin: 0 0 15px 0;
+ }
+
+ .item .class {
+ margin: 0 0 25px 30px;
+ }
+
+ .item .class ul.class_list {
+ margin: 0 0 20px 0;
+ }
+
+ .item .name {
+ background: #fafafa;
+ margin: 0;
+ font-weight: bold;
+ padding: 5px 10px;
+ border-radius: 3px;
+ display: inline-block;
+ min-width: 40%;
+ }
+ .item .name:hover {
+ background: #f6f6f6;
+ }
+
+ .item .empty_desc {
+ margin: 0 0 5px 0;
+ padding: 0;
+ }
+
+ .item .inheritance {
+ margin: 3px 0 0 30px;
+ }
+
+ .item .inherited {
+ color: #666;
+ }
+
+ .item .desc {
+ padding: 0 8px;
+ margin: 0;
+ }
+
+ .item .desc p {
+ margin: 0 0 10px 0;
+ }
+
+ .source_cont {
+ margin: 0;
+ padding: 0;
+ }
+
+ .source_link a {
+ background: #ffc300;
+ font-weight: 400;
+ font-size: .75em;
+ text-transform: uppercase;
+ color: #fff;
+ text-shadow: 1px 1px 0 #f4b700;
+
+ padding: 3px 8px;
+ border-radius: 2px;
+ transition: background .3s ease-in-out;
+ }
+ .source_link a:hover {
+ background: #FF7200;
+ text-shadow: none;
+ transition: background .3s ease-in-out;
+ }
+
+ .source {
+ display: none;
+ max-height: 600px;
+ overflow-y: scroll;
+ margin-bottom: 15px;
+ }
+
+ .source .codehilite {
+ margin: 0;
+ }
+
+ .desc h1, .desc h2, .desc h3 {
+ font-size: 100% !important;
+ }
+ .clear {
+ clear: both;
+ }
+
+ @media all and (max-width: 950px) {
+ #sidebar {
+ width: 35%;
+ }
+ #content {
+ width: 65%;
+ }
+ }
+ @media all and (max-width: 650px) {
+ #top {
+ display: none;
+ }
+ #sidebar {
+ float: none;
+ width: auto;
+ }
+ #content {
+ float: none;
+ width: auto;
+ padding: 30px;
+ }
+
+ #index ul {
+ padding: 0;
+ margin-bottom: 15px;
+ }
+ #index ul li {
+ display: inline-block;
+ margin-right: 30px;
+ }
+ #footer {
+ text-align: left;
+ }
+ #footer p {
+ display: block;
+ margin: inherit;
+ }
+ }
+
+ /*****************************/
+
+ </style>
+
+
+ <style type="text/css">
+
+/* ==========================================================================
+ EXAMPLE Media Queries for Responsive Design.
+ These examples override the primary ('mobile first') styles.
+ Modify as content requires.
+ ========================================================================== */
+
+@media only screen and (min-width: 35em) {
+ /* Style adjustments for viewports that meet the condition */
+}
+
+@media print,
+ (-o-min-device-pixel-ratio: 5/4),
+ (-webkit-min-device-pixel-ratio: 1.25),
+ (min-resolution: 120dpi) {
+ /* Style adjustments for high resolution devices */
+}
+
+/* ==========================================================================
+ Print styles.
+ Inlined to avoid required HTTP connection: h5bp.com/r
+ ========================================================================== */
+
+@media print {
+ * {
+ background: transparent !important;
+ color: #000 !important; /* Black prints faster: h5bp.com/s */
+ box-shadow: none !important;
+ text-shadow: none !important;
+ }
+
+ a,
+ a:visited {
+ text-decoration: underline;
+ }
+
+ a[href]:after {
+ content: " (" attr(href) ")";
+ }
+
+ abbr[title]:after {
+ content: " (" attr(title) ")";
+ }
+
+ /*
+ * Don't show links for images, or javascript/internal links
+ */
+
+ .ir a:after,
+ a[href^="javascript:"]:after,
+ a[href^="#"]:after {
+ content: "";
+ }
+
+ pre,
+ blockquote {
+ border: 1px solid #999;
+ page-break-inside: avoid;
+ }
+
+ thead {
+ display: table-header-group; /* h5bp.com/t */
+ }
+
+ tr,
+ img {
+ page-break-inside: avoid;
+ }
+
+ img {
+ max-width: 100% !important;
+ }
+
+ @page {
+ margin: 0.5cm;
+ }
+
+ p,
+ h2,
+ h3 {
+ orphans: 3;
+ widows: 3;
+ }
+
+ h2,
+ h3 {
+ page-break-after: avoid;
+ }
+}
+
+ </style>
+
+ <script type="text/javascript">
+ function toggle(id, $link) {
+ $node = document.getElementById(id);
+ if (!$node)
+ return;
+ if (!$node.style.display || $node.style.display == 'none') {
+ $node.style.display = 'block';
+ $link.innerHTML = 'Hide source ≢';
+ } else {
+ $node.style.display = 'none';
+ $link.innerHTML = 'Show source ≡';
+ }
+ }
+ </script>
+</head>
+<body>
+<a href="#" id="top">Top</a>
+
+<div id="container">
+
+
+ <div id="sidebar">
+ <h1>Index</h1>
+ <ul id="index">
+
+
+ <li class="set"><h3><a href="#header-classes">Classes</a></h3>
+ <ul>
+ <li class="mono">
+ <span class="class_name"><a href="#heronpy.api.bolt.window_bolt.SlidingWindowBolt">SlidingWindowBolt</a></span>
+
+
+ <ul>
+ <li class="mono"><a href="#heronpy.api.bolt.window_bolt.SlidingWindowBolt.is_tick">is_tick</a></li>
+ <li class="mono"><a href="#heronpy.api.bolt.window_bolt.SlidingWindowBolt.__init__">__init__</a></li>
+ <li class="mono"><a href="#heronpy.api.bolt.window_bolt.SlidingWindowBolt.ack">ack</a></li>
+ <li class="mono"><a href="#heronpy.api.bolt.window_bolt.SlidingWindowBolt.emit">emit</a></li>
+ <li class="mono"><a href="#heronpy.api.bolt.window_bolt.SlidingWindowBolt.fail">fail</a></li>
+ <li class="mono"><a href="#heronpy.api.bolt.window_bolt.SlidingWindowBolt.init_state">init_state</a></li>
+ <li class="mono"><a href="#heronpy.api.bolt.window_bolt.SlidingWindowBolt.initialize">initialize</a></li>
+ <li class="mono"><a href="#heronpy.api.bolt.window_bolt.SlidingWindowBolt.log">log</a></li>
+ <li class="mono"><a href="#heronpy.api.bolt.window_bolt.SlidingWindowBolt.pre_save">pre_save</a></li>
+ <li class="mono"><a href="#heronpy.api.bolt.window_bolt.SlidingWindowBolt.process">process</a></li>
+ <li class="mono"><a href="#heronpy.api.bolt.window_bolt.SlidingWindowBolt.processWindow">processWindow</a></li>
+ <li class="mono"><a href="#heronpy.api.bolt.window_bolt.SlidingWindowBolt.process_tick">process_tick</a></li>
+ <li class="mono"><a href="#heronpy.api.bolt.window_bolt.SlidingWindowBolt.spec">spec</a></li>
+ </ul>
+
+ </li>
+ <li class="mono">
+ <span class="class_name"><a href="#heronpy.api.bolt.window_bolt.TumblingWindowBolt">TumblingWindowBolt</a></span>
+
+
+ <ul>
+ <li class="mono"><a href="#heronpy.api.bolt.window_bolt.TumblingWindowBolt.is_tick">is_tick</a></li>
+ <li class="mono"><a href="#heronpy.api.bolt.window_bolt.TumblingWindowBolt.__init__">__init__</a></li>
+ <li class="mono"><a href="#heronpy.api.bolt.window_bolt.TumblingWindowBolt.ack">ack</a></li>
+ <li class="mono"><a href="#heronpy.api.bolt.window_bolt.TumblingWindowBolt.emit">emit</a></li>
+ <li class="mono"><a href="#heronpy.api.bolt.window_bolt.TumblingWindowBolt.fail">fail</a></li>
+ <li class="mono"><a href="#heronpy.api.bolt.window_bolt.TumblingWindowBolt.init_state">init_state</a></li>
+ <li class="mono"><a href="#heronpy.api.bolt.window_bolt.TumblingWindowBolt.initialize">initialize</a></li>
+ <li class="mono"><a href="#heronpy.api.bolt.window_bolt.TumblingWindowBolt.log">log</a></li>
+ <li class="mono"><a href="#heronpy.api.bolt.window_bolt.TumblingWindowBolt.pre_save">pre_save</a></li>
+ <li class="mono"><a href="#heronpy.api.bolt.window_bolt.TumblingWindowBolt.process">process</a></li>
+ <li class="mono"><a href="#heronpy.api.bolt.window_bolt.TumblingWindowBolt.processWindow">processWindow</a></li>
+ <li class="mono"><a href="#heronpy.api.bolt.window_bolt.TumblingWindowBolt.process_tick">process_tick</a></li>
+ <li class="mono"><a href="#heronpy.api.bolt.window_bolt.TumblingWindowBolt.spec">spec</a></li>
+ </ul>
+
+ </li>
+ <li class="mono">
+ <span class="class_name"><a href="#heronpy.api.bolt.window_bolt.WindowContext">WindowContext</a></span>
+
+ </li>
+ </ul>
+ </li>
+
+ </ul>
+ </div>
+
+ <article id="content">
+
+
+
+
+
+
+ <header id="section-intro">
+ <h1 class="title"><span class="name">heronpy.api.bolt.window_bolt</span> module</h1>
+ <p>window_bolt.py: API for defining windowed bolts in Heron</p>
+
+ <p class="source_link"><a href="javascript:void(0);" onclick="toggle('source-heronpy.api.bolt.window_bolt', this);">Show source ≡</a></p>
+ <div id="source-heronpy.api.bolt.window_bolt" class="source">
+ <pre><code>#!/usr/bin/env python
+# -*- encoding: utf-8 -*-
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+'''window_bolt.py: API for defining windowed bolts in Heron'''
+from abc import abstractmethod
+from collections import namedtuple, deque
+import time
+from heronpy.api.bolt.bolt import Bolt
+import heronpy.api.api_constants as api_constants
+from heronpy.api.state.stateful_component import StatefulComponent
+
+WindowContext = namedtuple('WindowContext', ('start', 'end'))
+
+class SlidingWindowBolt(Bolt, StatefulComponent):
+ """SlidingWindowBolt is a higer level bolt for Heron users who want to deal with
+ batches of tuples belonging to a certain time window. This bolt keeps track of
+ managing the window, adding/expiring tuples based on window configuration.
+ This way users will just have to deal with writing processWindow function
+ """
+ WINDOW_DURATION_SECS = 'slidingwindowbolt_duration_secs'
+ WINDOW_SLIDEINTERVAL_SECS = 'slidingwindowbolt_slideinterval_secs'
+
+ # pylint: disable=attribute-defined-outside-init
+ def init_state(self, stateful_state):
+ self.saved_state = stateful_state
+
+ # pylint: disable=unused-argument
+ def pre_save(self, checkpoint_id):
+ self.saved_state['tuples'] = self.current_tuples
+
+ @abstractmethod
+ def processWindow(self, window_info, tuples):
+ """The main interface that needs to be implemented.
+
+ This function is called every WINDOW_SLIDEINTERVAL_SECS seconds
+ and contains the data in the last WINDOW_DURATION_SECS seconds
+ in a list tuples
+
+ :type window_info: :class:`WindowContext`
+ :param window_info: The information about the window
+
+ :type tuples: :class:`list of Tuples`
+ :param tuples: The list of tuples in this window
+ """
+ pass
+
+ # pylint: disable=unused-argument
+ def initialize(self, config, context):
+ """We initialize the window duration and slide interval
+ """
+ if SlidingWindowBolt.WINDOW_DURATION_SECS in config:
+ self.window_duration = int(config[SlidingWindowBolt.WINDOW_DURATION_SECS])
+ else:
+ self.logger.fatal("Window Duration has to be specified in the config")
+ if SlidingWindowBolt.WINDOW_SLIDEINTERVAL_SECS in config:
+ self.slide_interval = int(config[SlidingWindowBolt.WINDOW_SLIDEINTERVAL_SECS])
+ else:
+ self.slide_interval = self.window_duration
+ if self.slide_interval > self.window_duration:
+ self.logger.fatal("Slide Interval should be <= Window Duration")
+
+ # By modifying the config, we are able to setup the tick timer
+ config[api_constants.TOPOLOGY_TICK_TUPLE_FREQ_SECS] = str(self.slide_interval)
+ self.current_tuples = deque()
+ if hasattr(self, 'saved_state'):
+ if 'tuples' in self.saved_state:
+ self.current_tuples = self.saved_state['tuples']
+
+ def process(self, tup):
+ """Process a single tuple of input
+
+ We add the (time, tuple) pair into our current_tuples. And then look for expiring
+ elemnents
+ """
+ curtime = int(time.time())
+ self.current_tuples.append((tup, curtime))
+ self._expire(curtime)
+
+ def _expire(self, tm):
+ while len(self.current_tuples) > 0:
+ if tm - self.window_duration > self.current_tuples[0][1]:
+ (tup, _) = self.current_tuples.popleft()
+ self.ack(tup)
+ else:
+ break
+
+ # pylint: disable=unused-argument
+ # pylint: disable=unused-variable
+ def process_tick(self, tup):
+ """Called every slide_interval
+ """
+ curtime = int(time.time())
+ window_info = WindowContext(curtime - self.window_duration, curtime)
+ tuple_batch = []
+ for (tup, tm) in self.current_tuples:
+ tuple_batch.append(tup)
+ self.processWindow(window_info, tuple_batch)
+ self._expire(curtime)
+
+
+class TumblingWindowBolt(Bolt, StatefulComponent):
+ """TumblingWindowBolt is a higer level bolt for Heron users who want to deal with
+ batches of tuples belonging to a certain time window. This bolt keeps track of
+ managing the window, adding/expiring tuples based on window configuration.
+ This way users will just have to deal with writing processWindow function
+ """
+ WINDOW_DURATION_SECS = 'tumblingwindowbolt_duration_secs'
+
+ # pylint: disable=attribute-defined-outside-init
+ def init_state(self, stateful_state):
+ self.saved_state = stateful_state
+
+ # pylint: disable=unused-argument
+ def pre_save(self, checkpoint_id):
+ self.saved_state['tuples'] = self.current_tuples
+
+ @abstractmethod
+ def processWindow(self, window_info, tuples):
+ """The main interface that needs to be implemented.
+
+ This function is called every WINDOW_DURATION_SECS seconds
+ and contains the data in the last WINDOW_DURATION_SECS seconds
+ in a list tuples
+
+ :type window_info: :class:`WindowContext`
+ :param window_info: The information about the window
+
+ :type tuples: :class:`list of Tuples`
+ :param tuples: The list of tuples in this window
+ """
+ pass
+
+ # pylint: disable=unused-argument
+ def initialize(self, config, context):
+ """We initialize the window duration and slide interval
+ """
+ if TumblingWindowBolt.WINDOW_DURATION_SECS in config:
+ self.window_duration = int(config[TumblingWindowBolt.WINDOW_DURATION_SECS])
+ else:
+ self.logger.fatal("Window Duration has to be specified in the config")
+
+ # By modifying the config, we are able to setup the tick timer
+ config[api_constants.TOPOLOGY_TICK_TUPLE_FREQ_SECS] = str(self.window_duration)
+ self.current_tuples = deque()
+ if hasattr(self, 'saved_state'):
+ if 'tuples' in self.saved_state:
+ self.current_tuples = self.saved_state['tuples']
+
+ def process(self, tup):
+ """Process a single tuple of input
+
+ We simply add the tuple into our current_tuples.
+ """
+ self.current_tuples.append(tup)
+
+ # pylint: disable=unused-argument
+ # pylint: disable=unused-variable
+ def process_tick(self, tup):
+ """Called every window_duration
+ """
+ curtime = int(time.time())
+ window_info = WindowContext(curtime - self.window_duration, curtime)
+ self.processWindow(window_info, list(self.current_tuples))
+ for tup in self.current_tuples:
+ self.ack(tup)
+ self.current_tuples.clear()
+</code></pre>
+ </div>
+
+ </header>
+
+ <section id="section-items">
+
+
+ <h2 class="section-title" id="header-classes">Classes</h2>
+
+ <div class="item">
+ <p id="heronpy.api.bolt.window_bolt.SlidingWindowBolt" class="name">class <span class="ident">SlidingWindowBolt</span></p>
+
+
+ <div class="desc"><p>SlidingWindowBolt is a higer level bolt for Heron users who want to deal with
+batches of tuples belonging to a certain time window. This bolt keeps track of
+managing the window, adding/expiring tuples based on window configuration.
+This way users will just have to deal with writing processWindow function</p></div>
+ <div class="source_cont">
+ <p class="source_link"><a href="javascript:void(0);" onclick="toggle('source-heronpy.api.bolt.window_bolt.SlidingWindowBolt', this);">Show source ≡</a></p>
+ <div id="source-heronpy.api.bolt.window_bolt.SlidingWindowBolt" class="source">
+ <pre><code>class SlidingWindowBolt(Bolt, StatefulComponent):
+ """SlidingWindowBolt is a higer level bolt for Heron users who want to deal with
+ batches of tuples belonging to a certain time window. This bolt keeps track of
+ managing the window, adding/expiring tuples based on window configuration.
+ This way users will just have to deal with writing processWindow function
+ """
+ WINDOW_DURATION_SECS = 'slidingwindowbolt_duration_secs'
+ WINDOW_SLIDEINTERVAL_SECS = 'slidingwindowbolt_slideinterval_secs'
+
+ # pylint: disable=attribute-defined-outside-init
+ def init_state(self, stateful_state):
+ self.saved_state = stateful_state
+
+ # pylint: disable=unused-argument
+ def pre_save(self, checkpoint_id):
+ self.saved_state['tuples'] = self.current_tuples
+
+ @abstractmethod
+ def processWindow(self, window_info, tuples):
+ """The main interface that needs to be implemented.
+
+ This function is called every WINDOW_SLIDEINTERVAL_SECS seconds
+ and contains the data in the last WINDOW_DURATION_SECS seconds
+ in a list tuples
+
+ :type window_info: :class:`WindowContext`
+ :param window_info: The information about the window
+
+ :type tuples: :class:`list of Tuples`
+ :param tuples: The list of tuples in this window
+ """
+ pass
+
+ # pylint: disable=unused-argument
+ def initialize(self, config, context):
+ """We initialize the window duration and slide interval
+ """
+ if SlidingWindowBolt.WINDOW_DURATION_SECS in config:
+ self.window_duration = int(config[SlidingWindowBolt.WINDOW_DURATION_SECS])
+ else:
+ self.logger.fatal("Window Duration has to be specified in the config")
+ if SlidingWindowBolt.WINDOW_SLIDEINTERVAL_SECS in config:
+ self.slide_interval = int(config[SlidingWindowBolt.WINDOW_SLIDEINTERVAL_SECS])
+ else:
+ self.slide_interval = self.window_duration
+ if self.slide_interval > self.window_duration:
+ self.logger.fatal("Slide Interval should be <= Window Duration")
+
+ # By modifying the config, we are able to setup the tick timer
+ config[api_constants.TOPOLOGY_TICK_TUPLE_FREQ_SECS] = str(self.slide_interval)
+ self.current_tuples = deque()
+ if hasattr(self, 'saved_state'):
+ if 'tuples' in self.saved_state:
+ self.current_tuples = self.saved_state['tuples']
+
+ def process(self, tup):
+ """Process a single tuple of input
+
+ We add the (time, tuple) pair into our current_tuples. And then look for expiring
+ elemnents
+ """
+ curtime = int(time.time())
+ self.current_tuples.append((tup, curtime))
+ self._expire(curtime)
+
+ def _expire(self, tm):
+ while len(self.current_tuples) > 0:
+ if tm - self.window_duration > self.current_tuples[0][1]:
+ (tup, _) = self.current_tuples.popleft()
+ self.ack(tup)
+ else:
+ break
+
+ # pylint: disable=unused-argument
+ # pylint: disable=unused-variable
+ def process_tick(self, tup):
+ """Called every slide_interval
+ """
+ curtime = int(time.time())
+ window_info = WindowContext(curtime - self.window_duration, curtime)
+ tuple_batch = []
+ for (tup, tm) in self.current_tuples:
+ tuple_batch.append(tup)
+ self.processWindow(window_info, tuple_batch)
+ self._expire(curtime)
+</code></pre>
+ </div>
+</div>
+
+
+ <div class="class">
+ <h3>Ancestors (in MRO)</h3>
+ <ul class="class_list">
+ <li><a href="#heronpy.api.bolt.window_bolt.SlidingWindowBolt">SlidingWindowBolt</a></li>
+ <li>heronpy.api.bolt.bolt.Bolt</li>
+ <li>heronpy.api.bolt.base_bolt.BaseBolt</li>
+ <li>heronpy.api.component.base_component.BaseComponent</li>
+ <li>heronpy.api.state.stateful_component.StatefulComponent</li>
+ <li>__builtin__.object</li>
+ </ul>
+ <h3>Class variables</h3>
+ <div class="item">
+ <p id="heronpy.api.bolt.window_bolt.SlidingWindowBolt.WINDOW_DURATION_SECS" class="name">var <span class="ident">WINDOW_DURATION_SECS</span></p>
+
+
+
+
+ <div class="source_cont">
+</div>
+
+ </div>
+ <div class="item">
+ <p id="heronpy.api.bolt.window_bolt.SlidingWindowBolt.WINDOW_SLIDEINTERVAL_SECS" class="name">var <span class="ident">WINDOW_SLIDEINTERVAL_SECS</span></p>
+
+
+
+
+ <div class="source_cont">
+</div>
+
+ </div>
+ <h3>Static methods</h3>
+
+ <div class="item">
+ <div class="name def" id="heronpy.api.bolt.window_bolt.SlidingWindowBolt.is_tick">
+ <p>def <span class="ident">is_tick</span>(</p><p>tup)</p>
+ </div>
+
+
+
+
+ <div class="desc"><p>Returns whether or not a given HeronTuple is a tick Tuple</p>
+<p>It is compatible with StreamParse API.</p></div>
+ <div class="source_cont">
+ <p class="source_link"><a href="javascript:void(0);" onclick="toggle('source-heronpy.api.bolt.window_bolt.SlidingWindowBolt.is_tick', this);">Show source ≡</a></p>
+ <div id="source-heronpy.api.bolt.window_bolt.SlidingWindowBolt.is_tick" class="source">
+ <pre><code>@staticmethod
+def is_tick(tup):
+ """Returns whether or not a given HeronTuple is a tick Tuple
+ It is compatible with StreamParse API.
+ """
+ return tup.stream == TupleHelper.TICK_TUPLE_ID
+</code></pre>
+ </div>
+</div>
+
+ </div>
+
+ <h3>Methods</h3>
+
+ <div class="item">
+ <div class="name def" id="heronpy.api.bolt.window_bolt.SlidingWindowBolt.__init__">
+ <p>def <span class="ident">__init__</span>(</p><p>self, delegate)</p>
+ </div>
+
+
+
+
+ <div class="desc"><p>Initializes BaseComponent</p>
+<p>:param delegate: SpoutInstance or BoltInstance</p></div>
+ <div class="source_cont">
+ <p class="source_link"><a href="javascript:void(0);" onclick="toggle('source-heronpy.api.bolt.window_bolt.SlidingWindowBolt.__init__', this);">Show source ≡</a></p>
+ <div id="source-heronpy.api.bolt.window_bolt.SlidingWindowBolt.__init__" class="source">
+ <pre><code>def __init__(self, delegate):
+ """Initializes BaseComponent
+ :param delegate: SpoutInstance or BoltInstance
+ """
+ self.delegate = delegate
+ self.logger = self.delegate.logger
+</code></pre>
+ </div>
+</div>
+
+ </div>
+
+
+ <div class="item">
+ <div class="name def" id="heronpy.api.bolt.window_bolt.SlidingWindowBolt.ack">
+ <p>def <span class="ident">ack</span>(</p><p>self, tup)</p>
+ </div>
+
+
+
+
+ <div class="desc"><p>Indicate that processing of a Tuple has succeeded</p>
+<p>It is compatible with StreamParse API.</p></div>
+ <div class="source_cont">
+ <p class="source_link"><a href="javascript:void(0);" onclick="toggle('source-heronpy.api.bolt.window_bolt.SlidingWindowBolt.ack', this);">Show source ≡</a></p>
+ <div id="source-heronpy.api.bolt.window_bolt.SlidingWindowBolt.ack" class="source">
+ <pre><code>def ack(self, tup):
+ """Indicate that processing of a Tuple has succeeded
+ It is compatible with StreamParse API.
+ """
+ self.delegate.ack(tup)
+</code></pre>
+ </div>
+</div>
+
+ </div>
+
+
+ <div class="item">
+ <div class="name def" id="heronpy.api.bolt.window_bolt.SlidingWindowBolt.emit">
+ <p>def <span class="ident">emit</span>(</p><p>self, tup, stream='default', anchors=None, direct_task=None, need_task_ids=False)</p>
+ </div>
+
+
+
+
+ <div class="desc"><p>Emits a new tuple from this Bolt</p>
+<p>It is compatible with StreamParse API.</p>
+<p>:type tup: list or tuple
+:param tup: the new output Tuple to send from this bolt,
+ which should contain only serializable data.
+:type stream: str
+:param stream: the ID of the stream to emit this Tuple to.
+ Leave empty to emit to the default stream.
+:type anchors: list
+:param anchors: a list of HeronTuples to which the emitted Tuples should be anchored.
+:type direct_task: int
+:param direct_task: the task to send the Tuple to if performing a direct emit.
+:type need_task_ids: bool
+:param need_task_ids: indicate whether or not you would like the task IDs the Tuple was emitted.</p></div>
+ <div class="source_cont">
+ <p class="source_link"><a href="javascript:void(0);" onclick="toggle('source-heronpy.api.bolt.window_bolt.SlidingWindowBolt.emit', this);">Show source ≡</a></p>
+ <div id="source-heronpy.api.bolt.window_bolt.SlidingWindowBolt.emit" class="source">
+ <pre><code>def emit(self, tup, stream=Stream.DEFAULT_STREAM_ID,
+ anchors=None, direct_task=None, need_task_ids=False):
+ """Emits a new tuple from this Bolt
+ It is compatible with StreamParse API.
+ :type tup: list or tuple
+ :param tup: the new output Tuple to send from this bolt,
+ which should contain only serializable data.
+ :type stream: str
+ :param stream: the ID of the stream to emit this Tuple to.
+ Leave empty to emit to the default stream.
+ :type anchors: list
+ :param anchors: a list of HeronTuples to which the emitted Tuples should be anchored.
+ :type direct_task: int
+ :param direct_task: the task to send the Tuple to if performing a direct emit.
+ :type need_task_ids: bool
+ :param need_task_ids: indicate whether or not you would like the task IDs the Tuple was emitted.
+ """
+ self.delegate.emit(tup, stream, anchors, direct_task, need_task_ids)
+</code></pre>
+ </div>
+</div>
+
+ </div>
+
+
+ <div class="item">
+ <div class="name def" id="heronpy.api.bolt.window_bolt.SlidingWindowBolt.fail">
+ <p>def <span class="ident">fail</span>(</p><p>self, tup)</p>
+ </div>
+
+
+
+
+ <div class="desc"><p>Indicate that processing of a Tuple has failed</p>
+<p>It is compatible with StreamParse API.</p></div>
+ <div class="source_cont">
+ <p class="source_link"><a href="javascript:void(0);" onclick="toggle('source-heronpy.api.bolt.window_bolt.SlidingWindowBolt.fail', this);">Show source ≡</a></p>
+ <div id="source-heronpy.api.bolt.window_bolt.SlidingWindowBolt.fail" class="source">
+ <pre><code>def fail(self, tup):
+ """Indicate that processing of a Tuple has failed
+ It is compatible with StreamParse API.
+ """
+ self.delegate.fail(tup)
+</code></pre>
+ </div>
+</div>
+
+ </div>
+
+
+ <div class="item">
+ <div class="name def" id="heronpy.api.bolt.window_bolt.SlidingWindowBolt.init_state">
+ <p>def <span class="ident">init_state</span>(</p><p>self, stateful_state)</p>
+ </div>
+
+
+
+
+ <div class="source_cont">
+ <p class="source_link"><a href="javascript:void(0);" onclick="toggle('source-heronpy.api.bolt.window_bolt.SlidingWindowBolt.init_state', this);">Show source ≡</a></p>
+ <div id="source-heronpy.api.bolt.window_bolt.SlidingWindowBolt.init_state" class="source">
+ <pre><code>def init_state(self, stateful_state):
+ self.saved_state = stateful_state
+</code></pre>
+ </div>
+</div>
+
+ </div>
+
+
+ <div class="item">
+ <div class="name def" id="heronpy.api.bolt.window_bolt.SlidingWindowBolt.initialize">
+ <p>def <span class="ident">initialize</span>(</p><p>self, config, context)</p>
+ </div>
+
+
+
+
+ <div class="desc"><p>We initialize the window duration and slide interval</p></div>
+ <div class="source_cont">
+ <p class="source_link"><a href="javascript:void(0);" onclick="toggle('source-heronpy.api.bolt.window_bolt.SlidingWindowBolt.initialize', this);">Show source ≡</a></p>
+ <div id="source-heronpy.api.bolt.window_bolt.SlidingWindowBolt.initialize" class="source">
+ <pre><code>def initialize(self, config, context):
+ """We initialize the window duration and slide interval
+ """
+ if SlidingWindowBolt.WINDOW_DURATION_SECS in config:
+ self.window_duration = int(config[SlidingWindowBolt.WINDOW_DURATION_SECS])
+ else:
+ self.logger.fatal("Window Duration has to be specified in the config")
+ if SlidingWindowBolt.WINDOW_SLIDEINTERVAL_SECS in config:
+ self.slide_interval = int(config[SlidingWindowBolt.WINDOW_SLIDEINTERVAL_SECS])
+ else:
+ self.slide_interval = self.window_duration
+ if self.slide_interval > self.window_duration:
+ self.logger.fatal("Slide Interval should be <= Window Duration")
+ # By modifying the config, we are able to setup the tick timer
+ config[api_constants.TOPOLOGY_TICK_TUPLE_FREQ_SECS] = str(self.slide_interval)
+ self.current_tuples = deque()
+ if hasattr(self, 'saved_state'):
+ if 'tuples' in self.saved_state:
+ self.current_tuples = self.saved_state['tuples']
+</code></pre>
+ </div>
+</div>
+
+ </div>
+
+
+ <div class="item">
+ <div class="name def" id="heronpy.api.bolt.window_bolt.SlidingWindowBolt.log">
+ <p>def <span class="ident">log</span>(</p><p>self, message, level=None)</p>
+ </div>
+
+
+
+
+ <div class="desc"><p>Log message, optionally providing a logging level</p>
+<p>:type message: str
+:param message: the log message to send
+:type level: str
+:param level: the logging level,
+ one of: trace (=debug), debug, info, warn or error (default: info)</p></div>
+ <div class="source_cont">
+ <p class="source_link"><a href="javascript:void(0);" onclick="toggle('source-heronpy.api.bolt.window_bolt.SlidingWindowBolt.log', this);">Show source ≡</a></p>
+ <div id="source-heronpy.api.bolt.window_bolt.SlidingWindowBolt.log" class="source">
+ <pre><code>def log(self, message, level=None):
+ """Log message, optionally providing a logging level
+ :type message: str
+ :param message: the log message to send
+ :type level: str
+ :param level: the logging level,
+ one of: trace (=debug), debug, info, warn or error (default: info)
+ """
+ self.delegate.log(message, level)
+</code></pre>
+ </div>
+</div>
+
+ </div>
+
+
+ <div class="item">
+ <div class="name def" id="heronpy.api.bolt.window_bolt.SlidingWindowBolt.pre_save">
+ <p>def <span class="ident">pre_save</span>(</p><p>self, checkpoint_id)</p>
+ </div>
+
+
+
+
+ <div class="source_cont">
+ <p class="source_link"><a href="javascript:void(0);" onclick="toggle('source-heronpy.api.bolt.window_bolt.SlidingWindowBolt.pre_save', this);">Show source ≡</a></p>
+ <div id="source-heronpy.api.bolt.window_bolt.SlidingWindowBolt.pre_save" class="source">
+ <pre><code>def pre_save(self, checkpoint_id):
+ self.saved_state['tuples'] = self.current_tuples
+</code></pre>
+ </div>
+</div>
+
+ </div>
+
+
+ <div class="item">
+ <div class="name def" id="heronpy.api.bolt.window_bolt.SlidingWindowBolt.process">
+ <p>def <span class="ident">process</span>(</p><p>self, tup)</p>
+ </div>
+
+
+
+
+ <div class="desc"><p>Process a single tuple of input</p>
+<p>We add the (time, tuple) pair into our current_tuples. And then look for expiring
+elemnents</p></div>
+ <div class="source_cont">
+ <p class="source_link"><a href="javascript:void(0);" onclick="toggle('source-heronpy.api.bolt.window_bolt.SlidingWindowBolt.process', this);">Show source ≡</a></p>
+ <div id="source-heronpy.api.bolt.window_bolt.SlidingWindowBolt.process" class="source">
+ <pre><code>def process(self, tup):
+ """Process a single tuple of input
+ We add the (time, tuple) pair into our current_tuples. And then look for expiring
+ elemnents
+ """
+ curtime = int(time.time())
+ self.current_tuples.append((tup, curtime))
+ self._expire(curtime)
+</code></pre>
+ </div>
+</div>
+
+ </div>
+
+
+ <div class="item">
+ <div class="name def" id="heronpy.api.bolt.window_bolt.SlidingWindowBolt.processWindow">
+ <p>def <span class="ident">processWindow</span>(</p><p>self, window_info, tuples)</p>
+ </div>
+
+
+
+
+ <div class="desc"><p>The main interface that needs to be implemented.</p>
+<p>This function is called every WINDOW_SLIDEINTERVAL_SECS seconds
+and contains the data in the last WINDOW_DURATION_SECS seconds
+in a list tuples</p>
+<p>:type window_info: :class:<code>WindowContext</code>
+:param window_info: The information about the window</p>
+<p>:type tuples: :class:<code>list of Tuples</code>
+:param tuples: The list of tuples in this window</p></div>
+ <div class="source_cont">
+ <p class="source_link"><a href="javascript:void(0);" onclick="toggle('source-heronpy.api.bolt.window_bolt.SlidingWindowBolt.processWindow', this);">Show source ≡</a></p>
+ <div id="source-heronpy.api.bolt.window_bolt.SlidingWindowBolt.processWindow" class="source">
+ <pre><code>@abstractmethod
+def processWindow(self, window_info, tuples):
+ """The main interface that needs to be implemented.
+ This function is called every WINDOW_SLIDEINTERVAL_SECS seconds
+ and contains the data in the last WINDOW_DURATION_SECS seconds
+ in a list tuples
+ :type window_info: :class:`WindowContext`
+ :param window_info: The information about the window
+ :type tuples: :class:`list of Tuples`
+ :param tuples: The list of tuples in this window
+ """
+ pass
+</code></pre>
+ </div>
+</div>
+
+ </div>
+
+
+ <div class="item">
+ <div class="name def" id="heronpy.api.bolt.window_bolt.SlidingWindowBolt.process_tick">
+ <p>def <span class="ident">process_tick</span>(</p><p>self, tup)</p>
+ </div>
+
+
+
+
+ <div class="desc"><p>Called every slide_interval</p></div>
+ <div class="source_cont">
+ <p class="source_link"><a href="javascript:void(0);" onclick="toggle('source-heronpy.api.bolt.window_bolt.SlidingWindowBolt.process_tick', this);">Show source ≡</a></p>
+ <div id="source-heronpy.api.bolt.window_bolt.SlidingWindowBolt.process_tick" class="source">
+ <pre><code>def process_tick(self, tup):
+ """Called every slide_interval
+ """
+ curtime = int(time.time())
+ window_info = WindowContext(curtime - self.window_duration, curtime)
+ tuple_batch = []
+ for (tup, tm) in self.current_tuples:
+ tuple_batch.append(tup)
+ self.processWindow(window_info, tuple_batch)
+ self._expire(curtime)
+</code></pre>
+ </div>
+</div>
+
+ </div>
+
+
+ <div class="item">
+ <div class="name def" id="heronpy.api.bolt.window_bolt.SlidingWindowBolt.spec">
+ <p>def <span class="ident">spec</span>(</p><p>cls, name=None, inputs=None, par=1, config=None, optional_outputs=None)</p>
+ </div>
+
+
+
+
+ <div class="desc"><p>Register this bolt to the topology and create <code>HeronComponentSpec</code></p>
+<p>This method takes an optional <code>outputs</code> argument for supporting dynamic output fields
+declaration. However, it is recommended that <code>outputs</code> should be declared as
+an attribute of your <code>Bolt</code> subclass. Also, some ways of declaring inputs is not supported
+in this implementation; please read the documentation below.</p>
+<p>:type name: str
+:param name: Name of this bolt.
+:type inputs: dict or list
+:param inputs: Streams that feed into this Bolt.</p>
+<pre><code> Two forms of this are acceptable:
+
+ 1. A `dict` mapping from ``HeronComponentSpec`` to ``Grouping``.
+ In this case, default stream is used.
+ 2. A `dict` mapping from ``GlobalStreamId`` to ``Grouping``.
+ This ``GlobalStreamId`` object itself is different from StreamParse, because
+ Heron does not use thrift, although its constructor method is compatible.
+ 3. A `list` of ``HeronComponentSpec``. In this case, default stream with
+ SHUFFLE grouping is used.
+ 4. A `list` of ``GlobalStreamId``. In this case, SHUFFLE grouping is used.
+</code></pre>
+<p>:type par: int
+:param par: Parallelism hint for this spout.
+:type config: dict
+:param config: Component-specific config settings.
+:type optional_outputs: list of (str or Stream) or tuple of (str or Stream)
+:param optional_outputs: Additional output fields for this bolt. These fields are added to
+ existing <code>outputs</code> class attributes of your bolt. This is an optional
+ argument, and exists only for supporting dynamic output field
+ declaration.</p></div>
+ <div class="source_cont">
+ <p class="source_link"><a href="javascript:void(0);" onclick="toggle('source-heronpy.api.bolt.window_bolt.SlidingWindowBolt.spec', this);">Show source ≡</a></p>
+ <div id="source-heronpy.api.bolt.window_bolt.SlidingWindowBolt.spec" class="source">
+ <pre><code>@classmethod
+def spec(cls, name=None, inputs=None, par=1, config=None, optional_outputs=None):
+ """Register this bolt to the topology and create ``HeronComponentSpec``
+ This method takes an optional ``outputs`` argument for supporting dynamic output fields
+ declaration. However, it is recommended that ``outputs`` should be declared as
+ an attribute of your ``Bolt`` subclass. Also, some ways of declaring inputs is not supported
+ in this implementation; please read the documentation below.
+ :type name: str
+ :param name: Name of this bolt.
+ :type inputs: dict or list
+ :param inputs: Streams that feed into this Bolt.
+ Two forms of this are acceptable:
+ 1. A `dict` mapping from ``HeronComponentSpec`` to ``Grouping``.
+ In this case, default stream is used.
+ 2. A `dict` mapping from ``GlobalStreamId`` to ``Grouping``.
+ This ``GlobalStreamId`` object itself is different from StreamParse, because
+ Heron does not use thrift, although its constructor method is compatible.
+ 3. A `list` of ``HeronComponentSpec``. In this case, default stream with
+ SHUFFLE grouping is used.
+ 4. A `list` of ``GlobalStreamId``. In this case, SHUFFLE grouping is used.
+ :type par: int
+ :param par: Parallelism hint for this spout.
+ :type config: dict
+ :param config: Component-specific config settings.
+ :type optional_outputs: list of (str or Stream) or tuple of (str or Stream)
+ :param optional_outputs: Additional output fields for this bolt. These fields are added to
+ existing ``outputs`` class attributes of your bolt. This is an optional
+ argument, and exists only for supporting dynamic output field
+ declaration.
+ """
+ python_class_path = "%s.%s" % (cls.__module__, cls.__name__)
+ if hasattr(cls, 'outputs'):
+ # avoid modification to cls.outputs
+ _outputs = copy.copy(cls.outputs)
+ else:
+ _outputs = []
+ if optional_outputs is not None:
+ assert isinstance(optional_outputs, (list, tuple))
+ for out in optional_outputs:
+ assert isinstance(out, (str, Stream))
+ _outputs.append(out)
+ return HeronComponentSpec(name, python_class_path, is_spout=False, par=par,
+ inputs=inputs, outputs=_outputs, config=config)
+</code></pre>
+ </div>
+</div>
+
+ </div>
+
+ </div>
+ </div>
+
+ <div class="item">
+ <p id="heronpy.api.bolt.window_bolt.TumblingWindowBolt" class="name">class <span class="ident">TumblingWindowBolt</span></p>
+
+
+ <div class="desc"><p>TumblingWindowBolt is a higer level bolt for Heron users who want to deal with
+batches of tuples belonging to a certain time window. This bolt keeps track of
+managing the window, adding/expiring tuples based on window configuration.
+This way users will just have to deal with writing processWindow function</p></div>
+ <div class="source_cont">
+ <p class="source_link"><a href="javascript:void(0);" onclick="toggle('source-heronpy.api.bolt.window_bolt.TumblingWindowBolt', this);">Show source ≡</a></p>
+ <div id="source-heronpy.api.bolt.window_bolt.TumblingWindowBolt" class="source">
+ <pre><code>class TumblingWindowBolt(Bolt, StatefulComponent):
+ """TumblingWindowBolt is a higer level bolt for Heron users who want to deal with
+ batches of tuples belonging to a certain time window. This bolt keeps track of
+ managing the window, adding/expiring tuples based on window configuration.
+ This way users will just have to deal with writing processWindow function
+ """
+ WINDOW_DURATION_SECS = 'tumblingwindowbolt_duration_secs'
+
+ # pylint: disable=attribute-defined-outside-init
+ def init_state(self, stateful_state):
+ self.saved_state = stateful_state
+
+ # pylint: disable=unused-argument
+ def pre_save(self, checkpoint_id):
+ self.saved_state['tuples'] = self.current_tuples
+
+ @abstractmethod
+ def processWindow(self, window_info, tuples):
+ """The main interface that needs to be implemented.
+
+ This function is called every WINDOW_DURATION_SECS seconds
+ and contains the data in the last WINDOW_DURATION_SECS seconds
+ in a list tuples
+
+ :type window_info: :class:`WindowContext`
+ :param window_info: The information about the window
+
+ :type tuples: :class:`list of Tuples`
+ :param tuples: The list of tuples in this window
+ """
+ pass
+
+ # pylint: disable=unused-argument
+ def initialize(self, config, context):
+ """We initialize the window duration and slide interval
+ """
+ if TumblingWindowBolt.WINDOW_DURATION_SECS in config:
+ self.window_duration = int(config[TumblingWindowBolt.WINDOW_DURATION_SECS])
+ else:
+ self.logger.fatal("Window Duration has to be specified in the config")
+
+ # By modifying the config, we are able to setup the tick timer
+ config[api_constants.TOPOLOGY_TICK_TUPLE_FREQ_SECS] = str(self.window_duration)
+ self.current_tuples = deque()
+ if hasattr(self, 'saved_state'):
+ if 'tuples' in self.saved_state:
+ self.current_tuples = self.saved_state['tuples']
+
+ def process(self, tup):
+ """Process a single tuple of input
+
+ We simply add the tuple into our current_tuples.
+ """
+ self.current_tuples.append(tup)
+
+ # pylint: disable=unused-argument
+ # pylint: disable=unused-variable
+ def process_tick(self, tup):
+ """Called every window_duration
+ """
+ curtime = int(time.time())
+ window_info = WindowContext(curtime - self.window_duration, curtime)
+ self.processWindow(window_info, list(self.current_tuples))
+ for tup in self.current_tuples:
+ self.ack(tup)
+ self.current_tuples.clear()
+</code></pre>
+ </div>
+</div>
+
+
+ <div class="class">
+ <h3>Ancestors (in MRO)</h3>
+ <ul class="class_list">
+ <li><a href="#heronpy.api.bolt.window_bolt.TumblingWindowBolt">TumblingWindowBolt</a></li>
+ <li>heronpy.api.bolt.bolt.Bolt</li>
+ <li>heronpy.api.bolt.base_bolt.BaseBolt</li>
+ <li>heronpy.api.component.base_component.BaseComponent</li>
+ <li>heronpy.api.state.stateful_component.StatefulComponent</li>
+ <li>__builtin__.object</li>
+ </ul>
+ <h3>Class variables</h3>
+ <div class="item">
+ <p id="heronpy.api.bolt.window_bolt.TumblingWindowBolt.WINDOW_DURATION_SECS" class="name">var <span class="ident">WINDOW_DURATION_SECS</span></p>
+
+
+
+
+ <div class="source_cont">
+</div>
+
+ </div>
+ <h3>Static methods</h3>
+
+ <div class="item">
+ <div class="name def" id="heronpy.api.bolt.window_bolt.TumblingWindowBolt.is_tick">
+ <p>def <span class="ident">is_tick</span>(</p><p>tup)</p>
+ </div>
+
+
+
+
+ <div class="desc"><p>Returns whether or not a given HeronTuple is a tick Tuple</p>
+<p>It is compatible with StreamParse API.</p></div>
+ <div class="source_cont">
+ <p class="source_link"><a href="javascript:void(0);" onclick="toggle('source-heronpy.api.bolt.window_bolt.TumblingWindowBolt.is_tick', this);">Show source ≡</a></p>
+ <div id="source-heronpy.api.bolt.window_bolt.TumblingWindowBolt.is_tick" class="source">
+ <pre><code>@staticmethod
+def is_tick(tup):
+ """Returns whether or not a given HeronTuple is a tick Tuple
+ It is compatible with StreamParse API.
+ """
+ return tup.stream == TupleHelper.TICK_TUPLE_ID
+</code></pre>
+ </div>
+</div>
+
+ </div>
+
+ <h3>Methods</h3>
+
+ <div class="item">
+ <div class="name def" id="heronpy.api.bolt.window_bolt.TumblingWindowBolt.__init__">
+ <p>def <span class="ident">__init__</span>(</p><p>self, delegate)</p>
+ </div>
+
+
+
+
+ <div class="desc"><p>Initializes BaseComponent</p>
+<p>:param delegate: SpoutInstance or BoltInstance</p></div>
+ <div class="source_cont">
+ <p class="source_link"><a href="javascript:void(0);" onclick="toggle('source-heronpy.api.bolt.window_bolt.TumblingWindowBolt.__init__', this);">Show source ≡</a></p>
+ <div id="source-heronpy.api.bolt.window_bolt.TumblingWindowBolt.__init__" class="source">
+ <pre><code>def __init__(self, delegate):
+ """Initializes BaseComponent
+ :param delegate: SpoutInstance or BoltInstance
+ """
+ self.delegate = delegate
+ self.logger = self.delegate.logger
+</code></pre>
+ </div>
+</div>
+
+ </div>
+
+
+ <div class="item">
+ <div class="name def" id="heronpy.api.bolt.window_bolt.TumblingWindowBolt.ack">
+ <p>def <span class="ident">ack</span>(</p><p>self, tup)</p>
+ </div>
+
+
+
+
+ <div class="desc"><p>Indicate that processing of a Tuple has succeeded</p>
+<p>It is compatible with StreamParse API.</p></div>
+ <div class="source_cont">
+ <p class="source_link"><a href="javascript:void(0);" onclick="toggle('source-heronpy.api.bolt.window_bolt.TumblingWindowBolt.ack', this);">Show source ≡</a></p>
+ <div id="source-heronpy.api.bolt.window_bolt.TumblingWindowBolt.ack" class="source">
+ <pre><code>def ack(self, tup):
+ """Indicate that processing of a Tuple has succeeded
+ It is compatible with StreamParse API.
+ """
+ self.delegate.ack(tup)
+</code></pre>
+ </div>
+</div>
+
+ </div>
+
+
+ <div class="item">
+ <div class="name def" id="heronpy.api.bolt.window_bolt.TumblingWindowBolt.emit">
+ <p>def <span class="ident">emit</span>(</p><p>self, tup, stream='default', anchors=None, direct_task=None, need_task_ids=False)</p>
+ </div>
+
+
+
+
+ <div class="desc"><p>Emits a new tuple from this Bolt</p>
+<p>It is compatible with StreamParse API.</p>
+<p>:type tup: list or tuple
+:param tup: the new output Tuple to send from this bolt,
+ which should contain only serializable data.
+:type stream: str
+:param stream: the ID of the stream to emit this Tuple to.
+ Leave empty to emit to the default stream.
+:type anchors: list
+:param anchors: a list of HeronTuples to which the emitted Tuples should be anchored.
+:type direct_task: int
+:param direct_task: the task to send the Tuple to if performing a direct emit.
+:type need_task_ids: bool
+:param need_task_ids: indicate whether or not you would like the task IDs the Tuple was emitted.</p></div>
+ <div class="source_cont">
+ <p class="source_link"><a href="javascript:void(0);" onclick="toggle('source-heronpy.api.bolt.window_bolt.TumblingWindowBolt.emit', this);">Show source ≡</a></p>
+ <div id="source-heronpy.api.bolt.window_bolt.TumblingWindowBolt.emit" class="source">
+ <pre><code>def emit(self, tup, stream=Stream.DEFAULT_STREAM_ID,
+ anchors=None, direct_task=None, need_task_ids=False):
+ """Emits a new tuple from this Bolt
+ It is compatible with StreamParse API.
+ :type tup: list or tuple
+ :param tup: the new output Tuple to send from this bolt,
+ which should contain only serializable data.
+ :type stream: str
+ :param stream: the ID of the stream to emit this Tuple to.
+ Leave empty to emit to the default stream.
+ :type anchors: list
+ :param anchors: a list of HeronTuples to which the emitted Tuples should be anchored.
+ :type direct_task: int
+ :param direct_task: the task to send the Tuple to if performing a direct emit.
+ :type need_task_ids: bool
+ :param need_task_ids: indicate whether or not you would like the task IDs the Tuple was emitted.
+ """
+ self.delegate.emit(tup, stream, anchors, direct_task, need_task_ids)
+</code></pre>
+ </div>
+</div>
+
+ </div>
+
+
+ <div class="item">
+ <div class="name def" id="heronpy.api.bolt.window_bolt.TumblingWindowBolt.fail">
+ <p>def <span class="ident">fail</span>(</p><p>self, tup)</p>
+ </div>
+
+
+
+
+ <div class="desc"><p>Indicate that processing of a Tuple has failed</p>
+<p>It is compatible with StreamParse API.</p></div>
+ <div class="source_cont">
+ <p class="source_link"><a href="javascript:void(0);" onclick="toggle('source-heronpy.api.bolt.window_bolt.TumblingWindowBolt.fail', this);">Show source ≡</a></p>
+ <div id="source-heronpy.api.bolt.window_bolt.TumblingWindowBolt.fail" class="source">
+ <pre><code>def fail(self, tup):
+ """Indicate that processing of a Tuple has failed
+ It is compatible with StreamParse API.
+ """
+ self.delegate.fail(tup)
+</code></pre>
+ </div>
+</div>
+
+ </div>
+
+
+ <div class="item">
+ <div class="name def" id="heronpy.api.bolt.window_bolt.TumblingWindowBolt.init_state">
+ <p>def <span class="ident">init_state</span>(</p><p>self, stateful_state)</p>
+ </div>
+
+
+
+
+ <div class="source_cont">
+ <p class="source_link"><a href="javascript:void(0);" onclick="toggle('source-heronpy.api.bolt.window_bolt.TumblingWindowBolt.init_state', this);">Show source ≡</a></p>
+ <div id="source-heronpy.api.bolt.window_bolt.TumblingWindowBolt.init_state" class="source">
+ <pre><code>def init_state(self, stateful_state):
+ self.saved_state = stateful_state
+</code></pre>
+ </div>
+</div>
+
+ </div>
+
+
+ <div class="item">
+ <div class="name def" id="heronpy.api.bolt.window_bolt.TumblingWindowBolt.initialize">
+ <p>def <span class="ident">initialize</span>(</p><p>self, config, context)</p>
+ </div>
+
+
+
+
+ <div class="desc"><p>We initialize the window duration and slide interval</p></div>
+ <div class="source_cont">
+ <p class="source_link"><a href="javascript:void(0);" onclick="toggle('source-heronpy.api.bolt.window_bolt.TumblingWindowBolt.initialize', this);">Show source ≡</a></p>
+ <div id="source-heronpy.api.bolt.window_bolt.TumblingWindowBolt.initialize" class="source">
+ <pre><code>def initialize(self, config, context):
+ """We initialize the window duration and slide interval
+ """
+ if TumblingWindowBolt.WINDOW_DURATION_SECS in config:
+ self.window_duration = int(config[TumblingWindowBolt.WINDOW_DURATION_SECS])
+ else:
+ self.logger.fatal("Window Duration has to be specified in the config")
+ # By modifying the config, we are able to setup the tick timer
+ config[api_constants.TOPOLOGY_TICK_TUPLE_FREQ_SECS] = str(self.window_duration)
+ self.current_tuples = deque()
+ if hasattr(self, 'saved_state'):
+ if 'tuples' in self.saved_state:
+ self.current_tuples = self.saved_state['tuples']
+</code></pre>
+ </div>
+</div>
+
+ </div>
+
+
+ <div class="item">
+ <div class="name def" id="heronpy.api.bolt.window_bolt.TumblingWindowBolt.log">
+ <p>def <span class="ident">log</span>(</p><p>self, message, level=None)</p>
+ </div>
+
+
+
+
+ <div class="desc"><p>Log message, optionally providing a logging level</p>
+<p>:type message: str
+:param message: the log message to send
+:type level: str
+:param level: the logging level,
+ one of: trace (=debug), debug, info, warn or error (default: info)</p></div>
+ <div class="source_cont">
+ <p class="source_link"><a href="javascript:void(0);" onclick="toggle('source-heronpy.api.bolt.window_bolt.TumblingWindowBolt.log', this);">Show source ≡</a></p>
+ <div id="source-heronpy.api.bolt.window_bolt.TumblingWindowBolt.log" class="source">
+ <pre><code>def log(self, message, level=None):
+ """Log message, optionally providing a logging level
+ :type message: str
+ :param message: the log message to send
+ :type level: str
+ :param level: the logging level,
+ one of: trace (=debug), debug, info, warn or error (default: info)
+ """
+ self.delegate.log(message, level)
+</code></pre>
+ </div>
+</div>
+
+ </div>
+
+
+ <div class="item">
+ <div class="name def" id="heronpy.api.bolt.window_bolt.TumblingWindowBolt.pre_save">
+ <p>def <span class="ident">pre_save</span>(</p><p>self, checkpoint_id)</p>
+ </div>
+
+
+
+
+ <div class="source_cont">
+ <p class="source_link"><a href="javascript:void(0);" onclick="toggle('source-heronpy.api.bolt.window_bolt.TumblingWindowBolt.pre_save', this);">Show source ≡</a></p>
+ <div id="source-heronpy.api.bolt.window_bolt.TumblingWindowBolt.pre_save" class="source">
+ <pre><code>def pre_save(self, checkpoint_id):
+ self.saved_state['tuples'] = self.current_tuples
+</code></pre>
+ </div>
+</div>
+
+ </div>
+
+
+ <div class="item">
+ <div class="name def" id="heronpy.api.bolt.window_bolt.TumblingWindowBolt.process">
+ <p>def <span class="ident">process</span>(</p><p>self, tup)</p>
+ </div>
+
+
+
+
+ <div class="desc"><p>Process a single tuple of input</p>
+<p>We simply add the tuple into our current_tuples.</p></div>
+ <div class="source_cont">
+ <p class="source_link"><a href="javascript:void(0);" onclick="toggle('source-heronpy.api.bolt.window_bolt.TumblingWindowBolt.process', this);">Show source ≡</a></p>
+ <div id="source-heronpy.api.bolt.window_bolt.TumblingWindowBolt.process" class="source">
+ <pre><code>def process(self, tup):
+ """Process a single tuple of input
+ We simply add the tuple into our current_tuples.
+ """
+ self.current_tuples.append(tup)
+</code></pre>
+ </div>
+</div>
+
+ </div>
+
+
+ <div class="item">
+ <div class="name def" id="heronpy.api.bolt.window_bolt.TumblingWindowBolt.processWindow">
+ <p>def <span class="ident">processWindow</span>(</p><p>self, window_info, tuples)</p>
+ </div>
+
+
+
+
+ <div class="desc"><p>The main interface that needs to be implemented.</p>
+<p>This function is called every WINDOW_DURATION_SECS seconds
+and contains the data in the last WINDOW_DURATION_SECS seconds
+in a list tuples</p>
+<p>:type window_info: :class:<code>WindowContext</code>
+:param window_info: The information about the window</p>
+<p>:type tuples: :class:<code>list of Tuples</code>
+:param tuples: The list of tuples in this window</p></div>
+ <div class="source_cont">
+ <p class="source_link"><a href="javascript:void(0);" onclick="toggle('source-heronpy.api.bolt.window_bolt.TumblingWindowBolt.processWindow', this);">Show source ≡</a></p>
+ <div id="source-heronpy.api.bolt.window_bolt.TumblingWindowBolt.processWindow" class="source">
+ <pre><code>@abstractmethod
+def processWindow(self, window_info, tuples):
+ """The main interface that needs to be implemented.
+ This function is called every WINDOW_DURATION_SECS seconds
+ and contains the data in the last WINDOW_DURATION_SECS seconds
+ in a list tuples
+ :type window_info: :class:`WindowContext`
+ :param window_info: The information about the window
+ :type tuples: :class:`list of Tuples`
+ :param tuples: The list of tuples in this window
+ """
+ pass
+</code></pre>
+ </div>
+</div>
+
+ </div>
+
+
+ <div class="item">
+ <div class="name def" id="heronpy.api.bolt.window_bolt.TumblingWindowBolt.process_tick">
+ <p>def <span class="ident">process_tick</span>(</p><p>self, tup)</p>
+ </div>
+
+
+
+
+ <div class="desc"><p>Called every window_duration</p></div>
+ <div class="source_cont">
+ <p class="source_link"><a href="javascript:void(0);" onclick="toggle('source-heronpy.api.bolt.window_bolt.TumblingWindowBolt.process_tick', this);">Show source ≡</a></p>
+ <div id="source-heronpy.api.bolt.window_bolt.TumblingWindowBolt.process_tick" class="source">
+ <pre><code>def process_tick(self, tup):
+ """Called every window_duration
+ """
+ curtime = int(time.time())
+ window_info = WindowContext(curtime - self.window_duration, curtime)
+ self.processWindow(window_info, list(self.current_tuples))
+ for tup in self.current_tuples:
+ self.ack(tup)
+ self.current_tuples.clear()
+</code></pre>
+ </div>
+</div>
+
+ </div>
+
+
+ <div class="item">
+ <div class="name def" id="heronpy.api.bolt.window_bolt.TumblingWindowBolt.spec">
+ <p>def <span class="ident">spec</span>(</p><p>cls, name=None, inputs=None, par=1, config=None, optional_outputs=None)</p>
+ </div>
+
+
+
+
+ <div class="desc"><p>Register this bolt to the topology and create <code>HeronComponentSpec</code></p>
+<p>This method takes an optional <code>outputs</code> argument for supporting dynamic output fields
+declaration. However, it is recommended that <code>outputs</code> should be declared as
+an attribute of your <code>Bolt</code> subclass. Also, some ways of declaring inputs is not supported
+in this implementation; please read the documentation below.</p>
+<p>:type name: str
+:param name: Name of this bolt.
+:type inputs: dict or list
+:param inputs: Streams that feed into this Bolt.</p>
+<pre><code> Two forms of this are acceptable:
+
+ 1. A `dict` mapping from ``HeronComponentSpec`` to ``Grouping``.
+ In this case, default stream is used.
+ 2. A `dict` mapping from ``GlobalStreamId`` to ``Grouping``.
+ This ``GlobalStreamId`` object itself is different from StreamParse, because
+ Heron does not use thrift, although its constructor method is compatible.
+ 3. A `list` of ``HeronComponentSpec``. In this case, default stream with
+ SHUFFLE grouping is used.
+ 4. A `list` of ``GlobalStreamId``. In this case, SHUFFLE grouping is used.
+</code></pre>
+<p>:type par: int
+:param par: Parallelism hint for this spout.
+:type config: dict
+:param config: Component-specific config settings.
+:type optional_outputs: list of (str or Stream) or tuple of (str or Stream)
+:param optional_outputs: Additional output fields for this bolt. These fields are added to
+ existing <code>outputs</code> class attributes of your bolt. This is an optional
+ argument, and exists only for supporting dynamic output field
+ declaration.</p></div>
+ <div class="source_cont">
+ <p class="source_link"><a href="javascript:void(0);" onclick="toggle('source-heronpy.api.bolt.window_bolt.TumblingWindowBolt.spec', this);">Show source ≡</a></p>
+ <div id="source-heronpy.api.bolt.window_bolt.TumblingWindowBolt.spec" class="source">
+ <pre><code>@classmethod
+def spec(cls, name=None, inputs=None, par=1, config=None, optional_outputs=None):
+ """Register this bolt to the topology and create ``HeronComponentSpec``
+ This method takes an optional ``outputs`` argument for supporting dynamic output fields
+ declaration. However, it is recommended that ``outputs`` should be declared as
+ an attribute of your ``Bolt`` subclass. Also, some ways of declaring inputs is not supported
+ in this implementation; please read the documentation below.
+ :type name: str
+ :param name: Name of this bolt.
+ :type inputs: dict or list
+ :param inputs: Streams that feed into this Bolt.
+ Two forms of this are acceptable:
+ 1. A `dict` mapping from ``HeronComponentSpec`` to ``Grouping``.
+ In this case, default stream is used.
+ 2. A `dict` mapping from ``GlobalStreamId`` to ``Grouping``.
+ This ``GlobalStreamId`` object itself is different from StreamParse, because
+ Heron does not use thrift, although its constructor method is compatible.
+ 3. A `list` of ``HeronComponentSpec``. In this case, default stream with
+ SHUFFLE grouping is used.
+ 4. A `list` of ``GlobalStreamId``. In this case, SHUFFLE grouping is used.
+ :type par: int
+ :param par: Parallelism hint for this spout.
+ :type config: dict
+ :param config: Component-specific config settings.
+ :type optional_outputs: list of (str or Stream) or tuple of (str or Stream)
+ :param optional_outputs: Additional output fields for this bolt. These fields are added to
+ existing ``outputs`` class attributes of your bolt. This is an optional
+ argument, and exists only for supporting dynamic output field
+ declaration.
+ """
+ python_class_path = "%s.%s" % (cls.__module__, cls.__name__)
+ if hasattr(cls, 'outputs'):
+ # avoid modification to cls.outputs
+ _outputs = copy.copy(cls.outputs)
+ else:
+ _outputs = []
+ if optional_outputs is not None:
+ assert isinstance(optional_outputs, (list, tuple))
+ for out in optional_outputs:
+ assert isinstance(out, (str, Stream))
+ _outputs.append(out)
+ return HeronComponentSpec(name, python_class_path, is_spout=False, par=par,
+ inputs=inputs, outputs=_outputs, config=config)
+</code></pre>
+ </div>
+</div>
+
+ </div>
+
+ </div>
+ </div>
+
+ <div class="item">
+ <p id="heronpy.api.bolt.window_bolt.WindowContext" class="name">class <span class="ident">WindowContext</span></p>
+
+
+ <div class="desc"><p>WindowContext(start, end)</p></div>
+ <div class="source_cont">
+</div>
+
+
+ <div class="class">
+ <h3>Ancestors (in MRO)</h3>
+ <ul class="class_list">
+ <li><a href="#heronpy.api.bolt.window_bolt.WindowContext">WindowContext</a></li>
+ <li>__builtin__.tuple</li>
+ <li>__builtin__.object</li>
+ </ul>
+ <h3>Instance variables</h3>
+ <div class="item">
+ <p id="heronpy.api.bolt.window_bolt.WindowContext.end" class="name">var <span class="ident">end</span></p>
+
+
+
+
+ <div class="desc"><p>Alias for field number 1</p></div>
+ <div class="source_cont">
+</div>
+
+ </div>
+ <div class="item">
+ <p id="heronpy.api.bolt.window_bolt.WindowContext.start" class="name">var <span class="ident">start</span></p>
+
+
+
+
+ <div class="desc"><p>Alias for field number 0</p></div>
+ <div class="source_cont">
+</div>
+
+ </div>
+ </div>
+ </div>
+
+ </section>
+
+ </article>
+ <div class="clear"> </div>
+ <footer id="footer">
+ <p>
+ Documentation generated by
+ <a href="https://github.com/BurntSushi/pdoc">pdoc 0.3.2</a>
+ </p>
+
+ <p>pdoc is in the public domain with the
+ <a href="http://unlicense.org">UNLICENSE</a></p>
+
+ <p>Design by <a href="http://nadh.in">Kailash Nadh</a></p>
+ </footer>
+</div>
+</body>
+</html>
diff --git a/content/api/python/api/component/component_spec.m.html b/content/api/python/api/component/component_spec.m.html
new file mode 100644
index 0000000..68af6e0
--- /dev/null
+++ b/content/api/python/api/component/component_spec.m.html
@@ -0,0 +1,2066 @@
+<!doctype html>
+<head>
+ <meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
+ <meta name="viewport" content="width=device-width, initial-scale=1, minimum-scale=1" />
+
+ <title>heronpy.api.component.component_spec API documentation</title>
+ <meta name="description" content="component_spec.py" />
+
+ <link href='http://fonts.googleapis.com/css?family=Source+Sans+Pro:400,300' rel='stylesheet' type='text/css'>
+
+ <style type="text/css">
+
+* {
+ box-sizing: border-box;
+}
+/*! normalize.css v1.1.1 | MIT License | git.io/normalize */
+
+/* ==========================================================================
+ HTML5 display definitions
+ ========================================================================== */
+
+/**
+ * Correct `block` display not defined in IE 6/7/8/9 and Firefox 3.
+ */
+
+article,
+aside,
+details,
+figcaption,
+figure,
+footer,
+header,
+hgroup,
+main,
+nav,
+section,
+summary {
+ display: block;
+}
+
+/**
+ * Correct `inline-block` display not defined in IE 6/7/8/9 and Firefox 3.
+ */
+
+audio,
+canvas,
+video {
+ display: inline-block;
+ *display: inline;
+ *zoom: 1;
+}
+
+/**
+ * Prevent modern browsers from displaying `audio` without controls.
+ * Remove excess height in iOS 5 devices.
+ */
+
+audio:not([controls]) {
+ display: none;
+ height: 0;
+}
+
+/**
+ * Address styling not present in IE 7/8/9, Firefox 3, and Safari 4.
+ * Known issue: no IE 6 support.
+ */
+
+[hidden] {
+ display: none;
+}
+
+/* ==========================================================================
+ Base
+ ========================================================================== */
+
+/**
+ * 1. Prevent system color scheme's background color being used in Firefox, IE,
+ * and Opera.
+ * 2. Prevent system color scheme's text color being used in Firefox, IE, and
+ * Opera.
+ * 3. Correct text resizing oddly in IE 6/7 when body `font-size` is set using
+ * `em` units.
+ * 4. Prevent iOS text size adjust after orientation change, without disabling
+ * user zoom.
+ */
+
+html {
+ background: #fff; /* 1 */
+ color: #000; /* 2 */
+ font-size: 100%; /* 3 */
+ -webkit-text-size-adjust: 100%; /* 4 */
+ -ms-text-size-adjust: 100%; /* 4 */
+}
+
+/**
+ * Address `font-family` inconsistency between `textarea` and other form
+ * elements.
+ */
+
+html,
+button,
+input,
+select,
+textarea {
+ font-family: sans-serif;
+}
+
+/**
+ * Address margins handled incorrectly in IE 6/7.
+ */
+
+body {
+ margin: 0;
+}
+
+/* ==========================================================================
+ Links
+ ========================================================================== */
+
+/**
+ * Address `outline` inconsistency between Chrome and other browsers.
+ */
+
+a:focus {
+ outline: thin dotted;
+}
+
+/**
+ * Improve readability when focused and also mouse hovered in all browsers.
+ */
+
+a:active,
+a:hover {
+ outline: 0;
+}
+
+/* ==========================================================================
+ Typography
+ ========================================================================== */
+
+/**
+ * Address font sizes and margins set differently in IE 6/7.
+ * Address font sizes within `section` and `article` in Firefox 4+, Safari 5,
+ * and Chrome.
+ */
+
+h1 {
+ font-size: 2em;
+ margin: 0.67em 0;
+}
+
+h2 {
+ font-size: 1.5em;
+ margin: 0.83em 0;
+}
+
+h3 {
+ font-size: 1.17em;
+ margin: 1em 0;
+}
+
+h4 {
+ font-size: 1em;
+ margin: 1.33em 0;
+}
+
+h5 {
+ font-size: 0.83em;
+ margin: 1.67em 0;
+}
+
+h6 {
+ font-size: 0.67em;
+ margin: 2.33em 0;
+}
+
+/**
+ * Address styling not present in IE 7/8/9, Safari 5, and Chrome.
+ */
+
+abbr[title] {
+ border-bottom: 1px dotted;
+}
+
+/**
+ * Address style set to `bolder` in Firefox 3+, Safari 4/5, and Chrome.
+ */
+
+b,
+strong {
+ font-weight: bold;
+}
+
+blockquote {
+ margin: 1em 40px;
+}
+
+/**
+ * Address styling not present in Safari 5 and Chrome.
+ */
+
+dfn {
+ font-style: italic;
+}
+
+/**
+ * Address differences between Firefox and other browsers.
+ * Known issue: no IE 6/7 normalization.
+ */
+
+hr {
+ -moz-box-sizing: content-box;
+ box-sizing: content-box;
+ height: 0;
+}
+
+/**
+ * Address styling not present in IE 6/7/8/9.
+ */
+
+mark {
+ background: #ff0;
+ color: #000;
+}
+
+/**
+ * Address margins set differently in IE 6/7.
+ */
+
+p,
+pre {
+ margin: 1em 0;
+}
+
+/**
+ * Correct font family set oddly in IE 6, Safari 4/5, and Chrome.
+ */
+
+code,
+kbd,
+pre,
+samp {
+ font-family: monospace, serif;
+ _font-family: 'courier new', monospace;
+ font-size: 1em;
+}
+
+/**
+ * Improve readability of pre-formatted text in all browsers.
+ */
+
+pre {
+ white-space: pre;
+ white-space: pre-wrap;
+ word-wrap: break-word;
+}
+
+/**
+ * Address CSS quotes not supported in IE 6/7.
+ */
+
+q {
+ quotes: none;
+}
+
+/**
+ * Address `quotes` property not supported in Safari 4.
+ */
+
+q:before,
+q:after {
+ content: '';
+ content: none;
+}
+
+/**
+ * Address inconsistent and variable font size in all browsers.
+ */
+
+small {
+ font-size: 80%;
+}
+
+/**
+ * Prevent `sub` and `sup` affecting `line-height` in all browsers.
+ */
+
+sub,
+sup {
+ font-size: 75%;
+ line-height: 0;
+ position: relative;
+ vertical-align: baseline;
+}
+
+sup {
+ top: -0.5em;
+}
+
+sub {
+ bottom: -0.25em;
+}
+
+/* ==========================================================================
+ Lists
+ ========================================================================== */
+
+/**
+ * Address margins set differently in IE 6/7.
+ */
+
+dl,
+menu,
+ol,
+ul {
+ margin: 1em 0;
+}
+
+dd {
+ margin: 0 0 0 40px;
+}
+
+/**
+ * Address paddings set differently in IE 6/7.
+ */
+
+menu,
+ol,
+ul {
+ padding: 0 0 0 40px;
+}
+
+/**
+ * Correct list images handled incorrectly in IE 7.
+ */
+
+nav ul,
+nav ol {
+ list-style: none;
+ list-style-image: none;
+}
+
+/* ==========================================================================
+ Embedded content
+ ========================================================================== */
+
+/**
+ * 1. Remove border when inside `a` element in IE 6/7/8/9 and Firefox 3.
+ * 2. Improve image quality when scaled in IE 7.
+ */
+
+img {
+ border: 0; /* 1 */
+ -ms-interpolation-mode: bicubic; /* 2 */
+}
+
+/**
+ * Correct overflow displayed oddly in IE 9.
+ */
+
+svg:not(:root) {
+ overflow: hidden;
+}
+
+/* ==========================================================================
+ Figures
+ ========================================================================== */
+
+/**
+ * Address margin not present in IE 6/7/8/9, Safari 5, and Opera 11.
+ */
+
+figure {
+ margin: 0;
+}
+
+/* ==========================================================================
+ Forms
+ ========================================================================== */
+
+/**
+ * Correct margin displayed oddly in IE 6/7.
+ */
+
+form {
+ margin: 0;
+}
+
+/**
+ * Define consistent border, margin, and padding.
+ */
+
+fieldset {
+ border: 1px solid #c0c0c0;
+ margin: 0 2px;
+ padding: 0.35em 0.625em 0.75em;
+}
+
+/**
+ * 1. Correct color not being inherited in IE 6/7/8/9.
+ * 2. Correct text not wrapping in Firefox 3.
+ * 3. Correct alignment displayed oddly in IE 6/7.
+ */
+
+legend {
+ border: 0; /* 1 */
+ padding: 0;
+ white-space: normal; /* 2 */
+ *margin-left: -7px; /* 3 */
+}
+
+/**
+ * 1. Correct font size not being inherited in all browsers.
+ * 2. Address margins set differently in IE 6/7, Firefox 3+, Safari 5,
+ * and Chrome.
+ * 3. Improve appearance and consistency in all browsers.
+ */
+
+button,
+input,
+select,
+textarea {
+ font-size: 100%; /* 1 */
+ margin: 0; /* 2 */
+ vertical-align: baseline; /* 3 */
+ *vertical-align: middle; /* 3 */
+}
+
+/**
+ * Address Firefox 3+ setting `line-height` on `input` using `!important` in
+ * the UA stylesheet.
+ */
+
+button,
+input {
+ line-height: normal;
+}
+
+/**
+ * Address inconsistent `text-transform` inheritance for `button` and `select`.
+ * All other form control elements do not inherit `text-transform` values.
+ * Correct `button` style inheritance in Chrome, Safari 5+, and IE 6+.
+ * Correct `select` style inheritance in Firefox 4+ and Opera.
+ */
+
+button,
+select {
+ text-transform: none;
+}
+
+/**
+ * 1. Avoid the WebKit bug in Android 4.0.* where (2) destroys native `audio`
+ * and `video` controls.
+ * 2. Correct inability to style clickable `input` types in iOS.
+ * 3. Improve usability and consistency of cursor style between image-type
+ * `input` and others.
+ * 4. Remove inner spacing in IE 7 without affecting normal text inputs.
+ * Known issue: inner spacing remains in IE 6.
+ */
+
+button,
+html input[type="button"], /* 1 */
+input[type="reset"],
+input[type="submit"] {
+ -webkit-appearance: button; /* 2 */
+ cursor: pointer; /* 3 */
+ *overflow: visible; /* 4 */
+}
+
+/**
+ * Re-set default cursor for disabled elements.
+ */
+
+button[disabled],
+html input[disabled] {
+ cursor: default;
+}
+
+/**
+ * 1. Address box sizing set to content-box in IE 8/9.
+ * 2. Remove excess padding in IE 8/9.
+ * 3. Remove excess padding in IE 7.
+ * Known issue: excess padding remains in IE 6.
+ */
+
+input[type="checkbox"],
+input[type="radio"] {
+ box-sizing: border-box; /* 1 */
+ padding: 0; /* 2 */
+ *height: 13px; /* 3 */
+ *width: 13px; /* 3 */
+}
+
+/**
+ * 1. Address `appearance` set to `searchfield` in Safari 5 and Chrome.
+ * 2. Address `box-sizing` set to `border-box` in Safari 5 and Chrome
+ * (include `-moz` to future-proof).
+ */
+
+input[type="search"] {
+ -webkit-appearance: textfield; /* 1 */
+ -moz-box-sizing: content-box;
+ -webkit-box-sizing: content-box; /* 2 */
+ box-sizing: content-box;
+}
+
+/**
+ * Remove inner padding and search cancel button in Safari 5 and Chrome
+ * on OS X.
+ */
+
+input[type="search"]::-webkit-search-cancel-button,
+input[type="search"]::-webkit-search-decoration {
+ -webkit-appearance: none;
+}
+
+/**
+ * Remove inner padding and border in Firefox 3+.
+ */
+
+button::-moz-focus-inner,
+input::-moz-focus-inner {
+ border: 0;
+ padding: 0;
+}
+
+/**
+ * 1. Remove default vertical scrollbar in IE 6/7/8/9.
+ * 2. Improve readability and alignment in all browsers.
+ */
+
+textarea {
+ overflow: auto; /* 1 */
+ vertical-align: top; /* 2 */
+}
+
+/* ==========================================================================
+ Tables
+ ========================================================================== */
+
+/**
+ * Remove most spacing between table cells.
+ */
+
+table {
+ border-collapse: collapse;
+ border-spacing: 0;
+}
+
+ </style>
+
+ <style type="text/css">
+
+ html, body {
+ margin: 0;
+ padding: 0;
+ min-height: 100%;
+ }
+ body {
+ background: #fff;
+ font-family: "Source Sans Pro", "Helvetica Neueue", Helvetica, sans;
+ font-weight: 300;
+ font-size: 16px;
+ line-height: 1.6em;
+ }
+ #content {
+ width: 70%;
+ max-width: 850px;
+ float: left;
+ padding: 30px 60px;
+ border-left: 1px solid #ddd;
+ }
+ #sidebar {
+ width: 25%;
+ float: left;
+ padding: 30px;
+ overflow: hidden;
+ }
+ #nav {
+ font-size: 130%;
+ margin: 0 0 15px 0;
+ }
+
+ #top {
+ display: block;
+ position: fixed;
+ bottom: 5px;
+ left: 5px;
+ font-size: .85em;
+ text-transform: uppercase;
+ }
+
+ #footer {
+ font-size: .75em;
+ padding: 5px 30px;
+ border-top: 1px solid #ddd;
+ text-align: right;
+ }
+ #footer p {
+ margin: 0 0 0 30px;
+ display: inline-block;
+ }
+
+ h1, h2, h3, h4, h5 {
+ font-weight: 300;
+ }
+ h1 {
+ font-size: 2.5em;
+ line-height: 1.1em;
+ margin: 0 0 .50em 0;
+ }
+
+ h2 {
+ font-size: 1.75em;
+ margin: 1em 0 .50em 0;
+ }
+
+ h3 {
+ margin: 25px 0 10px 0;
+ }
+
+ h4 {
+ margin: 0;
+ font-size: 105%;
+ }
+
+ a {
+ color: #058;
+ text-decoration: none;
+ transition: color .3s ease-in-out;
+ }
+
+ a:hover {
+ color: #e08524;
+ transition: color .3s ease-in-out;
+ }
+
+ pre, code, .mono, .name {
+ font-family: "Ubuntu Mono", "Cousine", "DejaVu Sans Mono", monospace;
+ }
+
+ .title .name {
+ font-weight: bold;
+ }
+ .section-title {
+ margin-top: 2em;
+ }
+ .ident {
+ color: #900;
+ }
+
+ code {
+ background: #f9f9f9;
+ }
+
+ pre {
+ background: #fefefe;
+ border: 1px solid #ddd;
+ box-shadow: 2px 2px 0 #f3f3f3;
+ margin: 0 30px;
+ padding: 15px 30px;
+ }
+
+ .codehilite {
+ margin: 0 30px 10px 30px;
+ }
+
+ .codehilite pre {
+ margin: 0;
+ }
+ .codehilite .err { background: #ff3300; color: #fff !important; }
+
+ table#module-list {
+ font-size: 110%;
+ }
+
+ table#module-list tr td:first-child {
+ padding-right: 10px;
+ white-space: nowrap;
+ }
+
+ table#module-list td {
+ vertical-align: top;
+ padding-bottom: 8px;
+ }
+
+ table#module-list td p {
+ margin: 0 0 7px 0;
+ }
+
+ .def {
+ display: table;
+ }
+
+ .def p {
+ display: table-cell;
+ vertical-align: top;
+ text-align: left;
+ }
+
+ .def p:first-child {
+ white-space: nowrap;
+ }
+
+ .def p:last-child {
+ width: 100%;
+ }
+
+
+ #index {
+ list-style-type: none;
+ margin: 0;
+ padding: 0;
+ }
+ ul#index .class_name {
+ /* font-size: 110%; */
+ font-weight: bold;
+ }
+ #index ul {
+ margin: 0;
+ }
+
+ .item {
+ margin: 0 0 15px 0;
+ }
+
+ .item .class {
+ margin: 0 0 25px 30px;
+ }
+
+ .item .class ul.class_list {
+ margin: 0 0 20px 0;
+ }
+
+ .item .name {
+ background: #fafafa;
+ margin: 0;
+ font-weight: bold;
+ padding: 5px 10px;
+ border-radius: 3px;
+ display: inline-block;
+ min-width: 40%;
+ }
+ .item .name:hover {
+ background: #f6f6f6;
+ }
+
+ .item .empty_desc {
+ margin: 0 0 5px 0;
+ padding: 0;
+ }
+
+ .item .inheritance {
+ margin: 3px 0 0 30px;
+ }
+
+ .item .inherited {
+ color: #666;
+ }
+
+ .item .desc {
+ padding: 0 8px;
+ margin: 0;
+ }
+
+ .item .desc p {
+ margin: 0 0 10px 0;
+ }
+
+ .source_cont {
+ margin: 0;
+ padding: 0;
+ }
+
+ .source_link a {
+ background: #ffc300;
+ font-weight: 400;
+ font-size: .75em;
+ text-transform: uppercase;
+ color: #fff;
+ text-shadow: 1px 1px 0 #f4b700;
+
+ padding: 3px 8px;
+ border-radius: 2px;
+ transition: background .3s ease-in-out;
+ }
+ .source_link a:hover {
+ background: #FF7200;
+ text-shadow: none;
+ transition: background .3s ease-in-out;
+ }
+
+ .source {
+ display: none;
+ max-height: 600px;
+ overflow-y: scroll;
+ margin-bottom: 15px;
+ }
+
+ .source .codehilite {
+ margin: 0;
+ }
+
+ .desc h1, .desc h2, .desc h3 {
+ font-size: 100% !important;
+ }
+ .clear {
+ clear: both;
+ }
+
+ @media all and (max-width: 950px) {
+ #sidebar {
+ width: 35%;
+ }
+ #content {
+ width: 65%;
+ }
+ }
+ @media all and (max-width: 650px) {
+ #top {
+ display: none;
+ }
+ #sidebar {
+ float: none;
+ width: auto;
+ }
+ #content {
+ float: none;
+ width: auto;
+ padding: 30px;
+ }
+
+ #index ul {
+ padding: 0;
+ margin-bottom: 15px;
+ }
+ #index ul li {
+ display: inline-block;
+ margin-right: 30px;
+ }
+ #footer {
+ text-align: left;
+ }
+ #footer p {
+ display: block;
+ margin: inherit;
+ }
+ }
+
+ /*****************************/
+
+ </style>
+
+
+ <style type="text/css">
+
+/* ==========================================================================
+ EXAMPLE Media Queries for Responsive Design.
+ These examples override the primary ('mobile first') styles.
+ Modify as content requires.
+ ========================================================================== */
+
+@media only screen and (min-width: 35em) {
+ /* Style adjustments for viewports that meet the condition */
+}
+
+@media print,
+ (-o-min-device-pixel-ratio: 5/4),
+ (-webkit-min-device-pixel-ratio: 1.25),
+ (min-resolution: 120dpi) {
+ /* Style adjustments for high resolution devices */
+}
+
+/* ==========================================================================
+ Print styles.
+ Inlined to avoid required HTTP connection: h5bp.com/r
+ ========================================================================== */
+
+@media print {
+ * {
+ background: transparent !important;
+ color: #000 !important; /* Black prints faster: h5bp.com/s */
+ box-shadow: none !important;
+ text-shadow: none !important;
+ }
+
+ a,
+ a:visited {
+ text-decoration: underline;
+ }
+
+ a[href]:after {
+ content: " (" attr(href) ")";
+ }
+
+ abbr[title]:after {
+ content: " (" attr(title) ")";
+ }
+
+ /*
+ * Don't show links for images, or javascript/internal links
+ */
+
+ .ir a:after,
+ a[href^="javascript:"]:after,
+ a[href^="#"]:after {
+ content: "";
+ }
+
+ pre,
+ blockquote {
+ border: 1px solid #999;
+ page-break-inside: avoid;
+ }
+
+ thead {
+ display: table-header-group; /* h5bp.com/t */
+ }
+
+ tr,
+ img {
+ page-break-inside: avoid;
+ }
+
+ img {
+ max-width: 100% !important;
+ }
+
+ @page {
+ margin: 0.5cm;
+ }
+
+ p,
+ h2,
+ h3 {
+ orphans: 3;
+ widows: 3;
+ }
+
+ h2,
+ h3 {
+ page-break-after: avoid;
+ }
+}
+
+ </style>
+
+ <script type="text/javascript">
+ function toggle(id, $link) {
+ $node = document.getElementById(id);
+ if (!$node)
+ return;
+ if (!$node.style.display || $node.style.display == 'none') {
+ $node.style.display = 'block';
+ $link.innerHTML = 'Hide source ≢';
+ } else {
+ $node.style.display = 'none';
+ $link.innerHTML = 'Show source ≡';
+ }
+ }
+ </script>
+</head>
+<body>
+<a href="#" id="top">Top</a>
+
+<div id="container">
+
+
+ <div id="sidebar">
+ <h1>Index</h1>
+ <ul id="index">
+ <li class="set"><h3><a href="#header-variables">Module variables</a></h3>
+
+ <ul>
+ <li class="mono"><a href="#heronpy.api.component.component_spec.TOPOLOGY_COMPONENT_PARALLELISM">TOPOLOGY_COMPONENT_PARALLELISM</a></li>
+ </ul>
+
+ </li>
+
+
+ <li class="set"><h3><a href="#header-classes">Classes</a></h3>
+ <ul>
+ <li class="mono">
+ <span class="class_name"><a href="#heronpy.api.component.component_spec.GlobalStreamId">GlobalStreamId</a></span>
+
+
+ <ul>
+ <li class="mono"><a href="#heronpy.api.component.component_spec.GlobalStreamId.__init__">__init__</a></li>
+ </ul>
+
+ </li>
+ <li class="mono">
+ <span class="class_name"><a href="#heronpy.api.component.component_spec.HeronComponentSpec">HeronComponentSpec</a></span>
+
+
+ <ul>
+ <li class="mono"><a href="#heronpy.api.component.component_spec.HeronComponentSpec.__init__">__init__</a></li>
+ <li class="mono"><a href="#heronpy.api.component.component_spec.HeronComponentSpec.get_out_streamids">get_out_streamids</a></li>
+ <li class="mono"><a href="#heronpy.api.component.component_spec.HeronComponentSpec.get_protobuf">get_protobuf</a></li>
+ </ul>
+
+ </li>
+ </ul>
+ </li>
+
+ </ul>
+ </div>
+
+ <article id="content">
+
+
+
+
+
+
+ <header id="section-intro">
+ <h1 class="title"><span class="name">heronpy.api.component.component_spec</span> module</h1>
+ <p>component_spec.py</p>
+
+ <p class="source_link"><a href="javascript:void(0);" onclick="toggle('source-heronpy.api.component.component_spec', this);">Show source ≡</a></p>
+ <div id="source-heronpy.api.component.component_spec" class="source">
+ <pre><code>#!/usr/bin/env python
+# -*- encoding: utf-8 -*-
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+'''component_spec.py'''
+import uuid
+
+from heronpy.api.serializer import default_serializer
+from heronpy.api.api_constants import TOPOLOGY_COMPONENT_PARALLELISM
+from heronpy.proto import topology_pb2
+
+from heronpy.api.stream import Stream, Grouping
+
+# pylint: disable=too-many-instance-attributes
+class HeronComponentSpec(object):
+ """Class to specify the information and location of components in a topology
+
+ This class is generated by the ``spec()`` method of Spout and Bolt class and
+ specifies how this component is located in the topology and how it is connected to
+ the other components. This class also retains the Python class path of the component,
+ so pex_loader can load the component appropriately.
+ """
+ def __init__(self, name, python_class_path, is_spout, par,
+ inputs=None, outputs=None, config=None):
+ self._sanitize_args(name, python_class_path, is_spout, par)
+
+ self.name = name
+ self.python_class_path = python_class_path
+ self.is_spout = is_spout
+ self.parallelism = par
+
+ # inputs, outputs, config will be sanitized later
+ self.inputs = inputs
+ self.outputs = outputs
+ self.custom_config = config
+
+ # This is used for identification, especially when name is not specified by argument
+ # Note that ``self.name`` might not be available until it is set by TopologyType metaclass
+ # so this is necessary for identification purposes. Used mainly by GlobalStreamId.
+ self.uuid = str(uuid.uuid4())
+
+ @staticmethod
+ def _sanitize_args(name, py_class_path, is_spout, par):
+ # name can be None at the time this spec is initialized
+ assert name is None or isinstance(name, str)
+ assert isinstance(py_class_path, str)
+ assert isinstance(is_spout, bool)
+ assert isinstance(par, int) and par > 0
+
+ def get_protobuf(self):
+ """Returns protobuf message (Spout or Bolt) of this component"""
+ if self.is_spout:
+ return self._get_spout()
+ else:
+ return self._get_bolt()
+
+ def _get_spout(self):
+ """Returns Spout protobuf message"""
+ spout = topology_pb2.Spout()
+ spout.comp.CopyFrom(self._get_base_component())
+
+ # Add output streams
+ self._add_out_streams(spout)
+ return spout
+
+ def _get_bolt(self):
+ """Returns Bolt protobuf message"""
+ bolt = topology_pb2.Bolt()
+ bolt.comp.CopyFrom(self._get_base_component())
+
+ # Add streams
+ self._add_in_streams(bolt)
+ self._add_out_streams(bolt)
+ return bolt
+
+ def _get_base_component(self):
+ """Returns Component protobuf message"""
+ comp = topology_pb2.Component()
+ comp.name = self.name
+ comp.spec = topology_pb2.ComponentObjectSpec.Value("PYTHON_CLASS_NAME")
+ comp.class_name = self.python_class_path
+ comp.config.CopyFrom(self._get_comp_config())
+ return comp
+
+ def _get_comp_config(self):
+ """Returns component-specific Config protobuf message
+
+ It first adds ``topology.component.parallelism``, and is overriden by
+ a user-defined component-specific configuration, specified by spec().
+ """
+ proto_config = topology_pb2.Config()
+
+ # first add parallelism
+ key = proto_config.kvs.add()
+ key.key = TOPOLOGY_COMPONENT_PARALLELISM
+ key.value = str(self.parallelism)
+ key.type = topology_pb2.ConfigValueType.Value("STRING_VALUE")
+
+ # iterate through self.custom_config
+ if self.custom_config is not None:
+ sanitized = self._sanitize_config(self.custom_config)
+ for key, value in list(sanitized.items()):
+ if isinstance(value, str):
+ kvs = proto_config.kvs.add()
+ kvs.key = key
+ kvs.value = value
+ kvs.type = topology_pb2.ConfigValueType.Value("STRING_VALUE")
+ else:
+ # need to serialize
+ kvs = proto_config.kvs.add()
+ kvs.key = key
+ kvs.serialized_value = default_serializer.serialize(value)
+ kvs.type = topology_pb2.ConfigValueType.Value("PYTHON_SERIALIZED_VALUE")
+
+ return proto_config
+
+ @staticmethod
+ def _sanitize_config(custom_config):
+ """Checks whether ``custom_config`` is sane and returns a sanitized dict <str -> (str|object)>
+
+ It checks if keys are all strings and sanitizes values of a given dictionary as follows:
+
+ - If string, number or boolean is given as a value, it is converted to string.
+ For string and number (int, float), it is converted to string by a built-in ``str()`` method.
+ For a boolean value, ``True`` is converted to "true" instead of "True", and ``False`` is
+ converted to "false" instead of "False", in order to keep the consistency with
+ Java configuration.
+
+ - If neither of the above is given as a value, it is inserted into the sanitized dict as it is.
+ These values will need to be serialized before adding to a protobuf message.
+ """
+ if not isinstance(custom_config, dict):
+ raise TypeError("Component-specific configuration must be given as a dict type, given: %s"
+ % str(type(custom_config)))
+ sanitized = {}
+ for key, value in list(custom_config.items()):
+ if not isinstance(key, str):
+ raise TypeError("Key for component-specific configuration must be string, given: %s:%s"
+ % (str(type(key)), str(key)))
+
+ if isinstance(value, bool):
+ sanitized[key] = "true" if value else "false"
+ elif isinstance(value, (str, int, float)):
+ sanitized[key] = str(value)
+ else:
+ sanitized[key] = value
+
+ return sanitized
+
+ def _add_in_streams(self, bolt):
+ """Adds inputs to a given protobuf Bolt message"""
+ if self.inputs is None:
+ return
+ # sanitize inputs and get a map <GlobalStreamId -> Grouping>
+ input_dict = self._sanitize_inputs()
+
+ for global_streamid, gtype in list(input_dict.items()):
+ in_stream = bolt.inputs.add()
+ in_stream.stream.CopyFrom(self._get_stream_id(global_streamid.component_id,
+ global_streamid.stream_id))
+ if isinstance(gtype, Grouping.FIELDS):
+ # it's a field grouping
+ in_stream.gtype = gtype.gtype
+ in_stream.grouping_fields.CopyFrom(self._get_stream_schema(gtype.fields))
+ elif isinstance(gtype, Grouping.CUSTOM):
+ # it's a custom grouping
+ in_stream.gtype = gtype.gtype
+ in_stream.custom_grouping_object = gtype.python_serialized
+ in_stream.type = topology_pb2.CustomGroupingObjectType.Value("PYTHON_OBJECT")
+ else:
+ in_stream.gtype = gtype
+
+ # pylint: disable=too-many-branches
+ def _sanitize_inputs(self):
+ """Sanitizes input fields and returns a map <GlobalStreamId -> Grouping>"""
+ ret = {}
+ if self.inputs is None:
+ return
+
+ if isinstance(self.inputs, dict):
+ # inputs are dictionary, must be either <HeronComponentSpec -> Grouping> or
+ # <GlobalStreamId -> Grouping>
+ for key, grouping in list(self.inputs.items()):
+ if not Grouping.is_grouping_sane(grouping):
+ raise ValueError('A given grouping is not supported')
+ if isinstance(key, HeronComponentSpec):
+ # use default streamid
+ if key.name is None:
+ # should not happen as TopologyType metaclass sets name attribute
+ # before calling this method
+ raise RuntimeError("In _sanitize_inputs(): HeronComponentSpec doesn't have a name")
+ global_streamid = GlobalStreamId(key.name, Stream.DEFAULT_STREAM_ID)
+ ret[global_streamid] = grouping
+ elif isinstance(key, GlobalStreamId):
+ ret[key] = grouping
+ else:
+ raise ValueError("%s is not supported as a key to inputs" % str(key))
+ elif isinstance(self.inputs, (list, tuple)):
+ # inputs are lists, must be either a list of HeronComponentSpec or GlobalStreamId
+ # will use SHUFFLE grouping
+ for input_obj in self.inputs:
+ if isinstance(input_obj, HeronComponentSpec):
+ if input_obj.name is None:
+ # should not happen as TopologyType metaclass sets name attribute
+ # before calling this method
+ raise RuntimeError("In _sanitize_inputs(): HeronComponentSpec doesn't have a name")
+ global_streamid = GlobalStreamId(input_obj.name, Stream.DEFAULT_STREAM_ID)
+ ret[global_streamid] = Grouping.SHUFFLE
+ elif isinstance(input_obj, GlobalStreamId):
+ ret[input_obj] = Grouping.SHUFFLE
+ else:
+ raise ValueError("%s is not supported as an input" % str(input_obj))
+ else:
+ raise TypeError("Inputs must be a list, dict, or None, given: %s" % str(self.inputs))
+
+ return ret
+
+ def _add_out_streams(self, spbl):
+ """Adds outputs to a given protobuf Bolt or Spout message"""
+ if self.outputs is None:
+ return
+
+ # sanitize outputs and get a map <stream_id -> out fields>
+ output_map = self._sanitize_outputs()
+
+ for stream_id, out_fields in list(output_map.items()):
+ out_stream = spbl.outputs.add()
+ out_stream.stream.CopyFrom(self._get_stream_id(self.name, stream_id))
+ out_stream.schema.CopyFrom(self._get_stream_schema(out_fields))
+
+ def _sanitize_outputs(self):
+ """Sanitizes output fields and returns a map <stream_id -> list of output fields>"""
+ ret = {}
+ if self.outputs is None:
+ return
+
+ if not isinstance(self.outputs, (list, tuple)):
+ raise TypeError("Argument to outputs must be either list or tuple, given: %s"
+ % str(type(self.outputs)))
+
+ for output in self.outputs:
+ if not isinstance(output, (str, Stream)):
+ raise TypeError("Outputs must be a list of strings or Streams, given: %s" % str(output))
+
+ if isinstance(output, str):
+ # it's a default stream
+ if Stream.DEFAULT_STREAM_ID not in ret:
+ ret[Stream.DEFAULT_STREAM_ID] = list()
+ ret[Stream.DEFAULT_STREAM_ID].append(output)
+ else:
+ # output is a Stream object
+ if output.stream_id == Stream.DEFAULT_STREAM_ID and Stream.DEFAULT_STREAM_ID in ret:
+ # some default stream fields are already in there
+ ret[Stream.DEFAULT_STREAM_ID].extend(output.fields)
+ else:
+ ret[output.stream_id] = output.fields
+ return ret
+
+ def get_out_streamids(self):
+ """Returns a set of output stream ids registered for this component"""
+ if self.outputs is None:
+ return set()
+
+ if not isinstance(self.outputs, (list, tuple)):
+ raise TypeError("Argument to outputs must be either list or tuple, given: %s"
+ % str(type(self.outputs)))
+ ret_lst = []
+ for output in self.outputs:
+ if not isinstance(output, (str, Stream)):
+ raise TypeError("Outputs must be a list of strings or Streams, given: %s" % str(output))
+ ret_lst.append(Stream.DEFAULT_STREAM_ID if isinstance(output, str) else output.stream_id)
+ return set(ret_lst)
+
+ def __getitem__(self, stream_id):
+ """Get GlobalStreamId for a given stream_id"""
+ if stream_id not in self.get_out_streamids():
+ raise ValueError("A given stream id does not exist on this component: %s" % stream_id)
+
+ component_id = self.name or self
+ return GlobalStreamId(componentId=component_id, streamId=stream_id)
+
+ @staticmethod
+ def _get_stream_id(comp_name, stream_id):
+ """Returns a StreamId protobuf message"""
+ proto_stream_id = topology_pb2.StreamId()
+ proto_stream_id.id = stream_id
+ proto_stream_id.component_name = comp_name
+ return proto_stream_id
+
+ @staticmethod
+ def _get_stream_schema(fields):
+ """Returns a StreamSchema protobuf message"""
+ stream_schema = topology_pb2.StreamSchema()
+ for field in fields:
+ key = stream_schema.keys.add()
+ key.key = field
+ key.type = topology_pb2.Type.Value("OBJECT")
+
+ return stream_schema
+
+class GlobalStreamId(object):
+ """Wrapper class to define stream_id and its component name
+
+ Constructor method is compatible with StreamParse's GlobalStreamId class, although
+ the object itself is completely different, as Heron does not use Thrift.
+ This is mainly used for declaring input fields when defining a topology, and internally
+ in HeronComponentSpec.
+
+ Note that topology writers never have to create an instance of this class by themselves,
+ as it is created automatically.
+ """
+ def __init__(self, componentId, streamId):
+ """
+ :type componentId: str or HeronComponentSpec
+ :param componentId: component id from which the tuple is emitted, or HeronComponentSpec object.
+ :type streamId: str
+ :param streamId: stream id through which the tuple is transmitted
+ """
+ if not isinstance(componentId, (str, HeronComponentSpec)):
+ raise TypeError('GlobalStreamId: componentId must be either string or HeronComponentSpec')
+ if not isinstance(streamId, str):
+ raise TypeError('GlobalStreamId: streamId must be string type')
+
+ self._component_id = componentId
+ self.stream_id = streamId
+
+ @property
+ def component_id(self):
+ """Returns component_id of this GlobalStreamId
+
+ Note that if HeronComponentSpec is specified as componentId and its name is not yet
+ available (i.e. when ``name`` argument was not given in ``spec()`` method in Bolt or Spout),
+ this property returns a message with uuid. However, this is provided only for safety
+ with __eq__(), __str__(), and __hash__() methods, and not meant to be called explicitly
+ before TopologyType class finally sets the name attribute of HeronComponentSpec.
+ """
+ if isinstance(self._component_id, HeronComponentSpec):
+ if self._component_id.name is None:
+ # HeronComponentSpec instance's name attribute might not be available until
+ # TopologyType metaclass finally sets it. This statement is to support __eq__(),
+ # __hash__() and __str__() methods with safety, as raising Exception is not
+ # appropriate this case.
+ return "<No name available for HeronComponentSpec yet, uuid: %s>" % self._component_id.uuid
+ return self._component_id.name
+ elif isinstance(self._component_id, str):
+ return self._component_id
+ else:
+ raise ValueError("Component Id for this GlobalStreamId is not properly set: <%s:%s>"
+ % (str(type(self._component_id)), str(self._component_id)))
+
+ def __eq__(self, other):
+ return hasattr(other, 'component_id') and self.component_id == other.component_id \
+ and hasattr(other, 'stream_id') and self.stream_id == other.stream_id
+
+ def __hash__(self):
+ return hash(self.__str__())
+
+ def __str__(self):
+ return "%s:%s" % (self.component_id, self.stream_id)
+</code></pre>
+ </div>
+
+ </header>
+
+ <section id="section-items">
+ <h2 class="section-title" id="header-variables">Module variables</h2>
+ <div class="item">
+ <p id="heronpy.api.component.component_spec.TOPOLOGY_COMPONENT_PARALLELISM" class="name">var <span class="ident">TOPOLOGY_COMPONENT_PARALLELISM</span></p>
+
+
+ <div class="source_cont">
+</div>
+
+ </div>
+
+
+ <h2 class="section-title" id="header-classes">Classes</h2>
+
+ <div class="item">
+ <p id="heronpy.api.component.component_spec.GlobalStreamId" class="name">class <span class="ident">GlobalStreamId</span></p>
+
+
+ <div class="desc"><p>Wrapper class to define stream_id and its component name</p>
+<p>Constructor method is compatible with StreamParse's GlobalStreamId class, although
+the object itself is completely different, as Heron does not use Thrift.
+This is mainly used for declaring input fields when defining a topology, and internally
+in HeronComponentSpec.</p>
+<p>Note that topology writers never have to create an instance of this class by themselves,
+as it is created automatically.</p></div>
+ <div class="source_cont">
+ <p class="source_link"><a href="javascript:void(0);" onclick="toggle('source-heronpy.api.component.component_spec.GlobalStreamId', this);">Show source ≡</a></p>
+ <div id="source-heronpy.api.component.component_spec.GlobalStreamId" class="source">
+ <pre><code>class GlobalStreamId(object):
+ """Wrapper class to define stream_id and its component name
+
+ Constructor method is compatible with StreamParse's GlobalStreamId class, although
+ the object itself is completely different, as Heron does not use Thrift.
+ This is mainly used for declaring input fields when defining a topology, and internally
+ in HeronComponentSpec.
+
+ Note that topology writers never have to create an instance of this class by themselves,
+ as it is created automatically.
+ """
+ def __init__(self, componentId, streamId):
+ """
+ :type componentId: str or HeronComponentSpec
+ :param componentId: component id from which the tuple is emitted, or HeronComponentSpec object.
+ :type streamId: str
+ :param streamId: stream id through which the tuple is transmitted
+ """
+ if not isinstance(componentId, (str, HeronComponentSpec)):
+ raise TypeError('GlobalStreamId: componentId must be either string or HeronComponentSpec')
+ if not isinstance(streamId, str):
+ raise TypeError('GlobalStreamId: streamId must be string type')
+
+ self._component_id = componentId
+ self.stream_id = streamId
+
+ @property
+ def component_id(self):
+ """Returns component_id of this GlobalStreamId
+
+ Note that if HeronComponentSpec is specified as componentId and its name is not yet
+ available (i.e. when ``name`` argument was not given in ``spec()`` method in Bolt or Spout),
+ this property returns a message with uuid. However, this is provided only for safety
+ with __eq__(), __str__(), and __hash__() methods, and not meant to be called explicitly
+ before TopologyType class finally sets the name attribute of HeronComponentSpec.
+ """
+ if isinstance(self._component_id, HeronComponentSpec):
+ if self._component_id.name is None:
+ # HeronComponentSpec instance's name attribute might not be available until
+ # TopologyType metaclass finally sets it. This statement is to support __eq__(),
+ # __hash__() and __str__() methods with safety, as raising Exception is not
+ # appropriate this case.
+ return "<No name available for HeronComponentSpec yet, uuid: %s>" % self._component_id.uuid
+ return self._component_id.name
+ elif isinstance(self._component_id, str):
+ return self._component_id
+ else:
+ raise ValueError("Component Id for this GlobalStreamId is not properly set: <%s:%s>"
+ % (str(type(self._component_id)), str(self._component_id)))
+
+ def __eq__(self, other):
+ return hasattr(other, 'component_id') and self.component_id == other.component_id \
+ and hasattr(other, 'stream_id') and self.stream_id == other.stream_id
+
+ def __hash__(self):
+ return hash(self.__str__())
+
+ def __str__(self):
+ return "%s:%s" % (self.component_id, self.stream_id)
+</code></pre>
+ </div>
+</div>
+
+
+ <div class="class">
+ <h3>Ancestors (in MRO)</h3>
+ <ul class="class_list">
+ <li><a href="#heronpy.api.component.component_spec.GlobalStreamId">GlobalStreamId</a></li>
+ <li>__builtin__.object</li>
+ </ul>
+ <h3>Instance variables</h3>
+ <div class="item">
+ <p id="heronpy.api.component.component_spec.GlobalStreamId.component_id" class="name">var <span class="ident">component_id</span></p>
+
+
+
+
+ <div class="desc"><p>Returns component_id of this GlobalStreamId</p>
+<p>Note that if HeronComponentSpec is specified as componentId and its name is not yet
+available (i.e. when <code>name</code> argument was not given in <code>spec()</code> method in Bolt or Spout),
+this property returns a message with uuid. However, this is provided only for safety
+with <strong>eq</strong>(), <strong>str</strong>(), and <strong>hash</strong>() methods, and not meant to be called explicitly
+before TopologyType class finally sets the name attribute of HeronComponentSpec.</p></div>
+ <div class="source_cont">
+</div>
+
+ </div>
+ <div class="item">
+ <p id="heronpy.api.component.component_spec.GlobalStreamId.stream_id" class="name">var <span class="ident">stream_id</span></p>
+
+
+
+
+ <div class="source_cont">
+</div>
+
+ </div>
+ <h3>Methods</h3>
+
+ <div class="item">
+ <div class="name def" id="heronpy.api.component.component_spec.GlobalStreamId.__init__">
+ <p>def <span class="ident">__init__</span>(</p><p>self, componentId, streamId)</p>
+ </div>
+
+
+
+
+ <div class="desc"><p>:type componentId: str or HeronComponentSpec
+:param componentId: component id from which the tuple is emitted, or HeronComponentSpec object.
+:type streamId: str
+:param streamId: stream id through which the tuple is transmitted</p></div>
+ <div class="source_cont">
+ <p class="source_link"><a href="javascript:void(0);" onclick="toggle('source-heronpy.api.component.component_spec.GlobalStreamId.__init__', this);">Show source ≡</a></p>
+ <div id="source-heronpy.api.component.component_spec.GlobalStreamId.__init__" class="source">
+ <pre><code>def __init__(self, componentId, streamId):
+ """
+ :type componentId: str or HeronComponentSpec
+ :param componentId: component id from which the tuple is emitted, or HeronComponentSpec object.
+ :type streamId: str
+ :param streamId: stream id through which the tuple is transmitted
+ """
+ if not isinstance(componentId, (str, HeronComponentSpec)):
+ raise TypeError('GlobalStreamId: componentId must be either string or HeronComponentSpec')
+ if not isinstance(streamId, str):
+ raise TypeError('GlobalStreamId: streamId must be string type')
+ self._component_id = componentId
+ self.stream_id = streamId
+</code></pre>
+ </div>
+</div>
+
+ </div>
+
+ </div>
+ </div>
+
+ <div class="item">
+ <p id="heronpy.api.component.component_spec.HeronComponentSpec" class="name">class <span class="ident">HeronComponentSpec</span></p>
+
+
+ <div class="desc"><p>Class to specify the information and location of components in a topology</p>
+<p>This class is generated by the <code>spec()</code> method of Spout and Bolt class and
+specifies how this component is located in the topology and how it is connected to
+the other components. This class also retains the Python class path of the component,
+so pex_loader can load the component appropriately.</p></div>
+ <div class="source_cont">
+ <p class="source_link"><a href="javascript:void(0);" onclick="toggle('source-heronpy.api.component.component_spec.HeronComponentSpec', this);">Show source ≡</a></p>
+ <div id="source-heronpy.api.component.component_spec.HeronComponentSpec" class="source">
+ <pre><code>class HeronComponentSpec(object):
+ """Class to specify the information and location of components in a topology
+
+ This class is generated by the ``spec()`` method of Spout and Bolt class and
+ specifies how this component is located in the topology and how it is connected to
+ the other components. This class also retains the Python class path of the component,
+ so pex_loader can load the component appropriately.
+ """
+ def __init__(self, name, python_class_path, is_spout, par,
+ inputs=None, outputs=None, config=None):
+ self._sanitize_args(name, python_class_path, is_spout, par)
+
+ self.name = name
+ self.python_class_path = python_class_path
+ self.is_spout = is_spout
+ self.parallelism = par
+
+ # inputs, outputs, config will be sanitized later
+ self.inputs = inputs
+ self.outputs = outputs
+ self.custom_config = config
+
+ # This is used for identification, especially when name is not specified by argument
+ # Note that ``self.name`` might not be available until it is set by TopologyType metaclass
+ # so this is necessary for identification purposes. Used mainly by GlobalStreamId.
+ self.uuid = str(uuid.uuid4())
+
+ @staticmethod
+ def _sanitize_args(name, py_class_path, is_spout, par):
+ # name can be None at the time this spec is initialized
+ assert name is None or isinstance(name, str)
+ assert isinstance(py_class_path, str)
+ assert isinstance(is_spout, bool)
+ assert isinstance(par, int) and par > 0
+
+ def get_protobuf(self):
+ """Returns protobuf message (Spout or Bolt) of this component"""
+ if self.is_spout:
+ return self._get_spout()
+ else:
+ return self._get_bolt()
+
+ def _get_spout(self):
+ """Returns Spout protobuf message"""
+ spout = topology_pb2.Spout()
+ spout.comp.CopyFrom(self._get_base_component())
+
+ # Add output streams
+ self._add_out_streams(spout)
+ return spout
+
+ def _get_bolt(self):
+ """Returns Bolt protobuf message"""
+ bolt = topology_pb2.Bolt()
+ bolt.comp.CopyFrom(self._get_base_component())
+
+ # Add streams
+ self._add_in_streams(bolt)
+ self._add_out_streams(bolt)
+ return bolt
+
+ def _get_base_component(self):
+ """Returns Component protobuf message"""
+ comp = topology_pb2.Component()
+ comp.name = self.name
+ comp.spec = topology_pb2.ComponentObjectSpec.Value("PYTHON_CLASS_NAME")
+ comp.class_name = self.python_class_path
+ comp.config.CopyFrom(self._get_comp_config())
+ return comp
+
+ def _get_comp_config(self):
+ """Returns component-specific Config protobuf message
+
+ It first adds ``topology.component.parallelism``, and is overriden by
+ a user-defined component-specific configuration, specified by spec().
+ """
+ proto_config = topology_pb2.Config()
+
+ # first add parallelism
+ key = proto_config.kvs.add()
+ key.key = TOPOLOGY_COMPONENT_PARALLELISM
+ key.value = str(self.parallelism)
+ key.type = topology_pb2.ConfigValueType.Value("STRING_VALUE")
+
+ # iterate through self.custom_config
+ if self.custom_config is not None:
+ sanitized = self._sanitize_config(self.custom_config)
+ for key, value in list(sanitized.items()):
+ if isinstance(value, str):
+ kvs = proto_config.kvs.add()
+ kvs.key = key
+ kvs.value = value
+ kvs.type = topology_pb2.ConfigValueType.Value("STRING_VALUE")
+ else:
+ # need to serialize
+ kvs = proto_config.kvs.add()
+ kvs.key = key
+ kvs.serialized_value = default_serializer.serialize(value)
+ kvs.type = topology_pb2.ConfigValueType.Value("PYTHON_SERIALIZED_VALUE")
+
+ return proto_config
+
+ @staticmethod
+ def _sanitize_config(custom_config):
+ """Checks whether ``custom_config`` is sane and returns a sanitized dict <str -> (str|object)>
+
+ It checks if keys are all strings and sanitizes values of a given dictionary as follows:
+
+ - If string, number or boolean is given as a value, it is converted to string.
+ For string and number (int, float), it is converted to string by a built-in ``str()`` method.
+ For a boolean value, ``True`` is converted to "true" instead of "True", and ``False`` is
+ converted to "false" instead of "False", in order to keep the consistency with
+ Java configuration.
+
+ - If neither of the above is given as a value, it is inserted into the sanitized dict as it is.
+ These values will need to be serialized before adding to a protobuf message.
+ """
+ if not isinstance(custom_config, dict):
+ raise TypeError("Component-specific configuration must be given as a dict type, given: %s"
+ % str(type(custom_config)))
+ sanitized = {}
+ for key, value in list(custom_config.items()):
+ if not isinstance(key, str):
+ raise TypeError("Key for component-specific configuration must be string, given: %s:%s"
+ % (str(type(key)), str(key)))
+
+ if isinstance(value, bool):
+ sanitized[key] = "true" if value else "false"
+ elif isinstance(value, (str, int, float)):
+ sanitized[key] = str(value)
+ else:
+ sanitized[key] = value
+
+ return sanitized
+
+ def _add_in_streams(self, bolt):
+ """Adds inputs to a given protobuf Bolt message"""
+ if self.inputs is None:
+ return
+ # sanitize inputs and get a map <GlobalStreamId -> Grouping>
+ input_dict = self._sanitize_inputs()
+
+ for global_streamid, gtype in list(input_dict.items()):
+ in_stream = bolt.inputs.add()
+ in_stream.stream.CopyFrom(self._get_stream_id(global_streamid.component_id,
+ global_streamid.stream_id))
+ if isinstance(gtype, Grouping.FIELDS):
+ # it's a field grouping
+ in_stream.gtype = gtype.gtype
+ in_stream.grouping_fields.CopyFrom(self._get_stream_schema(gtype.fields))
+ elif isinstance(gtype, Grouping.CUSTOM):
+ # it's a custom grouping
+ in_stream.gtype = gtype.gtype
+ in_stream.custom_grouping_object = gtype.python_serialized
+ in_stream.type = topology_pb2.CustomGroupingObjectType.Value("PYTHON_OBJECT")
+ else:
+ in_stream.gtype = gtype
+
+ # pylint: disable=too-many-branches
+ def _sanitize_inputs(self):
+ """Sanitizes input fields and returns a map <GlobalStreamId -> Grouping>"""
+ ret = {}
+ if self.inputs is None:
+ return
+
+ if isinstance(self.inputs, dict):
+ # inputs are dictionary, must be either <HeronComponentSpec -> Grouping> or
+ # <GlobalStreamId -> Grouping>
+ for key, grouping in list(self.inputs.items()):
+ if not Grouping.is_grouping_sane(grouping):
+ raise ValueError('A given grouping is not supported')
+ if isinstance(key, HeronComponentSpec):
+ # use default streamid
+ if key.name is None:
+ # should not happen as TopologyType metaclass sets name attribute
+ # before calling this method
+ raise RuntimeError("In _sanitize_inputs(): HeronComponentSpec doesn't have a name")
+ global_streamid = GlobalStreamId(key.name, Stream.DEFAULT_STREAM_ID)
+ ret[global_streamid] = grouping
+ elif isinstance(key, GlobalStreamId):
+ ret[key] = grouping
+ else:
+ raise ValueError("%s is not supported as a key to inputs" % str(key))
+ elif isinstance(self.inputs, (list, tuple)):
+ # inputs are lists, must be either a list of HeronComponentSpec or GlobalStreamId
+ # will use SHUFFLE grouping
+ for input_obj in self.inputs:
+ if isinstance(input_obj, HeronComponentSpec):
+ if input_obj.name is None:
+ # should not happen as TopologyType metaclass sets name attribute
+ # before calling this method
+ raise RuntimeError("In _sanitize_inputs(): HeronComponentSpec doesn't have a name")
+ global_streamid = GlobalStreamId(input_obj.name, Stream.DEFAULT_STREAM_ID)
+ ret[global_streamid] = Grouping.SHUFFLE
+ elif isinstance(input_obj, GlobalStreamId):
+ ret[input_obj] = Grouping.SHUFFLE
+ else:
+ raise ValueError("%s is not supported as an input" % str(input_obj))
+ else:
+ raise TypeError("Inputs must be a list, dict, or None, given: %s" % str(self.inputs))
+
+ return ret
+
+ def _add_out_streams(self, spbl):
+ """Adds outputs to a given protobuf Bolt or Spout message"""
+ if self.outputs is None:
+ return
+
+ # sanitize outputs and get a map <stream_id -> out fields>
+ output_map = self._sanitize_outputs()
+
+ for stream_id, out_fields in list(output_map.items()):
+ out_stream = spbl.outputs.add()
+ out_stream.stream.CopyFrom(self._get_stream_id(self.name, stream_id))
+ out_stream.schema.CopyFrom(self._get_stream_schema(out_fields))
+
+ def _sanitize_outputs(self):
+ """Sanitizes output fields and returns a map <stream_id -> list of output fields>"""
+ ret = {}
+ if self.outputs is None:
+ return
+
+ if not isinstance(self.outputs, (list, tuple)):
+ raise TypeError("Argument to outputs must be either list or tuple, given: %s"
+ % str(type(self.outputs)))
+
+ for output in self.outputs:
+ if not isinstance(output, (str, Stream)):
+ raise TypeError("Outputs must be a list of strings or Streams, given: %s" % str(output))
+
+ if isinstance(output, str):
+ # it's a default stream
+ if Stream.DEFAULT_STREAM_ID not in ret:
+ ret[Stream.DEFAULT_STREAM_ID] = list()
+ ret[Stream.DEFAULT_STREAM_ID].append(output)
+ else:
+ # output is a Stream object
+ if output.stream_id == Stream.DEFAULT_STREAM_ID and Stream.DEFAULT_STREAM_ID in ret:
+ # some default stream fields are already in there
+ ret[Stream.DEFAULT_STREAM_ID].extend(output.fields)
+ else:
+ ret[output.stream_id] = output.fields
+ return ret
+
+ def get_out_streamids(self):
+ """Returns a set of output stream ids registered for this component"""
+ if self.outputs is None:
+ return set()
+
+ if not isinstance(self.outputs, (list, tuple)):
+ raise TypeError("Argument to outputs must be either list or tuple, given: %s"
+ % str(type(self.outputs)))
+ ret_lst = []
+ for output in self.outputs:
+ if not isinstance(output, (str, Stream)):
+ raise TypeError("Outputs must be a list of strings or Streams, given: %s" % str(output))
+ ret_lst.append(Stream.DEFAULT_STREAM_ID if isinstance(output, str) else output.stream_id)
+ return set(ret_lst)
+
+ def __getitem__(self, stream_id):
+ """Get GlobalStreamId for a given stream_id"""
+ if stream_id not in self.get_out_streamids():
+ raise ValueError("A given stream id does not exist on this component: %s" % stream_id)
+
+ component_id = self.name or self
+ return GlobalStreamId(componentId=component_id, streamId=stream_id)
+
+ @staticmethod
+ def _get_stream_id(comp_name, stream_id):
+ """Returns a StreamId protobuf message"""
+ proto_stream_id = topology_pb2.StreamId()
+ proto_stream_id.id = stream_id
+ proto_stream_id.component_name = comp_name
+ return proto_stream_id
+
+ @staticmethod
+ def _get_stream_schema(fields):
+ """Returns a StreamSchema protobuf message"""
+ stream_schema = topology_pb2.StreamSchema()
+ for field in fields:
+ key = stream_schema.keys.add()
+ key.key = field
+ key.type = topology_pb2.Type.Value("OBJECT")
+
+ return stream_schema
+</code></pre>
+ </div>
+</div>
+
+
+ <div class="class">
+ <h3>Ancestors (in MRO)</h3>
+ <ul class="class_list">
+ <li><a href="#heronpy.api.component.component_spec.HeronComponentSpec">HeronComponentSpec</a></li>
+ <li>__builtin__.object</li>
+ </ul>
+ <h3>Instance variables</h3>
+ <div class="item">
+ <p id="heronpy.api.component.component_spec.HeronComponentSpec.custom_config" class="name">var <span class="ident">custom_config</span></p>
+
+
+
+
+ <div class="source_cont">
+</div>
+
+ </div>
+ <div class="item">
+ <p id="heronpy.api.component.component_spec.HeronComponentSpec.inputs" class="name">var <span class="ident">inputs</span></p>
+
+
+
+
+ <div class="source_cont">
+</div>
+
+ </div>
+ <div class="item">
+ <p id="heronpy.api.component.component_spec.HeronComponentSpec.is_spout" class="name">var <span class="ident">is_spout</span></p>
+
+
+
+
+ <div class="source_cont">
+</div>
+
+ </div>
+ <div class="item">
+ <p id="heronpy.api.component.component_spec.HeronComponentSpec.name" class="name">var <span class="ident">name</span></p>
+
+
+
+
+ <div class="source_cont">
+</div>
+
+ </div>
+ <div class="item">
+ <p id="heronpy.api.component.component_spec.HeronComponentSpec.outputs" class="name">var <span class="ident">outputs</span></p>
+
+
+
+
+ <div class="source_cont">
+</div>
+
+ </div>
+ <div class="item">
+ <p id="heronpy.api.component.component_spec.HeronComponentSpec.parallelism" class="name">var <span class="ident">parallelism</span></p>
+
+
+
+
+ <div class="source_cont">
+</div>
+
+ </div>
+ <div class="item">
+ <p id="heronpy.api.component.component_spec.HeronComponentSpec.python_class_path" class="name">var <span class="ident">python_class_path</span></p>
+
+
+
+
+ <div class="source_cont">
+</div>
+
+ </div>
+ <div class="item">
+ <p id="heronpy.api.component.component_spec.HeronComponentSpec.uuid" class="name">var <span class="ident">uuid</span></p>
+
+
+
+
+ <div class="source_cont">
+</div>
+
+ </div>
+ <h3>Methods</h3>
+
+ <div class="item">
+ <div class="name def" id="heronpy.api.component.component_spec.HeronComponentSpec.__init__">
+ <p>def <span class="ident">__init__</span>(</p><p>self, name, python_class_path, is_spout, par, inputs=None, outputs=None, config=None)</p>
+ </div>
+
+
+
+
+ <div class="source_cont">
+ <p class="source_link"><a href="javascript:void(0);" onclick="toggle('source-heronpy.api.component.component_spec.HeronComponentSpec.__init__', this);">Show source ≡</a></p>
+ <div id="source-heronpy.api.component.component_spec.HeronComponentSpec.__init__" class="source">
+ <pre><code>def __init__(self, name, python_class_path, is_spout, par,
+ inputs=None, outputs=None, config=None):
+ self._sanitize_args(name, python_class_path, is_spout, par)
+ self.name = name
+ self.python_class_path = python_class_path
+ self.is_spout = is_spout
+ self.parallelism = par
+ # inputs, outputs, config will be sanitized later
+ self.inputs = inputs
+ self.outputs = outputs
+ self.custom_config = config
+ # This is used for identification, especially when name is not specified by argument
+ # Note that ``self.name`` might not be available until it is set by TopologyType metaclass
+ # so this is necessary for identification purposes. Used mainly by GlobalStreamId.
+ self.uuid = str(uuid.uuid4())
+</code></pre>
+ </div>
+</div>
+
+ </div>
+
+
+ <div class="item">
+ <div class="name def" id="heronpy.api.component.component_spec.HeronComponentSpec.get_out_streamids">
+ <p>def <span class="ident">get_out_streamids</span>(</p><p>self)</p>
+ </div>
+
+
+
+
+ <div class="desc"><p>Returns a set of output stream ids registered for this component</p></div>
+ <div class="source_cont">
+ <p class="source_link"><a href="javascript:void(0);" onclick="toggle('source-heronpy.api.component.component_spec.HeronComponentSpec.get_out_streamids', this);">Show source ≡</a></p>
+ <div id="source-heronpy.api.component.component_spec.HeronComponentSpec.get_out_streamids" class="source">
+ <pre><code>def get_out_streamids(self):
+ """Returns a set of output stream ids registered for this component"""
+ if self.outputs is None:
+ return set()
+ if not isinstance(self.outputs, (list, tuple)):
+ raise TypeError("Argument to outputs must be either list or tuple, given: %s"
+ % str(type(self.outputs)))
+ ret_lst = []
+ for output in self.outputs:
+ if not isinstance(output, (str, Stream)):
+ raise TypeError("Outputs must be a list of strings or Streams, given: %s" % str(output))
+ ret_lst.append(Stream.DEFAULT_STREAM_ID if isinstance(output, str) else output.stream_id)
+ return set(ret_lst)
+</code></pre>
+ </div>
+</div>
+
+ </div>
+
+
+ <div class="item">
+ <div class="name def" id="heronpy.api.component.component_spec.HeronComponentSpec.get_protobuf">
+ <p>def <span class="ident">get_protobuf</span>(</p><p>self)</p>
+ </div>
+
+
+
+
+ <div class="desc"><p>Returns protobuf message (Spout or Bolt) of this component</p></div>
+ <div class="source_cont">
+ <p class="source_link"><a href="javascript:void(0);" onclick="toggle('source-heronpy.api.component.component_spec.HeronComponentSpec.get_protobuf', this);">Show source ≡</a></p>
+ <div id="source-heronpy.api.component.component_spec.HeronComponentSpec.get_protobuf" class="source">
+ <pre><code>def get_protobuf(self):
+ """Returns protobuf message (Spout or Bolt) of this component"""
+ if self.is_spout:
+ return self._get_spout()
+ else:
+ return self._get_bolt()
+</code></pre>
+ </div>
+</div>
+
+ </div>
+
+ </div>
+ </div>
+
+ </section>
+
+ </article>
+ <div class="clear"> </div>
+ <footer id="footer">
+ <p>
+ Documentation generated by
+ <a href="https://github.com/BurntSushi/pdoc">pdoc 0.3.2</a>
+ </p>
+
+ <p>pdoc is in the public domain with the
+ <a href="http://unlicense.org">UNLICENSE</a></p>
+
+ <p>Design by <a href="http://nadh.in">Kailash Nadh</a></p>
+ </footer>
+</div>
+</body>
+</html>
diff --git a/content/api/python/api/component/index.html b/content/api/python/api/component/index.html
index 06d2fde..7cc6ed2 100644
--- a/content/api/python/api/component/index.html
+++ b/content/api/python/api/component/index.html
@@ -975,6 +975,7 @@ table {
<li class="set"><h3><a href="#header-submodules">Sub-modules</a></h3>
<ul>
<li class="mono"><a href="base_component.m.html">heronpy.api.component.base_component</a></li>
+ <li class="mono"><a href="component_spec.m.html">heronpy.api.component.component_spec</a></li>
</ul>
</li>
</ul>
@@ -1028,6 +1029,13 @@ __import__('pkg_resources').declare_namespace(__name__)
<div class="desc"><p>base_component.py</p></div>
</div>
+ <div class="item">
+ <p class="name"><a href="component_spec.m.html">heronpy.api.component.component_spec</a></p>
+
+
+ <div class="desc"><p>component_spec.py</p></div>
+
+ </div>
</section>
</article>
diff --git a/content/api/python/api/index.html b/content/api/python/api/index.html
index 87a913e..3880aed 100644
--- a/content/api/python/api/index.html
+++ b/content/api/python/api/index.html
@@ -984,7 +984,9 @@ table {
<li class="mono"><a href="serializer.m.html">heronpy.api.serializer</a></li>
<li class="mono"><a href="spout/index.html">heronpy.api.spout</a></li>
<li class="mono"><a href="state/index.html">heronpy.api.state</a></li>
+ <li class="mono"><a href="stream.m.html">heronpy.api.stream</a></li>
<li class="mono"><a href="task_hook.m.html">heronpy.api.task_hook</a></li>
+ <li class="mono"><a href="topology.m.html">heronpy.api.topology</a></li>
<li class="mono"><a href="topology_context.m.html">heronpy.api.topology_context</a></li>
<li class="mono"><a href="tuple.m.html">heronpy.api.tuple</a></li>
</ul>
@@ -1111,6 +1113,13 @@ T...</p></div>
</div>
<div class="item">
+ <p class="name"><a href="stream.m.html">heronpy.api.stream</a></p>
+
+
+ <div class="desc"><p>stream.py: module for defining Stream and Grouping for python topology</p></div>
+
+ </div>
+ <div class="item">
<p class="name"><a href="task_hook.m.html">heronpy.api.task_hook</a></p>
@@ -1118,6 +1127,13 @@ T...</p></div>
</div>
<div class="item">
+ <p class="name"><a href="topology.m.html">heronpy.api.topology</a></p>
+
+
+ <div class="desc"><p>topology.py: module for defining Heron topologies in Python</p></div>
+
+ </div>
+ <div class="item">
<p class="name"><a href="topology_context.m.html">heronpy.api.topology_context</a></p>
diff --git a/content/api/python/streamlet/index.html b/content/api/python/api/spout/base_spout.m.html
similarity index 51%
copy from content/api/python/streamlet/index.html
copy to content/api/python/api/spout/base_spout.m.html
index 44e674d..3df9875 100644
--- a/content/api/python/streamlet/index.html
+++ b/content/api/python/api/spout/base_spout.m.html
@@ -3,8 +3,8 @@
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1, minimum-scale=1" />
- <title>heronpy.streamlet API documentation</title>
- <meta name="description" content="module for heronpy streamlet API" />
+ <title>heronpy.api.spout.base_spout API documentation</title>
+ <meta name="description" content="base_spout.py" />
<link href='http://fonts.googleapis.com/css?family=Source+Sans+Pro:400,300' rel='stylesheet' type='text/css'>
@@ -971,20 +971,23 @@ table {
<ul id="index">
-
- <li class="set"><h3><a href="#header-submodules">Sub-modules</a></h3>
+ <li class="set"><h3><a href="#header-classes">Classes</a></h3>
<ul>
- <li class="mono"><a href="config.m.html">heronpy.streamlet.config</a></li>
- <li class="mono"><a href="context.m.html">heronpy.streamlet.context</a></li>
- <li class="mono"><a href="generator.m.html">heronpy.streamlet.generator</a></li>
- <li class="mono"><a href="impl/index.html">heronpy.streamlet.impl</a></li>
- <li class="mono"><a href="keyedwindow.m.html">heronpy.streamlet.keyedwindow</a></li>
- <li class="mono"><a href="resources.m.html">heronpy.streamlet.resources</a></li>
- <li class="mono"><a href="transformoperator.m.html">heronpy.streamlet.transformoperator</a></li>
- <li class="mono"><a href="window.m.html">heronpy.streamlet.window</a></li>
- <li class="mono"><a href="windowconfig.m.html">heronpy.streamlet.windowconfig</a></li>
+ <li class="mono">
+ <span class="class_name"><a href="#heronpy.api.spout.base_spout.BaseSpout">BaseSpout</a></span>
+
+
+ <ul>
+ <li class="mono"><a href="#heronpy.api.spout.base_spout.BaseSpout.__init__">__init__</a></li>
+ <li class="mono"><a href="#heronpy.api.spout.base_spout.BaseSpout.emit">emit</a></li>
+ <li class="mono"><a href="#heronpy.api.spout.base_spout.BaseSpout.log">log</a></li>
+ <li class="mono"><a href="#heronpy.api.spout.base_spout.BaseSpout.spec">spec</a></li>
+ </ul>
+
+ </li>
</ul>
</li>
+
</ul>
</div>
@@ -996,29 +999,105 @@ table {
<header id="section-intro">
- <h1 class="title"><span class="name">heronpy.streamlet</span> module</h1>
- <p>module for heronpy streamlet API</p>
+ <h1 class="title"><span class="name">heronpy.api.spout.base_spout</span> module</h1>
+ <p>base_spout.py</p>
- <p class="source_link"><a href="javascript:void(0);" onclick="toggle('source-heronpy.streamlet', this);">Show source ≡</a></p>
- <div id="source-heronpy.streamlet" class="source">
- <pre><code># Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
+ <p class="source_link"><a href="javascript:void(0);" onclick="toggle('source-heronpy.api.spout.base_spout', this);">Show source ≡</a></p>
+ <div id="source-heronpy.api.spout.base_spout" class="source">
+ <pre><code># Copyright 2016 - Parsely, Inc. (d/b/a Parse.ly)
#
-# http://www.apache.org/licenses/LICENSE-2.0
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
#
-# Unless required by applicable law or agreed to in writing,
-# software distributed under the License is distributed on an
-# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-# KIND, either express or implied. See the License for the
-# specific language governing permissions and limitations
-# under the License.
-'''module for heronpy streamlet API'''
-__import__('pkg_resources').declare_namespace(__name__)
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+'''base_spout.py'''
+import copy
+
+from heronpy.api.component.component_spec import HeronComponentSpec
+from heronpy.api.component.base_component import BaseComponent
+from heronpy.api.stream import Stream
+
+class BaseSpout(BaseComponent):
+ """BaseSpout class
+
+ This is the base for heron spout, which wraps the implementation of publicly available methods.
+ This includes:
+ - <classmethod> spec()
+ - emit()
+
+ They are compatible with StreamParse API.
+ """
+ # pylint: disable=no-member
+ @classmethod
+ def spec(cls, name=None, par=1, config=None, optional_outputs=None):
+ """Register this spout to the topology and create ``HeronComponentSpec``
+
+ The usage of this method is compatible with StreamParse API, although it does not create
+ ``ShellBoltSpec`` but instead directly registers to a ``Topology`` class.
+
+ This method takes an optional ``outputs`` argument for supporting dynamic output fields
+ declaration. However, it is recommended that ``outputs`` should be declared as
+ an attribute of your ``Spout`` subclass. Also, some ways of declaring inputs is not supported
+ in this implementation; please read the documentation below.
+
+ :type name: str
+ :param name: Name of this spout.
+ :type par: int
+ :param par: Parallelism hint for this spout.
+ :type config: dict
+ :param config: Component-specific config settings.
+ :type optional_outputs: list of (str or Stream) or tuple of (str or Stream)
+ :param optional_outputs: Additional output fields for this spout. These fields are added to
+ existing ``outputs`` class attributes of your spout.
+ This is an optional argument, and exists only for supporting dynamic
+ output field declaration.
+ """
+ python_class_path = "%s.%s" % (cls.__module__, cls.__name__)
+
+ if hasattr(cls, 'outputs'):
+ # avoid modification to cls.outputs
+ _outputs = copy.copy(cls.outputs)
+ else:
+ _outputs = []
+
+ if optional_outputs is not None:
+ assert isinstance(optional_outputs, (list, tuple))
+ for out in optional_outputs:
+ assert isinstance(out, (str, Stream))
+ _outputs.append(out)
+
+ return HeronComponentSpec(name, python_class_path, is_spout=True, par=par,
+ inputs=None, outputs=_outputs, config=config)
+
+ # pylint: disable=unused-argument
+ def emit(self, tup, tup_id=None, stream=Stream.DEFAULT_STREAM_ID,
+ direct_task=None, need_task_ids=False):
+ """Emits a new tuple from this Spout
+
+ It is compatible with StreamParse API.
+
+ :type tup: list or tuple
+ :param tup: the new output Tuple to send from this spout,
+ should contain only serializable data.
+ :type tup_id: str or object
+ :param tup_id: the ID for the Tuple. Leave this blank for an unreliable emit.
+ (Same as messageId in Java)
+ :type stream: str
+ :param stream: the ID of the stream this Tuple should be emitted to.
+ Leave empty to emit to the default stream.
+ :type direct_task: int
+ :param direct_task: the task to send the Tuple to if performing a direct emit.
+ :type need_task_ids: bool
+ :param need_task_ids: indicate whether or not you would like the task IDs the Tuple was emitted.
+ """
+ return self.delegate.emit(tup, tup_id, stream, direct_task, need_task_ids)
</code></pre>
</div>
@@ -1027,71 +1106,292 @@ __import__('pkg_resources').declare_namespace(__name__)
<section id="section-items">
-
- <h2 class="section-title" id="header-submodules">Sub-modules</h2>
- <div class="item">
- <p class="name"><a href="config.m.html">heronpy.streamlet.config</a></p>
+ <h2 class="section-title" id="header-classes">Classes</h2>
-
- <div class="desc"><p>config.py: module for defining config</p></div>
-
- </div>
<div class="item">
- <p class="name"><a href="context.m.html">heronpy.streamlet.context</a></p>
+ <p id="heronpy.api.spout.base_spout.BaseSpout" class="name">class <span class="ident">BaseSpout</span></p>
- <div class="desc"><p>context.py: module for defining context</p></div>
+ <div class="desc"><p>BaseSpout class</p>
+<p>This is the base for heron spout, which wraps the implementation of publicly available methods.
+This includes:
+ - <classmethod> spec()
+ - emit()</p>
+<p>They are compatible with StreamParse API.</p></div>
+ <div class="source_cont">
+ <p class="source_link"><a href="javascript:void(0);" onclick="toggle('source-heronpy.api.spout.base_spout.BaseSpout', this);">Show source ≡</a></p>
+ <div id="source-heronpy.api.spout.base_spout.BaseSpout" class="source">
+ <pre><code>class BaseSpout(BaseComponent):
+ """BaseSpout class
+
+ This is the base for heron spout, which wraps the implementation of publicly available methods.
+ This includes:
+ - <classmethod> spec()
+ - emit()
+
+ They are compatible with StreamParse API.
+ """
+ # pylint: disable=no-member
+ @classmethod
+ def spec(cls, name=None, par=1, config=None, optional_outputs=None):
+ """Register this spout to the topology and create ``HeronComponentSpec``
+
+ The usage of this method is compatible with StreamParse API, although it does not create
+ ``ShellBoltSpec`` but instead directly registers to a ``Topology`` class.
+
+ This method takes an optional ``outputs`` argument for supporting dynamic output fields
+ declaration. However, it is recommended that ``outputs`` should be declared as
+ an attribute of your ``Spout`` subclass. Also, some ways of declaring inputs is not supported
+ in this implementation; please read the documentation below.
+
+ :type name: str
+ :param name: Name of this spout.
+ :type par: int
+ :param par: Parallelism hint for this spout.
+ :type config: dict
+ :param config: Component-specific config settings.
+ :type optional_outputs: list of (str or Stream) or tuple of (str or Stream)
+ :param optional_outputs: Additional output fields for this spout. These fields are added to
+ existing ``outputs`` class attributes of your spout.
+ This is an optional argument, and exists only for supporting dynamic
+ output field declaration.
+ """
+ python_class_path = "%s.%s" % (cls.__module__, cls.__name__)
+
+ if hasattr(cls, 'outputs'):
+ # avoid modification to cls.outputs
+ _outputs = copy.copy(cls.outputs)
+ else:
+ _outputs = []
+
+ if optional_outputs is not None:
+ assert isinstance(optional_outputs, (list, tuple))
+ for out in optional_outputs:
+ assert isinstance(out, (str, Stream))
+ _outputs.append(out)
+
+ return HeronComponentSpec(name, python_class_path, is_spout=True, par=par,
+ inputs=None, outputs=_outputs, config=config)
+
+ # pylint: disable=unused-argument
+ def emit(self, tup, tup_id=None, stream=Stream.DEFAULT_STREAM_ID,
+ direct_task=None, need_task_ids=False):
+ """Emits a new tuple from this Spout
+
+ It is compatible with StreamParse API.
+
+ :type tup: list or tuple
+ :param tup: the new output Tuple to send from this spout,
+ should contain only serializable data.
+ :type tup_id: str or object
+ :param tup_id: the ID for the Tuple. Leave this blank for an unreliable emit.
+ (Same as messageId in Java)
+ :type stream: str
+ :param stream: the ID of the stream this Tuple should be emitted to.
+ Leave empty to emit to the default stream.
+ :type direct_task: int
+ :param direct_task: the task to send the Tuple to if performing a direct emit.
+ :type need_task_ids: bool
+ :param need_task_ids: indicate whether or not you would like the task IDs the Tuple was emitted.
+ """
+ return self.delegate.emit(tup, tup_id, stream, direct_task, need_task_ids)
+</code></pre>
+ </div>
+</div>
- </div>
- <div class="item">
- <p class="name"><a href="generator.m.html">heronpy.streamlet.generator</a></p>
-
+
+ <div class="class">
+ <h3>Ancestors (in MRO)</h3>
+ <ul class="class_list">
+ <li><a href="#heronpy.api.spout.base_spout.BaseSpout">BaseSpout</a></li>
+ <li>heronpy.api.component.base_component.BaseComponent</li>
+ <li>__builtin__.object</li>
+ </ul>
+ <h3>Methods</h3>
+
+ <div class="item">
+ <div class="name def" id="heronpy.api.spout.base_spout.BaseSpout.__init__">
+ <p>def <span class="ident">__init__</span>(</p><p>self, delegate)</p>
+ </div>
+
+
+
- <div class="desc"><p>generator.py: API for defining generic sources in python</p></div>
+ <div class="desc"><p>Initializes BaseComponent</p>
+<p>:param delegate: SpoutInstance or BoltInstance</p></div>
+ <div class="source_cont">
+ <p class="source_link"><a href="javascript:void(0);" onclick="toggle('source-heronpy.api.spout.base_spout.BaseSpout.__init__', this);">Show source ≡</a></p>
+ <div id="source-heronpy.api.spout.base_spout.BaseSpout.__init__" class="source">
+ <pre><code>def __init__(self, delegate):
+ """Initializes BaseComponent
+ :param delegate: SpoutInstance or BoltInstance
+ """
+ self.delegate = delegate
+ self.logger = self.delegate.logger
+</code></pre>
+ </div>
+</div>
- </div>
- <div class="item">
- <p class="name"><a href="impl/index.html">heronpy.streamlet.impl</a></p>
-
+ </div>
- <div class="desc"><p>module for heronpy streamlet API impl</p></div>
+
+ <div class="item">
+ <div class="name def" id="heronpy.api.spout.base_spout.BaseSpout.emit">
+ <p>def <span class="ident">emit</span>(</p><p>self, tup, tup_id=None, stream='default', direct_task=None, need_task_ids=False)</p>
+ </div>
+
- </div>
- <div class="item">
- <p class="name"><a href="keyedwindow.m.html">heronpy.streamlet.keyedwindow</a></p>
-
+
- <div class="desc"><p>keyedwindow.py: module for defining KeyedWindow</p></div>
+ <div class="desc"><p>Emits a new tuple from this Spout</p>
+<p>It is compatible with StreamParse API.</p>
+<p>:type tup: list or tuple
+:param tup: the new output Tuple to send from this spout,
+ should contain only serializable data.
+:type tup_id: str or object
+:param tup_id: the ID for the Tuple. Leave this blank for an unreliable emit.
+ (Same as messageId in Java)
+:type stream: str
+:param stream: the ID of the stream this Tuple should be emitted to.
+ Leave empty to emit to the default stream.
+:type direct_task: int
+:param direct_task: the task to send the Tuple to if performing a direct emit.
+:type need_task_ids: bool
+:param need_task_ids: indicate whether or not you would like the task IDs the Tuple was emitted.</p></div>
+ <div class="source_cont">
+ <p class="source_link"><a href="javascript:void(0);" onclick="toggle('source-heronpy.api.spout.base_spout.BaseSpout.emit', this);">Show source ≡</a></p>
+ <div id="source-heronpy.api.spout.base_spout.BaseSpout.emit" class="source">
+ <pre><code>def emit(self, tup, tup_id=None, stream=Stream.DEFAULT_STREAM_ID,
+ direct_task=None, need_task_ids=False):
+ """Emits a new tuple from this Spout
+ It is compatible with StreamParse API.
+ :type tup: list or tuple
+ :param tup: the new output Tuple to send from this spout,
+ should contain only serializable data.
+ :type tup_id: str or object
+ :param tup_id: the ID for the Tuple. Leave this blank for an unreliable emit.
+ (Same as messageId in Java)
+ :type stream: str
+ :param stream: the ID of the stream this Tuple should be emitted to.
+ Leave empty to emit to the default stream.
+ :type direct_task: int
+ :param direct_task: the task to send the Tuple to if performing a direct emit.
+ :type need_task_ids: bool
+ :param need_task_ids: indicate whether or not you would like the task IDs the Tuple was emitted.
+ """
+ return self.delegate.emit(tup, tup_id, stream, direct_task, need_task_ids)
+</code></pre>
+ </div>
+</div>
- </div>
- <div class="item">
- <p class="name"><a href="resources.m.html">heronpy.streamlet.resources</a></p>
-
+ </div>
- <div class="desc"><p>resources.py: module for defining resources</p></div>
+
+ <div class="item">
+ <div class="name def" id="heronpy.api.spout.base_spout.BaseSpout.log">
+ <p>def <span class="ident">log</span>(</p><p>self, message, level=None)</p>
+ </div>
+
- </div>
- <div class="item">
- <p class="name"><a href="transformoperator.m.html">heronpy.streamlet.transformoperator</a></p>
-
+
- <div class="desc"><p>transformoperator.py: API for defining generic transformer in python</p></div>
+ <div class="desc"><p>Log message, optionally providing a logging level</p>
+<p>:type message: str
+:param message: the log message to send
+:type level: str
+:param level: the logging level,
+ one of: trace (=debug), debug, info, warn or error (default: info)</p></div>
+ <div class="source_cont">
+ <p class="source_link"><a href="javascript:void(0);" onclick="toggle('source-heronpy.api.spout.base_spout.BaseSpout.log', this);">Show source ≡</a></p>
+ <div id="source-heronpy.api.spout.base_spout.BaseSpout.log" class="source">
+ <pre><code>def log(self, message, level=None):
+ """Log message, optionally providing a logging level
+ :type message: str
+ :param message: the log message to send
+ :type level: str
+ :param level: the logging level,
+ one of: trace (=debug), debug, info, warn or error (default: info)
+ """
+ self.delegate.log(message, level)
+</code></pre>
+ </div>
+</div>
- </div>
- <div class="item">
- <p class="name"><a href="window.m.html">heronpy.streamlet.window</a></p>
-
+ </div>
- <div class="desc"><p>window.py: module for defining Window</p></div>
+
+ <div class="item">
+ <div class="name def" id="heronpy.api.spout.base_spout.BaseSpout.spec">
+ <p>def <span class="ident">spec</span>(</p><p>cls, name=None, par=1, config=None, optional_outputs=None)</p>
+ </div>
+
- </div>
- <div class="item">
- <p class="name"><a href="windowconfig.m.html">heronpy.streamlet.windowconfig</a></p>
-
+
- <div class="desc"><p>windowconfig.py: module for defining windowconfig</p></div>
+ <div class="desc"><p>Register this spout to the topology and create <code>HeronComponentSpec</code></p>
+<p>The usage of this method is compatible with StreamParse API, although it does not create
+<code>ShellBoltSpec</code> but instead directly registers to a <code>Topology</code> class.</p>
+<p>This method takes an optional <code>outputs</code> argument for supporting dynamic output fields
+declaration. However, it is recommended that <code>outputs</code> should be declared as
+an attribute of your <code>Spout</code> subclass. Also, some ways of declaring inputs is not supported
+in this implementation; please read the documentation below.</p>
+<p>:type name: str
+:param name: Name of this spout.
+:type par: int
+:param par: Parallelism hint for this spout.
+:type config: dict
+:param config: Component-specific config settings.
+:type optional_outputs: list of (str or Stream) or tuple of (str or Stream)
+:param optional_outputs: Additional output fields for this spout. These fields are added to
+ existing <code>outputs</code> class attributes of your spout.
+ This is an optional argument, and exists only for supporting dynamic
+ output field declaration.</p></div>
+ <div class="source_cont">
+ <p class="source_link"><a href="javascript:void(0);" onclick="toggle('source-heronpy.api.spout.base_spout.BaseSpout.spec', this);">Show source ≡</a></p>
+ <div id="source-heronpy.api.spout.base_spout.BaseSpout.spec" class="source">
+ <pre><code>@classmethod
+def spec(cls, name=None, par=1, config=None, optional_outputs=None):
+ """Register this spout to the topology and create ``HeronComponentSpec``
+ The usage of this method is compatible with StreamParse API, although it does not create
+ ``ShellBoltSpec`` but instead directly registers to a ``Topology`` class.
+ This method takes an optional ``outputs`` argument for supporting dynamic output fields
+ declaration. However, it is recommended that ``outputs`` should be declared as
+ an attribute of your ``Spout`` subclass. Also, some ways of declaring inputs is not supported
+ in this implementation; please read the documentation below.
+ :type name: str
+ :param name: Name of this spout.
+ :type par: int
+ :param par: Parallelism hint for this spout.
+ :type config: dict
+ :param config: Component-specific config settings.
+ :type optional_outputs: list of (str or Stream) or tuple of (str or Stream)
+ :param optional_outputs: Additional output fields for this spout. These fields are added to
+ existing ``outputs`` class attributes of your spout.
+ This is an optional argument, and exists only for supporting dynamic
+ output field declaration.
+ """
+ python_class_path = "%s.%s" % (cls.__module__, cls.__name__)
+ if hasattr(cls, 'outputs'):
+ # avoid modification to cls.outputs
+ _outputs = copy.copy(cls.outputs)
+ else:
+ _outputs = []
+ if optional_outputs is not None:
+ assert isinstance(optional_outputs, (list, tuple))
+ for out in optional_outputs:
+ assert isinstance(out, (str, Stream))
+ _outputs.append(out)
+ return HeronComponentSpec(name, python_class_path, is_spout=True, par=par,
+ inputs=None, outputs=_outputs, config=config)
+</code></pre>
+ </div>
+</div>
+ </div>
+
+ </div>
</div>
+
</section>
</article>
diff --git a/content/api/python/api/spout/index.html b/content/api/python/api/spout/index.html
index 333e0fa..f815976 100644
--- a/content/api/python/api/spout/index.html
+++ b/content/api/python/api/spout/index.html
@@ -972,6 +972,12 @@ table {
+ <li class="set"><h3><a href="#header-submodules">Sub-modules</a></h3>
+ <ul>
+ <li class="mono"><a href="base_spout.m.html">heronpy.api.spout.base_spout</a></li>
+ <li class="mono"><a href="spout.m.html">heronpy.api.spout.spout</a></li>
+ </ul>
+ </li>
</ul>
</div>
@@ -1015,6 +1021,21 @@ __import__('pkg_resources').declare_namespace(__name__)
+ <h2 class="section-title" id="header-submodules">Sub-modules</h2>
+ <div class="item">
+ <p class="name"><a href="base_spout.m.html">heronpy.api.spout.base_spout</a></p>
+
+
+ <div class="desc"><p>base_spout.py</p></div>
+
+ </div>
+ <div class="item">
+ <p class="name"><a href="spout.m.html">heronpy.api.spout.spout</a></p>
+
+
+ <div class="desc"><p>bolt.py: API for defining bolt in python</p></div>
+
+ </div>
</section>
</article>
diff --git a/content/api/python/api/spout/spout.m.html b/content/api/python/api/spout/spout.m.html
new file mode 100644
index 0000000..6b44318
--- /dev/null
+++ b/content/api/python/api/spout/spout.m.html
@@ -0,0 +1,1738 @@
+<!doctype html>
+<head>
+ <meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
+ <meta name="viewport" content="width=device-width, initial-scale=1, minimum-scale=1" />
+
+ <title>heronpy.api.spout.spout API documentation</title>
+ <meta name="description" content="bolt.py: API for defining bolt in python" />
+
+ <link href='http://fonts.googleapis.com/css?family=Source+Sans+Pro:400,300' rel='stylesheet' type='text/css'>
+
+ <style type="text/css">
+
+* {
+ box-sizing: border-box;
+}
+/*! normalize.css v1.1.1 | MIT License | git.io/normalize */
+
+/* ==========================================================================
+ HTML5 display definitions
+ ========================================================================== */
+
+/**
+ * Correct `block` display not defined in IE 6/7/8/9 and Firefox 3.
+ */
+
+article,
+aside,
+details,
+figcaption,
+figure,
+footer,
+header,
+hgroup,
+main,
+nav,
+section,
+summary {
+ display: block;
+}
+
+/**
+ * Correct `inline-block` display not defined in IE 6/7/8/9 and Firefox 3.
+ */
+
+audio,
+canvas,
+video {
+ display: inline-block;
+ *display: inline;
+ *zoom: 1;
+}
+
+/**
+ * Prevent modern browsers from displaying `audio` without controls.
+ * Remove excess height in iOS 5 devices.
+ */
+
+audio:not([controls]) {
+ display: none;
+ height: 0;
+}
+
+/**
+ * Address styling not present in IE 7/8/9, Firefox 3, and Safari 4.
+ * Known issue: no IE 6 support.
+ */
+
+[hidden] {
+ display: none;
+}
+
+/* ==========================================================================
+ Base
+ ========================================================================== */
+
+/**
+ * 1. Prevent system color scheme's background color being used in Firefox, IE,
+ * and Opera.
+ * 2. Prevent system color scheme's text color being used in Firefox, IE, and
+ * Opera.
+ * 3. Correct text resizing oddly in IE 6/7 when body `font-size` is set using
+ * `em` units.
+ * 4. Prevent iOS text size adjust after orientation change, without disabling
+ * user zoom.
+ */
+
+html {
+ background: #fff; /* 1 */
+ color: #000; /* 2 */
+ font-size: 100%; /* 3 */
+ -webkit-text-size-adjust: 100%; /* 4 */
+ -ms-text-size-adjust: 100%; /* 4 */
+}
+
+/**
+ * Address `font-family` inconsistency between `textarea` and other form
+ * elements.
+ */
+
+html,
+button,
+input,
+select,
+textarea {
+ font-family: sans-serif;
+}
+
+/**
+ * Address margins handled incorrectly in IE 6/7.
+ */
+
+body {
+ margin: 0;
+}
+
+/* ==========================================================================
+ Links
+ ========================================================================== */
+
+/**
+ * Address `outline` inconsistency between Chrome and other browsers.
+ */
+
+a:focus {
+ outline: thin dotted;
+}
+
+/**
+ * Improve readability when focused and also mouse hovered in all browsers.
+ */
+
+a:active,
+a:hover {
+ outline: 0;
+}
+
+/* ==========================================================================
+ Typography
+ ========================================================================== */
+
+/**
+ * Address font sizes and margins set differently in IE 6/7.
+ * Address font sizes within `section` and `article` in Firefox 4+, Safari 5,
+ * and Chrome.
+ */
+
+h1 {
+ font-size: 2em;
+ margin: 0.67em 0;
+}
+
+h2 {
+ font-size: 1.5em;
+ margin: 0.83em 0;
+}
+
+h3 {
+ font-size: 1.17em;
+ margin: 1em 0;
+}
+
+h4 {
+ font-size: 1em;
+ margin: 1.33em 0;
+}
+
+h5 {
+ font-size: 0.83em;
+ margin: 1.67em 0;
+}
+
+h6 {
+ font-size: 0.67em;
+ margin: 2.33em 0;
+}
+
+/**
+ * Address styling not present in IE 7/8/9, Safari 5, and Chrome.
+ */
+
+abbr[title] {
+ border-bottom: 1px dotted;
+}
+
+/**
+ * Address style set to `bolder` in Firefox 3+, Safari 4/5, and Chrome.
+ */
+
+b,
+strong {
+ font-weight: bold;
+}
+
+blockquote {
+ margin: 1em 40px;
+}
+
+/**
+ * Address styling not present in Safari 5 and Chrome.
+ */
+
+dfn {
+ font-style: italic;
+}
+
+/**
+ * Address differences between Firefox and other browsers.
+ * Known issue: no IE 6/7 normalization.
+ */
+
+hr {
+ -moz-box-sizing: content-box;
+ box-sizing: content-box;
+ height: 0;
+}
+
+/**
+ * Address styling not present in IE 6/7/8/9.
+ */
+
+mark {
+ background: #ff0;
+ color: #000;
+}
+
+/**
+ * Address margins set differently in IE 6/7.
+ */
+
+p,
+pre {
+ margin: 1em 0;
+}
+
+/**
+ * Correct font family set oddly in IE 6, Safari 4/5, and Chrome.
+ */
+
+code,
+kbd,
+pre,
+samp {
+ font-family: monospace, serif;
+ _font-family: 'courier new', monospace;
+ font-size: 1em;
+}
+
+/**
+ * Improve readability of pre-formatted text in all browsers.
+ */
+
+pre {
+ white-space: pre;
+ white-space: pre-wrap;
+ word-wrap: break-word;
+}
+
+/**
+ * Address CSS quotes not supported in IE 6/7.
+ */
+
+q {
+ quotes: none;
+}
+
+/**
+ * Address `quotes` property not supported in Safari 4.
+ */
+
+q:before,
+q:after {
+ content: '';
+ content: none;
+}
+
+/**
+ * Address inconsistent and variable font size in all browsers.
+ */
+
+small {
+ font-size: 80%;
+}
+
+/**
+ * Prevent `sub` and `sup` affecting `line-height` in all browsers.
+ */
+
+sub,
+sup {
+ font-size: 75%;
+ line-height: 0;
+ position: relative;
+ vertical-align: baseline;
+}
+
+sup {
+ top: -0.5em;
+}
+
+sub {
+ bottom: -0.25em;
+}
+
+/* ==========================================================================
+ Lists
+ ========================================================================== */
+
+/**
+ * Address margins set differently in IE 6/7.
+ */
+
+dl,
+menu,
+ol,
+ul {
+ margin: 1em 0;
+}
+
+dd {
+ margin: 0 0 0 40px;
+}
+
+/**
+ * Address paddings set differently in IE 6/7.
+ */
+
+menu,
+ol,
+ul {
+ padding: 0 0 0 40px;
+}
+
+/**
+ * Correct list images handled incorrectly in IE 7.
+ */
+
+nav ul,
+nav ol {
+ list-style: none;
+ list-style-image: none;
+}
+
+/* ==========================================================================
+ Embedded content
+ ========================================================================== */
+
+/**
+ * 1. Remove border when inside `a` element in IE 6/7/8/9 and Firefox 3.
+ * 2. Improve image quality when scaled in IE 7.
+ */
+
+img {
+ border: 0; /* 1 */
+ -ms-interpolation-mode: bicubic; /* 2 */
+}
+
+/**
+ * Correct overflow displayed oddly in IE 9.
+ */
+
+svg:not(:root) {
+ overflow: hidden;
+}
+
+/* ==========================================================================
+ Figures
+ ========================================================================== */
+
+/**
+ * Address margin not present in IE 6/7/8/9, Safari 5, and Opera 11.
+ */
+
+figure {
+ margin: 0;
+}
+
+/* ==========================================================================
+ Forms
+ ========================================================================== */
+
+/**
+ * Correct margin displayed oddly in IE 6/7.
+ */
+
+form {
+ margin: 0;
+}
+
+/**
+ * Define consistent border, margin, and padding.
+ */
+
+fieldset {
+ border: 1px solid #c0c0c0;
+ margin: 0 2px;
+ padding: 0.35em 0.625em 0.75em;
+}
+
+/**
+ * 1. Correct color not being inherited in IE 6/7/8/9.
+ * 2. Correct text not wrapping in Firefox 3.
+ * 3. Correct alignment displayed oddly in IE 6/7.
+ */
+
+legend {
+ border: 0; /* 1 */
+ padding: 0;
+ white-space: normal; /* 2 */
+ *margin-left: -7px; /* 3 */
+}
+
+/**
+ * 1. Correct font size not being inherited in all browsers.
+ * 2. Address margins set differently in IE 6/7, Firefox 3+, Safari 5,
+ * and Chrome.
+ * 3. Improve appearance and consistency in all browsers.
+ */
+
+button,
+input,
+select,
+textarea {
+ font-size: 100%; /* 1 */
+ margin: 0; /* 2 */
+ vertical-align: baseline; /* 3 */
+ *vertical-align: middle; /* 3 */
+}
+
+/**
+ * Address Firefox 3+ setting `line-height` on `input` using `!important` in
+ * the UA stylesheet.
+ */
+
+button,
+input {
+ line-height: normal;
+}
+
+/**
+ * Address inconsistent `text-transform` inheritance for `button` and `select`.
+ * All other form control elements do not inherit `text-transform` values.
+ * Correct `button` style inheritance in Chrome, Safari 5+, and IE 6+.
+ * Correct `select` style inheritance in Firefox 4+ and Opera.
+ */
+
+button,
+select {
+ text-transform: none;
+}
+
+/**
+ * 1. Avoid the WebKit bug in Android 4.0.* where (2) destroys native `audio`
+ * and `video` controls.
+ * 2. Correct inability to style clickable `input` types in iOS.
+ * 3. Improve usability and consistency of cursor style between image-type
+ * `input` and others.
+ * 4. Remove inner spacing in IE 7 without affecting normal text inputs.
+ * Known issue: inner spacing remains in IE 6.
+ */
+
+button,
+html input[type="button"], /* 1 */
+input[type="reset"],
+input[type="submit"] {
+ -webkit-appearance: button; /* 2 */
+ cursor: pointer; /* 3 */
+ *overflow: visible; /* 4 */
+}
+
+/**
+ * Re-set default cursor for disabled elements.
+ */
+
+button[disabled],
+html input[disabled] {
+ cursor: default;
+}
+
+/**
+ * 1. Address box sizing set to content-box in IE 8/9.
+ * 2. Remove excess padding in IE 8/9.
+ * 3. Remove excess padding in IE 7.
+ * Known issue: excess padding remains in IE 6.
+ */
+
+input[type="checkbox"],
+input[type="radio"] {
+ box-sizing: border-box; /* 1 */
+ padding: 0; /* 2 */
+ *height: 13px; /* 3 */
+ *width: 13px; /* 3 */
+}
+
+/**
+ * 1. Address `appearance` set to `searchfield` in Safari 5 and Chrome.
+ * 2. Address `box-sizing` set to `border-box` in Safari 5 and Chrome
+ * (include `-moz` to future-proof).
+ */
+
+input[type="search"] {
+ -webkit-appearance: textfield; /* 1 */
+ -moz-box-sizing: content-box;
+ -webkit-box-sizing: content-box; /* 2 */
+ box-sizing: content-box;
+}
+
+/**
+ * Remove inner padding and search cancel button in Safari 5 and Chrome
+ * on OS X.
+ */
+
+input[type="search"]::-webkit-search-cancel-button,
+input[type="search"]::-webkit-search-decoration {
+ -webkit-appearance: none;
+}
+
+/**
+ * Remove inner padding and border in Firefox 3+.
+ */
+
+button::-moz-focus-inner,
+input::-moz-focus-inner {
+ border: 0;
+ padding: 0;
+}
+
+/**
+ * 1. Remove default vertical scrollbar in IE 6/7/8/9.
+ * 2. Improve readability and alignment in all browsers.
+ */
+
+textarea {
+ overflow: auto; /* 1 */
+ vertical-align: top; /* 2 */
+}
+
+/* ==========================================================================
+ Tables
+ ========================================================================== */
+
+/**
+ * Remove most spacing between table cells.
+ */
+
+table {
+ border-collapse: collapse;
+ border-spacing: 0;
+}
+
+ </style>
+
+ <style type="text/css">
+
+ html, body {
+ margin: 0;
+ padding: 0;
+ min-height: 100%;
+ }
+ body {
+ background: #fff;
+ font-family: "Source Sans Pro", "Helvetica Neueue", Helvetica, sans;
+ font-weight: 300;
+ font-size: 16px;
+ line-height: 1.6em;
+ }
+ #content {
+ width: 70%;
+ max-width: 850px;
+ float: left;
+ padding: 30px 60px;
+ border-left: 1px solid #ddd;
+ }
+ #sidebar {
+ width: 25%;
+ float: left;
+ padding: 30px;
+ overflow: hidden;
+ }
+ #nav {
+ font-size: 130%;
+ margin: 0 0 15px 0;
+ }
+
+ #top {
+ display: block;
+ position: fixed;
+ bottom: 5px;
+ left: 5px;
+ font-size: .85em;
+ text-transform: uppercase;
+ }
+
+ #footer {
+ font-size: .75em;
+ padding: 5px 30px;
+ border-top: 1px solid #ddd;
+ text-align: right;
+ }
+ #footer p {
+ margin: 0 0 0 30px;
+ display: inline-block;
+ }
+
+ h1, h2, h3, h4, h5 {
+ font-weight: 300;
+ }
+ h1 {
+ font-size: 2.5em;
+ line-height: 1.1em;
+ margin: 0 0 .50em 0;
+ }
+
+ h2 {
+ font-size: 1.75em;
+ margin: 1em 0 .50em 0;
+ }
+
+ h3 {
+ margin: 25px 0 10px 0;
+ }
+
+ h4 {
+ margin: 0;
+ font-size: 105%;
+ }
+
+ a {
+ color: #058;
+ text-decoration: none;
+ transition: color .3s ease-in-out;
+ }
+
+ a:hover {
+ color: #e08524;
+ transition: color .3s ease-in-out;
+ }
+
+ pre, code, .mono, .name {
+ font-family: "Ubuntu Mono", "Cousine", "DejaVu Sans Mono", monospace;
+ }
+
+ .title .name {
+ font-weight: bold;
+ }
+ .section-title {
+ margin-top: 2em;
+ }
+ .ident {
+ color: #900;
+ }
+
+ code {
+ background: #f9f9f9;
+ }
+
+ pre {
+ background: #fefefe;
+ border: 1px solid #ddd;
+ box-shadow: 2px 2px 0 #f3f3f3;
+ margin: 0 30px;
+ padding: 15px 30px;
+ }
+
+ .codehilite {
+ margin: 0 30px 10px 30px;
+ }
+
+ .codehilite pre {
+ margin: 0;
+ }
+ .codehilite .err { background: #ff3300; color: #fff !important; }
+
+ table#module-list {
+ font-size: 110%;
+ }
+
+ table#module-list tr td:first-child {
+ padding-right: 10px;
+ white-space: nowrap;
+ }
+
+ table#module-list td {
+ vertical-align: top;
+ padding-bottom: 8px;
+ }
+
+ table#module-list td p {
+ margin: 0 0 7px 0;
+ }
+
+ .def {
+ display: table;
+ }
+
+ .def p {
+ display: table-cell;
+ vertical-align: top;
+ text-align: left;
+ }
+
+ .def p:first-child {
+ white-space: nowrap;
+ }
+
+ .def p:last-child {
+ width: 100%;
+ }
+
+
+ #index {
+ list-style-type: none;
+ margin: 0;
+ padding: 0;
+ }
+ ul#index .class_name {
+ /* font-size: 110%; */
+ font-weight: bold;
+ }
+ #index ul {
+ margin: 0;
+ }
+
+ .item {
+ margin: 0 0 15px 0;
+ }
+
+ .item .class {
+ margin: 0 0 25px 30px;
+ }
+
+ .item .class ul.class_list {
+ margin: 0 0 20px 0;
+ }
+
+ .item .name {
+ background: #fafafa;
+ margin: 0;
+ font-weight: bold;
+ padding: 5px 10px;
+ border-radius: 3px;
+ display: inline-block;
+ min-width: 40%;
+ }
+ .item .name:hover {
+ background: #f6f6f6;
+ }
+
+ .item .empty_desc {
+ margin: 0 0 5px 0;
+ padding: 0;
+ }
+
+ .item .inheritance {
+ margin: 3px 0 0 30px;
+ }
+
+ .item .inherited {
+ color: #666;
+ }
+
+ .item .desc {
+ padding: 0 8px;
+ margin: 0;
+ }
+
+ .item .desc p {
+ margin: 0 0 10px 0;
+ }
+
+ .source_cont {
+ margin: 0;
+ padding: 0;
+ }
+
+ .source_link a {
+ background: #ffc300;
+ font-weight: 400;
+ font-size: .75em;
+ text-transform: uppercase;
+ color: #fff;
+ text-shadow: 1px 1px 0 #f4b700;
+
+ padding: 3px 8px;
+ border-radius: 2px;
+ transition: background .3s ease-in-out;
+ }
+ .source_link a:hover {
+ background: #FF7200;
+ text-shadow: none;
+ transition: background .3s ease-in-out;
+ }
+
+ .source {
+ display: none;
+ max-height: 600px;
+ overflow-y: scroll;
+ margin-bottom: 15px;
+ }
+
+ .source .codehilite {
+ margin: 0;
+ }
+
+ .desc h1, .desc h2, .desc h3 {
+ font-size: 100% !important;
+ }
+ .clear {
+ clear: both;
+ }
+
+ @media all and (max-width: 950px) {
+ #sidebar {
+ width: 35%;
+ }
+ #content {
+ width: 65%;
+ }
+ }
+ @media all and (max-width: 650px) {
+ #top {
+ display: none;
+ }
+ #sidebar {
+ float: none;
+ width: auto;
+ }
+ #content {
+ float: none;
+ width: auto;
+ padding: 30px;
+ }
+
+ #index ul {
+ padding: 0;
+ margin-bottom: 15px;
+ }
+ #index ul li {
+ display: inline-block;
+ margin-right: 30px;
+ }
+ #footer {
+ text-align: left;
+ }
+ #footer p {
+ display: block;
+ margin: inherit;
+ }
+ }
+
+ /*****************************/
+
+ </style>
+
+
+ <style type="text/css">
+
+/* ==========================================================================
+ EXAMPLE Media Queries for Responsive Design.
+ These examples override the primary ('mobile first') styles.
+ Modify as content requires.
+ ========================================================================== */
+
+@media only screen and (min-width: 35em) {
+ /* Style adjustments for viewports that meet the condition */
+}
+
+@media print,
+ (-o-min-device-pixel-ratio: 5/4),
+ (-webkit-min-device-pixel-ratio: 1.25),
+ (min-resolution: 120dpi) {
+ /* Style adjustments for high resolution devices */
+}
+
+/* ==========================================================================
+ Print styles.
+ Inlined to avoid required HTTP connection: h5bp.com/r
+ ========================================================================== */
+
+@media print {
+ * {
+ background: transparent !important;
+ color: #000 !important; /* Black prints faster: h5bp.com/s */
+ box-shadow: none !important;
+ text-shadow: none !important;
+ }
+
+ a,
+ a:visited {
+ text-decoration: underline;
+ }
+
+ a[href]:after {
+ content: " (" attr(href) ")";
+ }
+
+ abbr[title]:after {
+ content: " (" attr(title) ")";
+ }
+
+ /*
+ * Don't show links for images, or javascript/internal links
+ */
+
+ .ir a:after,
+ a[href^="javascript:"]:after,
+ a[href^="#"]:after {
+ content: "";
+ }
+
+ pre,
+ blockquote {
+ border: 1px solid #999;
+ page-break-inside: avoid;
+ }
+
+ thead {
+ display: table-header-group; /* h5bp.com/t */
+ }
+
+ tr,
+ img {
+ page-break-inside: avoid;
+ }
+
+ img {
+ max-width: 100% !important;
+ }
+
+ @page {
+ margin: 0.5cm;
+ }
+
+ p,
+ h2,
+ h3 {
+ orphans: 3;
+ widows: 3;
+ }
+
+ h2,
+ h3 {
+ page-break-after: avoid;
+ }
+}
+
+ </style>
+
+ <script type="text/javascript">
+ function toggle(id, $link) {
+ $node = document.getElementById(id);
+ if (!$node)
+ return;
+ if (!$node.style.display || $node.style.display == 'none') {
+ $node.style.display = 'block';
+ $link.innerHTML = 'Hide source ≢';
+ } else {
+ $node.style.display = 'none';
+ $link.innerHTML = 'Show source ≡';
+ }
+ }
+ </script>
+</head>
+<body>
+<a href="#" id="top">Top</a>
+
+<div id="container">
+
+
+ <div id="sidebar">
+ <h1>Index</h1>
+ <ul id="index">
+
+
+ <li class="set"><h3><a href="#header-classes">Classes</a></h3>
+ <ul>
+ <li class="mono">
+ <span class="class_name"><a href="#heronpy.api.spout.spout.Spout">Spout</a></span>
+
+
+ <ul>
+ <li class="mono"><a href="#heronpy.api.spout.spout.Spout.__init__">__init__</a></li>
+ <li class="mono"><a href="#heronpy.api.spout.spout.Spout.ack">ack</a></li>
+ <li class="mono"><a href="#heronpy.api.spout.spout.Spout.activate">activate</a></li>
+ <li class="mono"><a href="#heronpy.api.spout.spout.Spout.close">close</a></li>
+ <li class="mono"><a href="#heronpy.api.spout.spout.Spout.deactivate">deactivate</a></li>
+ <li class="mono"><a href="#heronpy.api.spout.spout.Spout.emit">emit</a></li>
+ <li class="mono"><a href="#heronpy.api.spout.spout.Spout.fail">fail</a></li>
+ <li class="mono"><a href="#heronpy.api.spout.spout.Spout.initialize">initialize</a></li>
+ <li class="mono"><a href="#heronpy.api.spout.spout.Spout.log">log</a></li>
+ <li class="mono"><a href="#heronpy.api.spout.spout.Spout.next_tuple">next_tuple</a></li>
+ <li class="mono"><a href="#heronpy.api.spout.spout.Spout.spec">spec</a></li>
+ </ul>
+
+ </li>
+ </ul>
+ </li>
+
+ </ul>
+ </div>
+
+ <article id="content">
+
+
+
+
+
+
+ <header id="section-intro">
+ <h1 class="title"><span class="name">heronpy.api.spout.spout</span> module</h1>
+ <p>bolt.py: API for defining bolt in python</p>
+
+ <p class="source_link"><a href="javascript:void(0);" onclick="toggle('source-heronpy.api.spout.spout', this);">Show source ≡</a></p>
+ <div id="source-heronpy.api.spout.spout" class="source">
+ <pre><code>#!/usr/bin/env python
+# -*- encoding: utf-8 -*-
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+'''bolt.py: API for defining bolt in python'''
+from abc import abstractmethod
+from heronpy.api.spout.base_spout import BaseSpout
+
+class Spout(BaseSpout):
+ """API for defining a spout for Heron in Python
+
+ Topology writers need to inherit this ``Spout`` class to define their own custom spout, by
+ implementing ``initialize()``, ``next_tuple()``, ``ack()`` and ``fail()`` methods.
+ In addition, ``close()``, ``activate()`` and ``deactivate()`` are available to be implemented.
+ """
+
+ @abstractmethod
+ def initialize(self, config, context):
+ """Called when a task for this component is initialized within a worker on the cluster
+
+ It provides the spout with the environment in which the spout executes. Note that
+ you should NOT override ``__init__()`` for initialization of your spout, as it is
+ used internally by Heron Instance; instead, you should use this method to initialize
+ any custom instance variables or connections to data sources.
+
+ *Should be implemented by a subclass.*
+
+ :type config: dict
+ :param config: The Heron configuration for this bolt. This is the configuration provided to
+ the topology merged in with cluster configuration on this machine.
+ Note that types of string values in the config have been automatically converted,
+ meaning that number strings and boolean strings are converted to appropriate
+ types.
+ :type context: :class:`TopologyContext`
+ :param context: This object can be used to get information about this task's place within the
+ topology, including the task id and component id of this task, input and output
+ information, etc.
+ """
+ pass
+
+ @abstractmethod
+ def close(self):
+ """Called when this spout is going to be shutdown
+
+ There is no guarantee that close() will be called.
+ """
+ pass
+
+ @abstractmethod
+ def next_tuple(self):
+ """When this method is called, Heron is requesting that the Spout emit tuples
+
+ It is compatible with StreamParse API.
+
+ This method should be non-blocking, so if the Spout has no tuples to emit,
+ this method should return; next_tuple(), ack(), and fail() are all called in a tight
+ loop in a single thread in the spout task. WHen there are no tuples to emit, it is
+ courteous to have next_tuple sleep for a short amount of time (like a single millisecond)
+ so as not to waste too much CPU.
+
+ **Must be implemented by a subclass, otherwise NotImplementedError is raised.**
+ """
+ raise NotImplementedError("Spout not implementing next_tuple() method")
+
+ @abstractmethod
+ def ack(self, tup_id):
+ """Determine that the tuple emitted by this spout with the tup_id has been fully processed
+
+ It is compatible with StreamParse API.
+
+ Heron has determined that the tuple emitted by this spout with the tup_id identifier
+ has been fully processed. Typically, an implementation of this method will take that
+ message off the queue and prevent it from being replayed.
+
+ *Should be implemented by a subclass.*
+
+ :param tup_id: the ID of the HeronTuple that has been fully acknowledged.
+ """
+ pass
+
+ @abstractmethod
+ def fail(self, tup_id):
+ """Determine that the tuple emitted by this spout with the tup_id has failed to be processed
+
+ It is compatible with StreamParse API.
+
+ The tuple emitted by this spout with the tup_id identifier has failed to be
+ fully processed. Typically, an implementation of this method will put that
+ message back on the queue to be replayed at a later time.
+
+ *Should be implemented by a subclass.*
+
+ :param tup_id: the ID of the HeronTuple that has failed either due to a bolt calling ``fail()``
+ or timeout
+ """
+ pass
+
+ @abstractmethod
+ def activate(self):
+ """Called when a spout has been activated out of a deactivated mode
+
+ next_tuple() will be called on this spout soon. A spout can become activated
+ after having been deactivated when the topology is manipulated using the
+ `heron` client.
+ """
+ pass
+
+ @abstractmethod
+ def deactivate(self):
+ """Called when a spout has been deactivated
+
+ next_tuple() will not be called while a spout is deactivated.
+ The spout may or may not be reactivated in the future.
+ """
+ pass
+</code></pre>
+ </div>
+
+ </header>
+
+ <section id="section-items">
+
+
+ <h2 class="section-title" id="header-classes">Classes</h2>
+
+ <div class="item">
+ <p id="heronpy.api.spout.spout.Spout" class="name">class <span class="ident">Spout</span></p>
+
+
+ <div class="desc"><p>API for defining a spout for Heron in Python</p>
+<p>Topology writers need to inherit this <code>Spout</code> class to define their own custom spout, by
+implementing <code>initialize()</code>, <code>next_tuple()</code>, <code>ack()</code> and <code>fail()</code> methods.
+In addition, <code>close()</code>, <code>activate()</code> and <code>deactivate()</code> are available to be implemented.</p></div>
+ <div class="source_cont">
+ <p class="source_link"><a href="javascript:void(0);" onclick="toggle('source-heronpy.api.spout.spout.Spout', this);">Show source ≡</a></p>
+ <div id="source-heronpy.api.spout.spout.Spout" class="source">
+ <pre><code>class Spout(BaseSpout):
+ """API for defining a spout for Heron in Python
+
+ Topology writers need to inherit this ``Spout`` class to define their own custom spout, by
+ implementing ``initialize()``, ``next_tuple()``, ``ack()`` and ``fail()`` methods.
+ In addition, ``close()``, ``activate()`` and ``deactivate()`` are available to be implemented.
+ """
+
+ @abstractmethod
+ def initialize(self, config, context):
+ """Called when a task for this component is initialized within a worker on the cluster
+
+ It provides the spout with the environment in which the spout executes. Note that
+ you should NOT override ``__init__()`` for initialization of your spout, as it is
+ used internally by Heron Instance; instead, you should use this method to initialize
+ any custom instance variables or connections to data sources.
+
+ *Should be implemented by a subclass.*
+
+ :type config: dict
+ :param config: The Heron configuration for this bolt. This is the configuration provided to
+ the topology merged in with cluster configuration on this machine.
+ Note that types of string values in the config have been automatically converted,
+ meaning that number strings and boolean strings are converted to appropriate
+ types.
+ :type context: :class:`TopologyContext`
+ :param context: This object can be used to get information about this task's place within the
+ topology, including the task id and component id of this task, input and output
+ information, etc.
+ """
+ pass
+
+ @abstractmethod
+ def close(self):
+ """Called when this spout is going to be shutdown
+
+ There is no guarantee that close() will be called.
+ """
+ pass
+
+ @abstractmethod
+ def next_tuple(self):
+ """When this method is called, Heron is requesting that the Spout emit tuples
+
+ It is compatible with StreamParse API.
+
+ This method should be non-blocking, so if the Spout has no tuples to emit,
+ this method should return; next_tuple(), ack(), and fail() are all called in a tight
+ loop in a single thread in the spout task. WHen there are no tuples to emit, it is
+ courteous to have next_tuple sleep for a short amount of time (like a single millisecond)
+ so as not to waste too much CPU.
+
+ **Must be implemented by a subclass, otherwise NotImplementedError is raised.**
+ """
+ raise NotImplementedError("Spout not implementing next_tuple() method")
+
+ @abstractmethod
+ def ack(self, tup_id):
+ """Determine that the tuple emitted by this spout with the tup_id has been fully processed
+
+ It is compatible with StreamParse API.
+
+ Heron has determined that the tuple emitted by this spout with the tup_id identifier
+ has been fully processed. Typically, an implementation of this method will take that
+ message off the queue and prevent it from being replayed.
+
+ *Should be implemented by a subclass.*
+
+ :param tup_id: the ID of the HeronTuple that has been fully acknowledged.
+ """
+ pass
+
+ @abstractmethod
+ def fail(self, tup_id):
+ """Determine that the tuple emitted by this spout with the tup_id has failed to be processed
+
+ It is compatible with StreamParse API.
+
+ The tuple emitted by this spout with the tup_id identifier has failed to be
+ fully processed. Typically, an implementation of this method will put that
+ message back on the queue to be replayed at a later time.
+
+ *Should be implemented by a subclass.*
+
+ :param tup_id: the ID of the HeronTuple that has failed either due to a bolt calling ``fail()``
+ or timeout
+ """
+ pass
+
+ @abstractmethod
+ def activate(self):
+ """Called when a spout has been activated out of a deactivated mode
+
+ next_tuple() will be called on this spout soon. A spout can become activated
+ after having been deactivated when the topology is manipulated using the
+ `heron` client.
+ """
+ pass
+
+ @abstractmethod
+ def deactivate(self):
+ """Called when a spout has been deactivated
+
+ next_tuple() will not be called while a spout is deactivated.
+ The spout may or may not be reactivated in the future.
+ """
+ pass
+</code></pre>
+ </div>
+</div>
+
+
+ <div class="class">
+ <h3>Ancestors (in MRO)</h3>
+ <ul class="class_list">
+ <li><a href="#heronpy.api.spout.spout.Spout">Spout</a></li>
+ <li>heronpy.api.spout.base_spout.BaseSpout</li>
+ <li>heronpy.api.component.base_component.BaseComponent</li>
+ <li>__builtin__.object</li>
+ </ul>
+ <h3>Methods</h3>
+
+ <div class="item">
+ <div class="name def" id="heronpy.api.spout.spout.Spout.__init__">
+ <p>def <span class="ident">__init__</span>(</p><p>self, delegate)</p>
+ </div>
+
+
+
+
+ <div class="desc"><p>Initializes BaseComponent</p>
+<p>:param delegate: SpoutInstance or BoltInstance</p></div>
+ <div class="source_cont">
+ <p class="source_link"><a href="javascript:void(0);" onclick="toggle('source-heronpy.api.spout.spout.Spout.__init__', this);">Show source ≡</a></p>
+ <div id="source-heronpy.api.spout.spout.Spout.__init__" class="source">
+ <pre><code>def __init__(self, delegate):
+ """Initializes BaseComponent
+ :param delegate: SpoutInstance or BoltInstance
+ """
+ self.delegate = delegate
+ self.logger = self.delegate.logger
+</code></pre>
+ </div>
+</div>
+
+ </div>
+
+
+ <div class="item">
+ <div class="name def" id="heronpy.api.spout.spout.Spout.ack">
+ <p>def <span class="ident">ack</span>(</p><p>self, tup_id)</p>
+ </div>
+
+
+
+
+ <div class="desc"><p>Determine that the tuple emitted by this spout with the tup_id has been fully processed</p>
+<p>It is compatible with StreamParse API.</p>
+<p>Heron has determined that the tuple emitted by this spout with the tup_id identifier
+has been fully processed. Typically, an implementation of this method will take that
+message off the queue and prevent it from being replayed.</p>
+<p><em>Should be implemented by a subclass.</em></p>
+<p>:param tup_id: the ID of the HeronTuple that has been fully acknowledged.</p></div>
+ <div class="source_cont">
+ <p class="source_link"><a href="javascript:void(0);" onclick="toggle('source-heronpy.api.spout.spout.Spout.ack', this);">Show source ≡</a></p>
+ <div id="source-heronpy.api.spout.spout.Spout.ack" class="source">
+ <pre><code>@abstractmethod
+def ack(self, tup_id):
+ """Determine that the tuple emitted by this spout with the tup_id has been fully processed
+ It is compatible with StreamParse API.
+ Heron has determined that the tuple emitted by this spout with the tup_id identifier
+ has been fully processed. Typically, an implementation of this method will take that
+ message off the queue and prevent it from being replayed.
+ *Should be implemented by a subclass.*
+ :param tup_id: the ID of the HeronTuple that has been fully acknowledged.
+ """
+ pass
+</code></pre>
+ </div>
+</div>
+
+ </div>
+
+
+ <div class="item">
+ <div class="name def" id="heronpy.api.spout.spout.Spout.activate">
+ <p>def <span class="ident">activate</span>(</p><p>self)</p>
+ </div>
+
+
+
+
+ <div class="desc"><p>Called when a spout has been activated out of a deactivated mode</p>
+<p>next_tuple() will be called on this spout soon. A spout can become activated
+after having been deactivated when the topology is manipulated using the
+<code>heron</code> client.</p></div>
+ <div class="source_cont">
+ <p class="source_link"><a href="javascript:void(0);" onclick="toggle('source-heronpy.api.spout.spout.Spout.activate', this);">Show source ≡</a></p>
+ <div id="source-heronpy.api.spout.spout.Spout.activate" class="source">
+ <pre><code>@abstractmethod
+def activate(self):
+ """Called when a spout has been activated out of a deactivated mode
+ next_tuple() will be called on this spout soon. A spout can become activated
+ after having been deactivated when the topology is manipulated using the
+ `heron` client.
+ """
+ pass
+</code></pre>
+ </div>
+</div>
+
+ </div>
+
+
+ <div class="item">
+ <div class="name def" id="heronpy.api.spout.spout.Spout.close">
+ <p>def <span class="ident">close</span>(</p><p>self)</p>
+ </div>
+
+
+
+
+ <div class="desc"><p>Called when this spout is going to be shutdown</p>
+<p>There is no guarantee that close() will be called.</p></div>
+ <div class="source_cont">
+ <p class="source_link"><a href="javascript:void(0);" onclick="toggle('source-heronpy.api.spout.spout.Spout.close', this);">Show source ≡</a></p>
+ <div id="source-heronpy.api.spout.spout.Spout.close" class="source">
+ <pre><code>@abstractmethod
+def close(self):
+ """Called when this spout is going to be shutdown
+ There is no guarantee that close() will be called.
+ """
+ pass
+</code></pre>
+ </div>
+</div>
+
+ </div>
+
+
+ <div class="item">
+ <div class="name def" id="heronpy.api.spout.spout.Spout.deactivate">
+ <p>def <span class="ident">deactivate</span>(</p><p>self)</p>
+ </div>
+
+
+
+
+ <div class="desc"><p>Called when a spout has been deactivated</p>
+<p>next_tuple() will not be called while a spout is deactivated.
+The spout may or may not be reactivated in the future.</p></div>
+ <div class="source_cont">
+ <p class="source_link"><a href="javascript:void(0);" onclick="toggle('source-heronpy.api.spout.spout.Spout.deactivate', this);">Show source ≡</a></p>
+ <div id="source-heronpy.api.spout.spout.Spout.deactivate" class="source">
+ <pre><code>@abstractmethod
+def deactivate(self):
+ """Called when a spout has been deactivated
+ next_tuple() will not be called while a spout is deactivated.
+ The spout may or may not be reactivated in the future.
+ """
+ pass
+</code></pre>
+ </div>
+</div>
+
+ </div>
+
+
+ <div class="item">
+ <div class="name def" id="heronpy.api.spout.spout.Spout.emit">
+ <p>def <span class="ident">emit</span>(</p><p>self, tup, tup_id=None, stream='default', direct_task=None, need_task_ids=False)</p>
+ </div>
+
+
+
+
+ <div class="desc"><p>Emits a new tuple from this Spout</p>
+<p>It is compatible with StreamParse API.</p>
+<p>:type tup: list or tuple
+:param tup: the new output Tuple to send from this spout,
+ should contain only serializable data.
+:type tup_id: str or object
+:param tup_id: the ID for the Tuple. Leave this blank for an unreliable emit.
+ (Same as messageId in Java)
+:type stream: str
+:param stream: the ID of the stream this Tuple should be emitted to.
+ Leave empty to emit to the default stream.
+:type direct_task: int
+:param direct_task: the task to send the Tuple to if performing a direct emit.
+:type need_task_ids: bool
+:param need_task_ids: indicate whether or not you would like the task IDs the Tuple was emitted.</p></div>
+ <div class="source_cont">
+ <p class="source_link"><a href="javascript:void(0);" onclick="toggle('source-heronpy.api.spout.spout.Spout.emit', this);">Show source ≡</a></p>
+ <div id="source-heronpy.api.spout.spout.Spout.emit" class="source">
+ <pre><code>def emit(self, tup, tup_id=None, stream=Stream.DEFAULT_STREAM_ID,
+ direct_task=None, need_task_ids=False):
+ """Emits a new tuple from this Spout
+ It is compatible with StreamParse API.
+ :type tup: list or tuple
+ :param tup: the new output Tuple to send from this spout,
+ should contain only serializable data.
+ :type tup_id: str or object
+ :param tup_id: the ID for the Tuple. Leave this blank for an unreliable emit.
+ (Same as messageId in Java)
+ :type stream: str
+ :param stream: the ID of the stream this Tuple should be emitted to.
+ Leave empty to emit to the default stream.
+ :type direct_task: int
+ :param direct_task: the task to send the Tuple to if performing a direct emit.
+ :type need_task_ids: bool
+ :param need_task_ids: indicate whether or not you would like the task IDs the Tuple was emitted.
+ """
+ return self.delegate.emit(tup, tup_id, stream, direct_task, need_task_ids)
+</code></pre>
+ </div>
+</div>
+
+ </div>
+
+
+ <div class="item">
+ <div class="name def" id="heronpy.api.spout.spout.Spout.fail">
+ <p>def <span class="ident">fail</span>(</p><p>self, tup_id)</p>
+ </div>
+
+
+
+
+ <div class="desc"><p>Determine that the tuple emitted by this spout with the tup_id has failed to be processed</p>
+<p>It is compatible with StreamParse API.</p>
+<p>The tuple emitted by this spout with the tup_id identifier has failed to be
+fully processed. Typically, an implementation of this method will put that
+message back on the queue to be replayed at a later time.</p>
+<p><em>Should be implemented by a subclass.</em></p>
+<p>:param tup_id: the ID of the HeronTuple that has failed either due to a bolt calling <code>fail()</code>
+ or timeout</p></div>
+ <div class="source_cont">
+ <p class="source_link"><a href="javascript:void(0);" onclick="toggle('source-heronpy.api.spout.spout.Spout.fail', this);">Show source ≡</a></p>
+ <div id="source-heronpy.api.spout.spout.Spout.fail" class="source">
+ <pre><code>@abstractmethod
+def fail(self, tup_id):
+ """Determine that the tuple emitted by this spout with the tup_id has failed to be processed
+ It is compatible with StreamParse API.
+ The tuple emitted by this spout with the tup_id identifier has failed to be
+ fully processed. Typically, an implementation of this method will put that
+ message back on the queue to be replayed at a later time.
+ *Should be implemented by a subclass.*
+ :param tup_id: the ID of the HeronTuple that has failed either due to a bolt calling ``fail()``
+ or timeout
+ """
+ pass
+</code></pre>
+ </div>
+</div>
+
+ </div>
+
+
+ <div class="item">
+ <div class="name def" id="heronpy.api.spout.spout.Spout.initialize">
+ <p>def <span class="ident">initialize</span>(</p><p>self, config, context)</p>
+ </div>
+
+
+
+
+ <div class="desc"><p>Called when a task for this component is initialized within a worker on the cluster</p>
+<p>It provides the spout with the environment in which the spout executes. Note that
+you should NOT override <code>__init__()</code> for initialization of your spout, as it is
+used internally by Heron Instance; instead, you should use this method to initialize
+any custom instance variables or connections to data sources.</p>
+<p><em>Should be implemented by a subclass.</em></p>
+<p>:type config: dict
+:param config: The Heron configuration for this bolt. This is the configuration provided to
+ the topology merged in with cluster configuration on this machine.
+ Note that types of string values in the config have been automatically converted,
+ meaning that number strings and boolean strings are converted to appropriate
+ types.
+:type context: :class:<code>TopologyContext</code>
+:param context: This object can be used to get information about this task's place within the
+ topology, including the task id and component id of this task, input and output
+ information, etc.</p></div>
+ <div class="source_cont">
+ <p class="source_link"><a href="javascript:void(0);" onclick="toggle('source-heronpy.api.spout.spout.Spout.initialize', this);">Show source ≡</a></p>
+ <div id="source-heronpy.api.spout.spout.Spout.initialize" class="source">
+ <pre><code>@abstractmethod
+def initialize(self, config, context):
+ """Called when a task for this component is initialized within a worker on the cluster
... 60590 lines suppressed ...