You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@beam.apache.org by gi...@apache.org on 2019/10/28 10:11:30 UTC
[beam] branch asf-site updated: Publishing website 2019/10/28
10:11:16 at commit 98eb81d
This is an automated email from the ASF dual-hosted git repository.
git-site-role pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/asf-site by this push:
new a0db495 Publishing website 2019/10/28 10:11:16 at commit 98eb81d
a0db495 is described below
commit a0db495e8b70607c4bfea8c2083950d14a1179cf
Author: jenkins <bu...@apache.org>
AuthorDate: Mon Oct 28 10:11:17 2019 +0000
Publishing website 2019/10/28 10:11:16 at commit 98eb81d
---
.../io/developing-io-overview/index.html | 3 +-
.../documentation/programming-guide/index.html | 3 +-
.../images/dofn-sequence-diagram.png | Bin 126448 -> 0 bytes
.../images/dofn-sequence-diagram.svg | 94 ++++++++++++++++++
.../images/source-sequence-diagram.png | Bin 73657 -> 0 bytes
.../images/source-sequence-diagram.svg | 106 +++++++++++++++++++++
6 files changed, 204 insertions(+), 2 deletions(-)
diff --git a/website/generated-content/documentation/io/developing-io-overview/index.html b/website/generated-content/documentation/io/developing-io-overview/index.html
index 858b1e5..f2ad792 100644
--- a/website/generated-content/documentation/io/developing-io-overview/index.html
+++ b/website/generated-content/documentation/io/developing-io-overview/index.html
@@ -590,7 +590,8 @@ read operations in sorted key order.</p>
information to IO developers such as the constraints that
apply to the objects or particular cases such as streaming mode.</p>
-<p><img src="/images/source-sequence-diagram.png" alt="This is a sequence diagram that shows the lifecycle of the Source" /></p>
+<p><!-- The source for the sequence diagram can be found in the the SVG resource. -->
+<img src="/images/source-sequence-diagram.svg" alt="This is a sequence diagram that shows the lifecycle of the Source" /></p>
<h3 id="using-pardo-and-groupbykey">Using ParDo and GroupByKey</h3>
diff --git a/website/generated-content/documentation/programming-guide/index.html b/website/generated-content/documentation/programming-guide/index.html
index c18446f..350df21 100644
--- a/website/generated-content/documentation/programming-guide/index.html
+++ b/website/generated-content/documentation/programming-guide/index.html
@@ -1374,7 +1374,8 @@ transforms, including <code class="highlighter-rouge">Filter</code>, <code class
apply to the objects or particular cases such as failover or
instance reuse. They also give instanciation use cases.</p>
-<p><img src="/images/dofn-sequence-diagram.png" alt="This is a sequence diagram that shows the lifecycle of the DoFn" /></p>
+<!-- The source for the sequence diagram can be found in the the SVG resource. -->
+<p><img src="/images/dofn-sequence-diagram.svg" alt="This is a sequence diagram that shows the lifecycle of the DoFn" /></p>
<h4 id="groupbykey">4.2.2. GroupByKey</h4>
diff --git a/website/generated-content/images/dofn-sequence-diagram.png b/website/generated-content/images/dofn-sequence-diagram.png
deleted file mode 100644
index cf6570d..0000000
Binary files a/website/generated-content/images/dofn-sequence-diagram.png and /dev/null differ
diff --git a/website/generated-content/images/dofn-sequence-diagram.svg b/website/generated-content/images/dofn-sequence-diagram.svg
new file mode 100644
index 0000000..898b1ae
--- /dev/null
+++ b/website/generated-content/images/dofn-sequence-diagram.svg
@@ -0,0 +1,94 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+-->
+<svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" contentScriptType="application/ecmascript" contentStyleType="text/css" height="863px" preserveAspectRatio="none" style="width:740px;height:863px;" version="1.1" viewBox="0 0 740 863" width="740px" zoomAndPan="magnify"><defs/><g><rect fill="#6B9FE6" height="781.5625" style="stroke: #6B9FE6; stroke-width: 1.0;" width="10" x="49" y="60.8125"/><rect fill="#6B9FE6" height="425.2813" style="stroke: #6B9FE6; stro [...]
+@startuml
+
+hide footbox
+skinparam backgroundColor transparent
+skinparam shadowing false
+skinparam defaultFontName "Roboto"
+
+skinparam sequenceArrowThickness 2
+
+skinparam note {
+ BackgroundColor #cee2f2
+ BorderColor #cee2f2
+}
+
+skinparam sequence {
+ ArrowColor #67666a
+
+ LifeLineBorderColor #6b9fe6
+ LifeLineBackgroundColor #6b9fe6
+
+ GroupBackgroundColor #8ac483
+ GroupBorderColor #8ac483
+
+ ParticipantBackgroundColor #8ac483
+ ParticipantBorderColor #8ac483
+}
+
+participant "User pipeline" as Pipeline
+participant DoFn << Serializable >>
+note right of DoFn: can have non-transient instance\nvariable state that will be deserialized
+note right of DoFn: do not include enclosing class serializable state; use static\nnested DoFn or define as anonymous class in static method
+note right of DoFn: no shared (global) static variable access (no sync mechanism) but a beam\nstate (based on engine mechanisms) can be injected to processElement
+note right of DoFn: keep as pure function as possible or idempotent side effects\nbecause DoFns can be retried on failed bundles
+
+participant Runner
+
+activate Pipeline
+Pipeline -> DoFn: **create DoFn **
+DoFn -> Runner: **passed instance or deserialized on workers**
+
+note right Pipeline: If state variables are known at pipeline construction step\ninitialize state variables by constructor
+
+group DoFn Lifecycle
+ Runner -> DoFn: **call setup**
+ activate Runner
+ activate DoFn
+ note right DoFn: reused instance to process other bundles on the same worker
+ note right DoFn: If state variables do not depend on the main pipeline program and are the\nsame for all DoFn instances initialize them in setup
+ group For each bundle
+ Runner -> DoFn: **call startBundle**
+ group For each element
+ Runner -> DoFn: **call processElement**
+ note right DoFn: If state variables are computed by the pipeline\npass it in a PcollectionView as a side input
+ DoFn - -> Runner: output
+ end
+ DoFn <- Runner: call onTimer
+ DoFn <- Runner: **call finishBundle**
+ end
+ DoFn -> Runner: **If DoFn is no more needed: call tearDown**
+ note right DoFn: Call of teardown is best effort; do not use for side effects
+end
+
+@enduml
+
+PlantUML version 1.2019.11(Sun Sep 22 12:02:15 CEST 2019)
+(GPL source distribution)
+Java Runtime: OpenJDK Runtime Environment
+JVM: OpenJDK 64-Bit Server VM
+Java Version: 1.8.0_222-b10
+Operating System: Linux
+Default Encoding: UTF-8
+Language: en
+Country: CA
+--></g></svg>
diff --git a/website/generated-content/images/source-sequence-diagram.png b/website/generated-content/images/source-sequence-diagram.png
deleted file mode 100644
index d544eea..0000000
Binary files a/website/generated-content/images/source-sequence-diagram.png and /dev/null differ
diff --git a/website/generated-content/images/source-sequence-diagram.svg b/website/generated-content/images/source-sequence-diagram.svg
new file mode 100644
index 0000000..02facd6
--- /dev/null
+++ b/website/generated-content/images/source-sequence-diagram.svg
@@ -0,0 +1,106 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+-->
+<svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" contentScriptType="application/ecmascript" contentStyleType="text/css" height="538px" preserveAspectRatio="none" style="width:574px;height:538px;" version="1.1" viewBox="0 0 574 538" width="574px" zoomAndPan="magnify"><defs/><g><rect fill="#6B9FE6" height="435.2813" style="stroke: #6B9FE6; stroke-width: 1.0;" width="10" x="250.5" y="82.0469"/><rect fill="#6B9FE6" height="152.1719" style="stroke: #6B9FE6; s [...]
+@startuml
+
+hide footbox
+skinparam backgroundColor transparent
+skinparam shadowing false
+skinparam defaultFontName "Roboto"
+
+skinparam sequenceArrowThickness 2
+
+skinparam note {
+ BackgroundColor #cee2f2
+ BorderColor #cee2f2
+}
+
+skinparam sequence {
+ ArrowColor #67666a
+
+ LifeLineBorderColor #6b9fe6
+ LifeLineBackgroundColor #6b9fe6
+
+ GroupBackgroundColor #8ac483
+ GroupBorderColor #8ac483
+
+ ParticipantBackgroundColor #8ac483
+ ParticipantBorderColor #8ac483
+}
+
+participant Runner
+participant "Source" << Serializable >>
+participant "Reader"
+participant Middleware
+
+Runner -> Source: **create source**
+activate Source
+
+Runner -> Source: get estimated size
+
+Source -> Middleware: **estimate size**
+activate Middleware
+
+Source - -> Runner: size of data
+
+Runner -> Runner: compute size / number of executors\n = desired bundle size
+
+Runner -> Source: split source (desired bundle size)
+note right
+ streaming: split based on number of executors
+end note
+
+Source - -> Runner: list<source>
+note right
+ streaming: runner asks the source for watermark
+end note
+
+Runner -> Source: **for each source create a reader**
+
+Source -> Reader: **create a reader**
+activate Reader
+
+Reader - -> Runner: **readers**
+note right
+ streaming: + checkpoint
+end note
+
+Runner -> Reader: **for each reader : start reader**
+
+Runner -> Reader: **read elements until none to read**
+
+Reader -> Middleware: **get element**
+
+deactivate Middleware
+
+Runner -> Reader: **close reader**
+deactivate Reader
+@enduml
+
+PlantUML version 1.2019.11(Sun Sep 22 12:02:15 CEST 2019)
+(GPL source distribution)
+Java Runtime: OpenJDK Runtime Environment
+JVM: OpenJDK 64-Bit Server VM
+Java Version: 1.8.0_222-b10
+Operating System: Linux
+Default Encoding: UTF-8
+Language: en
+Country: CA
+--></g></svg>