You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@samza.apache.org by xi...@apache.org on 2018/05/25 22:28:41 UTC
svn commit: r1832281 [13/15] - in /samza/site: ./ community/ contribute/
learn/documentation/latest/ learn/documentation/latest/api/
learn/documentation/latest/api/javadocs/
learn/documentation/latest/api/javadocs/org/apache/samza/
learn/documentation/...
Modified: samza/site/learn/tutorials/latest/hello-samza-high-level-code.html
URL: http://svn.apache.org/viewvc/samza/site/learn/tutorials/latest/hello-samza-high-level-code.html?rev=1832281&r1=1832280&r2=1832281&view=diff
==============================================================================
--- samza/site/learn/tutorials/latest/hello-samza-high-level-code.html (original)
+++ samza/site/learn/tutorials/latest/hello-samza-high-level-code.html Fri May 25 22:28:38 2018
@@ -44,9 +44,7 @@
<!-- this icon only shows in versioned pages -->
-
-
- <a href="http://samza.apache.org/learn/tutorials/0.14/hello-samza-high-level-code.html"><i id="switch-version-button"></i></a>
+ <a href="http://samza.apache.org/learn/tutorials/0.14/hello-samza-high-level-code"><i id="switch-version-button"></i></a>
<!-- links for the navigation bar -->
@@ -157,9 +155,8 @@
<p>Check out the hello-samza project:</p>
-<div class="highlight"><pre><code class="bash">git clone https://git.apache.org/samza-hello-samza.git hello-samza
-<span class="nb">cd </span>hello-samza
-git checkout latest</code></pre></div>
+<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span></span>git clone https://git.apache.org/samza-hello-samza.git hello-samza
+<span class="nb">cd</span> hello-samza</code></pre></figure>
<p>This project already contains implementations of the wikipedia application using both the low-level task API and the high-level API. The low-level task implementations are in the <code>samza.examples.wikipedia.task</code> package. The high-level application implementation is in the <code>samza.examples.wikipedia.application</code> package.</p>
@@ -182,22 +179,22 @@ Create a new file named <em>my-wikipedia
<p>Let’s start by adding some of the core properties to the file:</p>
-<div class="highlight"><pre><code class="bash"><span class="c"># Licensed to the Apache Software Foundation (ASF) under one</span>
-<span class="c"># or more contributor license agreements. See the NOTICE file</span>
-<span class="c"># distributed with this work for additional information</span>
-<span class="c"># regarding copyright ownership. The ASF licenses this file</span>
-<span class="c"># to you under the Apache License, Version 2.0 (the</span>
-<span class="c"># "License"); you may not use this file except in compliance</span>
-<span class="c"># with the License. You may obtain a copy of the License at</span>
-<span class="c">#</span>
-<span class="c"># http://www.apache.org/licenses/LICENSE-2.0</span>
-<span class="c">#</span>
-<span class="c"># Unless required by applicable law or agreed to in writing,</span>
-<span class="c"># software distributed under the License is distributed on an</span>
-<span class="c"># "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY</span>
-<span class="c"># KIND, either express or implied. See the License for the</span>
-<span class="c"># specific language governing permissions and limitations</span>
-<span class="c"># under the License.</span>
+<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span></span><span class="c1"># Licensed to the Apache Software Foundation (ASF) under one</span>
+<span class="c1"># or more contributor license agreements. See the NOTICE file</span>
+<span class="c1"># distributed with this work for additional information</span>
+<span class="c1"># regarding copyright ownership. The ASF licenses this file</span>
+<span class="c1"># to you under the Apache License, Version 2.0 (the</span>
+<span class="c1"># "License"); you may not use this file except in compliance</span>
+<span class="c1"># with the License. You may obtain a copy of the License at</span>
+<span class="c1">#</span>
+<span class="c1"># http://www.apache.org/licenses/LICENSE-2.0</span>
+<span class="c1">#</span>
+<span class="c1"># Unless required by applicable law or agreed to in writing,</span>
+<span class="c1"># software distributed under the License is distributed on an</span>
+<span class="c1"># "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY</span>
+<span class="c1"># KIND, either express or implied. See the License for the</span>
+<span class="c1"># specific language governing permissions and limitations</span>
+<span class="c1"># under the License.</span>
app.class<span class="o">=</span>samza.examples.wikipedia.application.MyWikipediaApplication
app.runner.class<span class="o">=</span>org.apache.samza.runtime.RemoteApplicationRunner
@@ -206,7 +203,7 @@ job.factory.class<span class="o">=</span
job.name<span class="o">=</span>my-wikipedia-application
job.default.system<span class="o">=</span>kafka
-yarn.package.path<span class="o">=</span>file://<span class="k">${</span><span class="nv">basedir</span><span class="k">}</span>/target/<span class="k">${</span><span class="nv">project</span><span class="p">.artifactId</span><span class="k">}</span>-<span class="k">${</span><span class="nv">pom</span><span class="p">.version</span><span class="k">}</span>-dist.tar.gz</code></pre></div>
+yarn.package.path<span class="o">=</span>file://<span class="si">${</span><span class="nv">basedir</span><span class="si">}</span>/target/<span class="si">${</span><span class="nv">project</span><span class="p">.artifactId</span><span class="si">}</span>-<span class="si">${</span><span class="nv">pom</span><span class="p">.version</span><span class="si">}</span>-dist.tar.gz</code></pre></figure>
<p>Be sure to include the Apache header. The project will not compile without it. </p>
@@ -229,14 +226,14 @@ yarn.package.path<span class="o">=</span
<p>This Wikipedia application will consume events from Wikipedia and produce stats to a Kafka topic. We need to define those systems in config before Samza can use them. Add the following lines to the config:</p>
-<div class="highlight"><pre><code class="bash">systems.wikipedia.samza.factory<span class="o">=</span>samza.examples.wikipedia.system.WikipediaSystemFactory
+<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span></span>systems.wikipedia.samza.factory<span class="o">=</span>samza.examples.wikipedia.system.WikipediaSystemFactory
systems.wikipedia.host<span class="o">=</span>irc.wikimedia.org
-systems.wikipedia.port<span class="o">=</span>6667
+systems.wikipedia.port<span class="o">=</span><span class="m">6667</span>
systems.kafka.samza.factory<span class="o">=</span>org.apache.samza.system.kafka.KafkaSystemFactory
systems.kafka.consumer.zookeeper.connect<span class="o">=</span>localhost:2181/
systems.kafka.producer.bootstrap.servers<span class="o">=</span>localhost:9092
-systems.kafka.default.stream.replication.factor<span class="o">=</span>1</code></pre></div>
+systems.kafka.default.stream.replication.factor<span class="o">=</span><span class="m">1</span></code></pre></figure>
<p>The above configuration defines 2 systems; one called <em>wikipedia</em> and one called <em>kafka</em>.</p>
@@ -260,14 +257,14 @@ systems.kafka.default.stream.replication
<p>While the <em>wikipedia</em> system is necessary for case 1, it does not support producers (we can’t write Samza output to Wikipedia), which are needed for cases 2-3. So it is more convenient to use <em>kafka</em> as the default system. We can then explicitly configure the input streams to use the <em>wikipedia</em> system.</p>
-<div class="highlight"><pre><code class="bash">streams.en-wikipedia.samza.system<span class="o">=</span>wikipedia
-streams.en-wikipedia.samza.physical.name<span class="o">=</span><span class="c">#en.wikipedia</span>
+<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span></span>streams.en-wikipedia.samza.system<span class="o">=</span>wikipedia
+streams.en-wikipedia.samza.physical.name<span class="o">=</span><span class="c1">#en.wikipedia</span>
streams.en-wiktionary.samza.system<span class="o">=</span>wikipedia
-streams.en-wiktionary.samza.physical.name<span class="o">=</span><span class="c">#en.wiktionary</span>
+streams.en-wiktionary.samza.physical.name<span class="o">=</span><span class="c1">#en.wiktionary</span>
streams.en-wikinews.samza.system<span class="o">=</span>wikipedia
-streams.en-wikinews.samza.physical.name<span class="o">=</span><span class="c">#en.wikinews</span></code></pre></div>
+streams.en-wikinews.samza.physical.name<span class="o">=</span><span class="c1">#en.wikinews</span></code></pre></figure>
<p>The above configurations declare 3 streams with IDs, <em>en-wikipedia</em>, <em>en-wiktionary</em>, and <em>en-wikinews</em>. It associates each stream with the <em>wikipedia</em> system we defined earlier and set the physical name to the corresponding Wikipedia channel. </p>
@@ -283,7 +280,7 @@ streams.en-wikinews.samza.physical.name<
<p>Create a new class named <code>MyWikipediaApplication</code> in the <code>samza.examples.wikipedia.application</code> package. The class must implement <a href="/learn/documentation/latest/api/javadocs/org/apache/samza/application/StreamApplication.html">StreamApplication</a> and should look like this:</p>
-<div class="highlight"><pre><code class="java"><span class="cm">/*</span>
+<figure class="highlight"><pre><code class="language-java" data-lang="java"><span></span><span class="cm">/*</span>
<span class="cm"> * Licensed to the Apache Software Foundation (ASF) under one</span>
<span class="cm"> * or more contributor license agreements. See the NOTICE file</span>
<span class="cm"> * distributed with this work for additional information</span>
@@ -301,7 +298,7 @@ streams.en-wikinews.samza.physical.name<
<span class="cm"> * specific language governing permissions and limitations</span>
<span class="cm"> * under the License.</span>
<span class="cm"> */</span>
-<span class="kn">package</span> <span class="n">samza</span><span class="o">.</span><span class="na">examples</span><span class="o">.</span><span class="na">wikipedia</span><span class="o">.</span><span class="na">application</span><span class="o">;</span>
+<span class="kn">package</span> <span class="nn">samza.examples.wikipedia.application</span><span class="o">;</span>
<span class="kn">import</span> <span class="nn">org.apache.samza.application.StreamApplication</span><span class="o">;</span>
<span class="kn">import</span> <span class="nn">org.apache.samza.config.Config</span><span class="o">;</span>
@@ -312,7 +309,7 @@ streams.en-wikinews.samza.physical.name<
<span class="kd">public</span> <span class="kt">void</span> <span class="nf">init</span><span class="o">(</span><span class="n">StreamGraph</span> <span class="n">streamGraph</span><span class="o">,</span> <span class="n">Config</span> <span class="n">config</span><span class="o">)</span> <span class="o">{</span>
<span class="o">}</span>
-<span class="o">}</span></code></pre></div>
+<span class="o">}</span></code></pre></figure>
<p>Be sure to include the Apache header. The project will not compile without it.</p>
@@ -324,9 +321,9 @@ streams.en-wikinews.samza.physical.name<
<p>The Wikipedia application consumes events from three channels. Let’s declare each of those channels as an input streams via the <a href="/learn/documentation/latest/api/javadocs/org/apache/samza/operators/StreamGraph.html">StreamGraph</a> in the <a href="/learn/documentation/latest/api/javadocs/org/apache/samza/application/StreamApplication.html#init-org.apache.samza.operators.StreamGraph-org.apache.samza.config.Config-">init</a> method.</p>
-<div class="highlight"><pre><code class="java"><span class="n">MessageStream</span><span class="o"><</span><span class="n">WikipediaFeedEvent</span><span class="o">></span> <span class="n">wikipediaEvents</span> <span class="o">=</span> <span class="n">streamGraph</span><span class="o">.</span><span class="na">getInputStream</span><span class="o">(</span><span class="s">"en-wikipedia"</span><span class="o">,</span> <span class="k">new</span> <span class="n">NoOpSerde</span><span class="o"><>());</span>
+<figure class="highlight"><pre><code class="language-java" data-lang="java"><span></span><span class="n">MessageStream</span><span class="o"><</span><span class="n">WikipediaFeedEvent</span><span class="o">></span> <span class="n">wikipediaEvents</span> <span class="o">=</span> <span class="n">streamGraph</span><span class="o">.</span><span class="na">getInputStream</span><span class="o">(</span><span class="s">"en-wikipedia"</span><span class="o">,</span> <span class="k">new</span> <span class="n">NoOpSerde</span><span class="o"><>());</span>
<span class="n">MessageStream</span><span class="o"><</span><span class="n">WikipediaFeedEvent</span><span class="o">></span> <span class="n">wiktionaryEvents</span> <span class="o">=</span> <span class="n">streamGraph</span><span class="o">.</span><span class="na">getInputStream</span><span class="o">(</span><span class="s">"en-wiktionary"</span><span class="o">,</span> <span class="k">new</span> <span class="n">NoOpSerde</span><span class="o"><>());</span>
-<span class="n">MessageStream</span><span class="o"><</span><span class="n">WikipediaFeedEvent</span><span class="o">></span> <span class="n">wikiNewsEvents</span> <span class="o">=</span> <span class="n">streamGraph</span><span class="o">.</span><span class="na">getInputStream</span><span class="o">(</span><span class="s">"en-wikinews"</span><span class="o">,</span> <span class="k">new</span> <span class="n">NoOpSerde</span><span class="o"><>());</span></code></pre></div>
+<span class="n">MessageStream</span><span class="o"><</span><span class="n">WikipediaFeedEvent</span><span class="o">></span> <span class="n">wikiNewsEvents</span> <span class="o">=</span> <span class="n">streamGraph</span><span class="o">.</span><span class="na">getInputStream</span><span class="o">(</span><span class="s">"en-wikinews"</span><span class="o">,</span> <span class="k">new</span> <span class="n">NoOpSerde</span><span class="o"><>());</span></code></pre></figure>
<p>The first argument to the <a href="/learn/documentation/latest/api/javadocs/org/apache/samza/operators/StreamGraph.html#getInputStream-java.lang.String-org.apache.samza.serializers.Serde-">getInputStream</a> method is the stream ID. Each ID must match the corresponding stream IDs we configured earlier.
The second argument is the <code>Serde</code> used to deserialize the message. We’ve set this to a <code>NoOpSerde</code> since our <code>wikipedia</code> system already returns <code>WikipediaFeedEvent</code>s and there is no need for further deserialization.</p>
@@ -339,7 +336,7 @@ The second argument is the <code>Serde</
<p>Add the following snippet to the <a href="/learn/documentation/latest/api/javadocs/org/apache/samza/application/StreamApplication.html#init-org.apache.samza.operators.StreamGraph-org.apache.samza.config.Config-">init</a> method. It merges all the input streams into a new one called <em>allWikipediaEvents</em></p>
-<div class="highlight"><pre><code class="java"><span class="n">MessageStream</span><span class="o"><</span><span class="n">WikipediaFeed</span><span class="o">.</span><span class="na">WikipediaFeedEvent</span><span class="o">></span> <span class="n">allWikipediaEvents</span> <span class="o">=</span> <span class="n">MessageStream</span><span class="o">.</span><span class="na">mergeAll</span><span class="o">(</span><span class="n">ImmutableList</span><span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="n">wikipediaEvents</span><span class="o">,</span> <span class="n">wiktionaryEvents</span><span class="o">,</span> <span class="n">wikiNewsEvents</span><span class="o">));</span></code></pre></div>
+<figure class="highlight"><pre><code class="language-java" data-lang="java"><span></span><span class="n">MessageStream</span><span class="o"><</span><span class="n">WikipediaFeed</span><span class="o">.</span><span class="na">WikipediaFeedEvent</span><span class="o">></span> <span class="n">allWikipediaEvents</span> <span class="o">=</span> <span class="n">MessageStream</span><span class="o">.</span><span class="na">mergeAll</span><span class="o">(</span><span class="n">ImmutableList</span><span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="n">wikipediaEvents</span><span class="o">,</span> <span class="n">wiktionaryEvents</span><span class="o">,</span> <span class="n">wikiNewsEvents</span><span class="o">));</span></code></pre></figure>
<p>Note there is a <a href="/learn/documentation/latest/api/javadocs/org/apache/samza/operators/MessageStream.html#merge-java.util.Collection-">merge</a> operator instance method on MessageStream, but the static <a href="/learn/documentation/latest/api/javadocs/org/apache/samza/operators/MessageStream.html#mergeAll-java.util.Collection-">mergeAll</a> method is a more convenient alternative if you need to merge many streams.</p>
@@ -349,7 +346,7 @@ The second argument is the <code>Serde</
<p>In the <a href="/learn/documentation/latest/api/javadocs/org/apache/samza/application/StreamApplication.html#init-org.apache.samza.operators.StreamGraph-org.apache.samza.config.Config-">init</a> method, invoke the <a href="/learn/documentation/latest/api/javadocs/org/apache/samza/operators/MessageStream.html#map-org.apache.samza.operators.functions.MapFunction-">map</a> operation on <code>allWikipediaEvents</code>, passing the <code>WikipediaParser::parseEvent</code> method reference as follows:</p>
-<div class="highlight"><pre><code class="java"><span class="n">allWikipediaEvents</span><span class="o">.</span><span class="na">map</span><span class="o">(</span><span class="nl">WikipediaParser:</span><span class="o">:</span><span class="n">parseEvent</span><span class="o">);</span></code></pre></div>
+<figure class="highlight"><pre><code class="language-java" data-lang="java"><span></span><span class="n">allWikipediaEvents</span><span class="o">.</span><span class="na">map</span><span class="o">(</span><span class="n">WikipediaParser</span><span class="o">::</span><span class="n">parseEvent</span><span class="o">);</span></code></pre></figure>
<h4 id="window">Window</h4>
@@ -357,16 +354,16 @@ The second argument is the <code>Serde</
<p>First, we need a container class for statistics we want to track. Add the following static class after the <a href="/learn/documentation/latest/api/javadocs/org/apache/samza/application/StreamApplication.html#init-org.apache.samza.operators.StreamGraph-org.apache.samza.config.Config-">init</a> method.</p>
-<div class="highlight"><pre><code class="java"><span class="kd">private</span> <span class="kd">static</span> <span class="kd">class</span> <span class="nc">WikipediaStats</span> <span class="o">{</span>
+<figure class="highlight"><pre><code class="language-java" data-lang="java"><span></span><span class="kd">private</span> <span class="kd">static</span> <span class="kd">class</span> <span class="nc">WikipediaStats</span> <span class="o">{</span>
<span class="kt">int</span> <span class="n">edits</span> <span class="o">=</span> <span class="mi">0</span><span class="o">;</span>
<span class="kt">int</span> <span class="n">byteDiff</span> <span class="o">=</span> <span class="mi">0</span><span class="o">;</span>
<span class="n">Set</span><span class="o"><</span><span class="n">String</span><span class="o">></span> <span class="n">titles</span> <span class="o">=</span> <span class="k">new</span> <span class="n">HashSet</span><span class="o"><</span><span class="n">String</span><span class="o">>();</span>
<span class="n">Map</span><span class="o"><</span><span class="n">String</span><span class="o">,</span> <span class="n">Integer</span><span class="o">></span> <span class="n">counts</span> <span class="o">=</span> <span class="k">new</span> <span class="n">HashMap</span><span class="o"><</span><span class="n">String</span><span class="o">,</span> <span class="n">Integer</span><span class="o">>();</span>
-<span class="o">}</span></code></pre></div>
+<span class="o">}</span></code></pre></figure>
<p>Now we need to define the logic to aggregate the stats over the duration of the window. To do this, we implement <a href="/learn/documentation/latest/api/javadocs/org/apache/samza/operators/functions/FoldLeftFunction.html">FoldLeftFunction</a> by adding the following class after the <code>WikipediaStats</code> class:</p>
-<div class="highlight"><pre><code class="java"><span class="kd">private</span> <span class="kd">class</span> <span class="nc">WikipediaStatsAggregator</span> <span class="kd">implements</span> <span class="n">FoldLeftFunction</span><span class="o"><</span><span class="n">Map</span><span class="o"><</span><span class="n">String</span><span class="o">,</span> <span class="n">Object</span><span class="o">>,</span> <span class="n">WikipediaStats</span><span class="o">></span> <span class="o">{</span>
+<figure class="highlight"><pre><code class="language-java" data-lang="java"><span></span><span class="kd">private</span> <span class="kd">class</span> <span class="nc">WikipediaStatsAggregator</span> <span class="kd">implements</span> <span class="n">FoldLeftFunction</span><span class="o"><</span><span class="n">Map</span><span class="o"><</span><span class="n">String</span><span class="o">,</span> <span class="n">Object</span><span class="o">>,</span> <span class="n">WikipediaStats</span><span class="o">></span> <span class="o">{</span>
<span class="nd">@Override</span>
<span class="kd">public</span> <span class="n">WikipediaStats</span> <span class="nf">apply</span><span class="o">(</span><span class="n">Map</span><span class="o"><</span><span class="n">String</span><span class="o">,</span> <span class="n">Object</span><span class="o">></span> <span class="n">edit</span><span class="o">,</span> <span class="n">WikipediaStats</span> <span class="n">stats</span><span class="o">)</span> <span class="o">{</span>
@@ -384,14 +381,14 @@ The second argument is the <code>Serde</
<span class="k">return</span> <span class="n">stats</span><span class="o">;</span>
<span class="o">}</span>
-<span class="o">}</span></code></pre></div>
+<span class="o">}</span></code></pre></figure>
<p>Note: the type parameters for <a href="/learn/documentation/latest/api/javadocs/org/apache/samza/operators/functions/FoldLeftFunction.html">FoldLeftFunction</a> reflect the upstream data type and the window value type, respectively. In our case, the upstream type is the output of the parser and the window value is our <code>WikipediaStats</code> class.</p>
<p>Finally, we can define our <a href="/learn/documentation/latest/api/javadocs/org/apache/samza/operators/windows/Window.html">window</a> back in the <a href="/learn/documentation/latest/api/javadocs/org/apache/samza/application/StreamApplication.html#init-org.apache.samza.operators.StreamGraph-org.apache.samza.config.Config-">init</a> method by chaining the result of the parser:</p>
-<div class="highlight"><pre><code class="java"><span class="n">allWikipediaEvents</span><span class="o">.</span><span class="na">map</span><span class="o">(</span><span class="nl">WikipediaParser:</span><span class="o">:</span><span class="n">parseEvent</span><span class="o">)</span>
- <span class="o">.</span><span class="na">window</span><span class="o">(</span><span class="n">Windows</span><span class="o">.</span><span class="na">tumblingWindow</span><span class="o">(</span><span class="n">Duration</span><span class="o">.</span><span class="na">ofSeconds</span><span class="o">(</span><span class="mi">10</span><span class="o">),</span> <span class="nl">WikipediaStats:</span><span class="o">:</span><span class="k">new</span><span class="o">,</span> <span class="k">new</span> <span class="n">WikipediaStatsAggregator</span><span class="o">(),</span> <span class="k">new</span> <span class="n">JsonSerdeV2</span><span class="o"><>(</span><span class="n">WikipediaStats</span><span class="o">.</span><span class="na">class</span><span class="o">)));</span></code></pre></div>
+<figure class="highlight"><pre><code class="language-java" data-lang="java"><span></span><span class="n">allWikipediaEvents</span><span class="o">.</span><span class="na">map</span><span class="o">(</span><span class="n">WikipediaParser</span><span class="o">::</span><span class="n">parseEvent</span><span class="o">)</span>
+ <span class="o">.</span><span class="na">window</span><span class="o">(</span><span class="n">Windows</span><span class="o">.</span><span class="na">tumblingWindow</span><span class="o">(</span><span class="n">Duration</span><span class="o">.</span><span class="na">ofSeconds</span><span class="o">(</span><span class="mi">10</span><span class="o">),</span> <span class="n">WikipediaStats</span><span class="o">::</span><span class="k">new</span><span class="o">,</span> <span class="k">new</span> <span class="n">WikipediaStatsAggregator</span><span class="o">(),</span> <span class="k">new</span> <span class="n">JsonSerdeV2</span><span class="o"><>(</span><span class="n">WikipediaStats</span><span class="o">.</span><span class="na">class</span><span class="o">)));</span></code></pre></figure>
<p>This defines an unkeyed <a href="/learn/documentation/latest/api/javadocs/org/apache/samza/operators/windows/Windows.html">tumbling window</a> that spans 10s, which instantiates a new <code>WikipediaStats</code> object at the beginning of each window and aggregates the stats using <code>WikipediaStatsAggregator</code>.</p>
@@ -401,7 +398,7 @@ The second argument is the <code>Serde</
<p>We will do a <a href="/learn/documentation/latest/api/javadocs/org/apache/samza/operators/MessageStream.html#map-org.apache.samza.operators.functions.MapFunction-">map</a> at the end to format our window output. Let’s begin by defining a simple container class for our formatted output.</p>
-<div class="highlight"><pre><code class="java"><span class="kd">static</span> <span class="kd">class</span> <span class="nc">WikipediaStatsOutput</span> <span class="o">{</span>
+<figure class="highlight"><pre><code class="language-java" data-lang="java"><span></span> <span class="kd">static</span> <span class="kd">class</span> <span class="nc">WikipediaStatsOutput</span> <span class="o">{</span>
<span class="kd">public</span> <span class="kt">int</span> <span class="n">edits</span><span class="o">;</span>
<span class="kd">public</span> <span class="kt">int</span> <span class="n">bytesAdded</span><span class="o">;</span>
<span class="kd">public</span> <span class="kt">int</span> <span class="n">uniqueTitles</span><span class="o">;</span>
@@ -414,26 +411,26 @@ The second argument is the <code>Serde</
<span class="k">this</span><span class="o">.</span><span class="na">uniqueTitles</span> <span class="o">=</span> <span class="n">uniqueTitles</span><span class="o">;</span>
<span class="k">this</span><span class="o">.</span><span class="na">counts</span> <span class="o">=</span> <span class="n">counts</span><span class="o">;</span>
<span class="o">}</span>
- <span class="o">}</span></code></pre></div>
+ <span class="o">}</span></code></pre></figure>
<p>Paste the following after the aggregator class:</p>
-<div class="highlight"><pre><code class="java"><span class="kd">private</span> <span class="n">WikipediaStatsOutput</span> <span class="nf">formatOutput</span><span class="o">(</span><span class="n">WindowPane</span><span class="o"><</span><span class="n">Void</span><span class="o">,</span> <span class="n">WikipediaStats</span><span class="o">></span> <span class="n">statsWindowPane</span><span class="o">)</span> <span class="o">{</span>
+<figure class="highlight"><pre><code class="language-java" data-lang="java"><span></span> <span class="kd">private</span> <span class="n">WikipediaStatsOutput</span> <span class="nf">formatOutput</span><span class="o">(</span><span class="n">WindowPane</span><span class="o"><</span><span class="n">Void</span><span class="o">,</span> <span class="n">WikipediaStats</span><span class="o">></span> <span class="n">statsWindowPane</span><span class="o">)</span> <span class="o">{</span>
<span class="n">WikipediaStats</span> <span class="n">stats</span> <span class="o">=</span> <span class="n">statsWindowPane</span><span class="o">.</span><span class="na">getMessage</span><span class="o">();</span>
- <span class="k">return</span> <span class="k">new</span> <span class="nf">WikipediaStatsOutput</span><span class="o">(</span>
+ <span class="k">return</span> <span class="k">new</span> <span class="n">WikipediaStatsOutput</span><span class="o">(</span>
<span class="n">stats</span><span class="o">.</span><span class="na">edits</span><span class="o">,</span> <span class="n">stats</span><span class="o">.</span><span class="na">byteDiff</span><span class="o">,</span> <span class="n">stats</span><span class="o">.</span><span class="na">titles</span><span class="o">.</span><span class="na">size</span><span class="o">(),</span> <span class="n">stats</span><span class="o">.</span><span class="na">counts</span><span class="o">);</span>
- <span class="o">}</span></code></pre></div>
+ <span class="o">}</span></code></pre></figure>
<p>Now, we can invoke the method by adding another <a href="/learn/documentation/latest/api/javadocs/org/apache/samza/operators/MessageStream.html#map-org.apache.samza.operators.functions.MapFunction-">map</a> operation to the chain in <a href="/learn/documentation/latest/api/javadocs/org/apache/samza/application/StreamApplication.html#init-org.apache.samza.operators.StreamGraph-org.apache.samza.config.Config-">init</a>. The operator chain should now look like this:</p>
-<div class="highlight"><pre><code class="java"><span class="n">allWikipediaEvents</span><span class="o">.</span><span class="na">map</span><span class="o">(</span><span class="nl">WikipediaParser:</span><span class="o">:</span><span class="n">parseEvent</span><span class="o">)</span>
- <span class="o">.</span><span class="na">window</span><span class="o">(</span><span class="n">Windows</span><span class="o">.</span><span class="na">tumblingWindow</span><span class="o">(</span><span class="n">Duration</span><span class="o">.</span><span class="na">ofSeconds</span><span class="o">(</span><span class="mi">10</span><span class="o">),</span> <span class="nl">WikipediaStats:</span><span class="o">:</span><span class="k">new</span><span class="o">,</span> <span class="k">new</span> <span class="n">WikipediaStatsAggregator</span><span class="o">()))</span>
- <span class="o">.</span><span class="na">map</span><span class="o">(</span><span class="k">this</span><span class="o">::</span><span class="n">formatOutput</span><span class="o">);</span></code></pre></div>
+<figure class="highlight"><pre><code class="language-java" data-lang="java"><span></span><span class="n">allWikipediaEvents</span><span class="o">.</span><span class="na">map</span><span class="o">(</span><span class="n">WikipediaParser</span><span class="o">::</span><span class="n">parseEvent</span><span class="o">)</span>
+ <span class="o">.</span><span class="na">window</span><span class="o">(</span><span class="n">Windows</span><span class="o">.</span><span class="na">tumblingWindow</span><span class="o">(</span><span class="n">Duration</span><span class="o">.</span><span class="na">ofSeconds</span><span class="o">(</span><span class="mi">10</span><span class="o">),</span> <span class="n">WikipediaStats</span><span class="o">::</span><span class="k">new</span><span class="o">,</span> <span class="k">new</span> <span class="n">WikipediaStatsAggregator</span><span class="o">()))</span>
+ <span class="o">.</span><span class="na">map</span><span class="o">(</span><span class="k">this</span><span class="o">::</span><span class="n">formatOutput</span><span class="o">);</span></code></pre></figure>
<p>Next we need to get the output stream to which we will send the stats. Insert the following line below the creation of the 3 input streams:</p>
-<div class="highlight"><pre><code class="java"><span class="n">OutputStream</span><span class="o"><</span><span class="n">WikipediaStatsOutput</span><span class="o">></span> <span class="n">wikipediaStats</span> <span class="o">=</span>
- <span class="n">graph</span><span class="o">.</span><span class="na">getOutputStream</span><span class="o">(</span><span class="s">"wikipedia-stats"</span><span class="o">,</span> <span class="k">new</span> <span class="n">JsonSerdeV2</span><span class="o"><>(</span><span class="n">WikipediaStatsOutput</span><span class="o">.</span><span class="na">class</span><span class="o">));</span></code></pre></div>
+<figure class="highlight"><pre><code class="language-java" data-lang="java"><span></span> <span class="n">OutputStream</span><span class="o"><</span><span class="n">WikipediaStatsOutput</span><span class="o">></span> <span class="n">wikipediaStats</span> <span class="o">=</span>
+ <span class="n">graph</span><span class="o">.</span><span class="na">getOutputStream</span><span class="o">(</span><span class="s">"wikipedia-stats"</span><span class="o">,</span> <span class="k">new</span> <span class="n">JsonSerdeV2</span><span class="o"><>(</span><span class="n">WikipediaStatsOutput</span><span class="o">.</span><span class="na">class</span><span class="o">));</span></code></pre></figure>
<p>The <a href="/learn/documentation/latest/api/javadocs/org/apache/samza/operators/OutputStream.html">OutputStream</a> is parameterized by the type of the output.</p>
@@ -442,10 +439,10 @@ The second parameter is the <a href="/le
<p>Finally, we can send our output to the output stream using the <a href="/learn/documentation/latest/api/javadocs/org/apache/samza/operators/MessageStream.html#sendTo-org.apache.samza.operators.OutputStream-">sendTo</a> operator:</p>
-<div class="highlight"><pre><code class="java"><span class="n">allWikipediaEvents</span><span class="o">.</span><span class="na">map</span><span class="o">(</span><span class="nl">WikipediaParser:</span><span class="o">:</span><span class="n">parseEvent</span><span class="o">)</span>
- <span class="o">.</span><span class="na">window</span><span class="o">(</span><span class="n">Windows</span><span class="o">.</span><span class="na">tumblingWindow</span><span class="o">(</span><span class="n">Duration</span><span class="o">.</span><span class="na">ofSeconds</span><span class="o">(</span><span class="mi">10</span><span class="o">),</span> <span class="nl">WikipediaStats:</span><span class="o">:</span><span class="k">new</span><span class="o">,</span> <span class="k">new</span> <span class="n">WikipediaStatsAggregator</span><span class="o">()))</span>
+<figure class="highlight"><pre><code class="language-java" data-lang="java"><span></span><span class="n">allWikipediaEvents</span><span class="o">.</span><span class="na">map</span><span class="o">(</span><span class="n">WikipediaParser</span><span class="o">::</span><span class="n">parseEvent</span><span class="o">)</span>
+ <span class="o">.</span><span class="na">window</span><span class="o">(</span><span class="n">Windows</span><span class="o">.</span><span class="na">tumblingWindow</span><span class="o">(</span><span class="n">Duration</span><span class="o">.</span><span class="na">ofSeconds</span><span class="o">(</span><span class="mi">10</span><span class="o">),</span> <span class="n">WikipediaStats</span><span class="o">::</span><span class="k">new</span><span class="o">,</span> <span class="k">new</span> <span class="n">WikipediaStatsAggregator</span><span class="o">()))</span>
<span class="o">.</span><span class="na">map</span><span class="o">(</span><span class="k">this</span><span class="o">::</span><span class="n">formatOutput</span><span class="o">)</span>
- <span class="o">.</span><span class="na">sendTo</span><span class="o">(</span><span class="n">wikipediaStats</span><span class="o">);</span></code></pre></div>
+ <span class="o">.</span><span class="na">sendTo</span><span class="o">(</span><span class="n">wikipediaStats</span><span class="o">);</span></code></pre></figure>
<p>Tip: Because the MessageStream type information is preserved in the operator chain, it is often easier to define the OutputStream inline with the <a href="/learn/documentation/latest/api/javadocs/org/apache/samza/operators/MessageStream.html#sendTo-org.apache.samza.operators.OutputStream-">sendTo</a> operator and then refactor it for readability. That way you don’t have to hunt down the types.</p>
@@ -457,38 +454,38 @@ The second parameter is the <a href="/le
<p>We start by defining the store in the config file:</p>
-<div class="highlight"><pre><code class="bash">serializers.registry.string.class<span class="o">=</span>org.apache.samza.serializers.StringSerdeFactory
+<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span></span>serializers.registry.string.class<span class="o">=</span>org.apache.samza.serializers.StringSerdeFactory
serializers.registry.integer.class<span class="o">=</span>org.apache.samza.serializers.IntegerSerdeFactory
stores.wikipedia-stats.factory<span class="o">=</span>org.apache.samza.storage.kv.RocksDbKeyValueStorageEngineFactory
stores.wikipedia-stats.changelog<span class="o">=</span>kafka.wikipedia-stats-changelog
stores.wikipedia-stats.key.serde<span class="o">=</span>string
-stores.wikipedia-stats.msg.serde<span class="o">=</span>integer</code></pre></div>
+stores.wikipedia-stats.msg.serde<span class="o">=</span>integer</code></pre></figure>
<p>These properties declare a <a href="http://rocksdb.org/">RocksDB</a> key-value store named “wikipedia-stats”. The store is replicated to a changelog stream called “wikipedia-stats-changelog” on the <em>kafka</em> system for durability. It uses the <em>string</em> and <em>integer</em> serdes for keys and values respectively.</p>
<p>Next, we add a total count member variable to the <code>WikipediaStats</code> class, and to the <code>WikipediaStatsOutput</code> class:</p>
-<div class="highlight"><pre><code class="java"><span class="kt">int</span> <span class="n">totalEdits</span> <span class="o">=</span> <span class="mi">0</span><span class="o">;</span></code></pre></div>
+<figure class="highlight"><pre><code class="language-java" data-lang="java"><span></span><span class="kt">int</span> <span class="n">totalEdits</span> <span class="o">=</span> <span class="mi">0</span><span class="o">;</span></code></pre></figure>
<p>To use the store in the application, we need to get it from the <a href="/learn/documentation/latest/api/javadocs/org/apache/samza/task/TaskContext.html">TaskContext</a>. Also, since we want to emit the total edit count along with the window edit count, it’s easiest to update both of them in our aggregator. Declare the store as a member variable of the <code>WikipediaStatsAggregator</code> class:</p>
-<div class="highlight"><pre><code class="java"><span class="kd">private</span> <span class="n">KeyValueStore</span><span class="o"><</span><span class="n">String</span><span class="o">,</span> <span class="n">Integer</span><span class="o">></span> <span class="n">store</span><span class="o">;</span></code></pre></div>
+<figure class="highlight"><pre><code class="language-java" data-lang="java"><span></span><span class="kd">private</span> <span class="n">KeyValueStore</span><span class="o"><</span><span class="n">String</span><span class="o">,</span> <span class="n">Integer</span><span class="o">></span> <span class="n">store</span><span class="o">;</span></code></pre></figure>
<p>Then override the <a href="/learn/documentation/latest/api/javadocs/org/apache/samza/operators/functions/InitableFunction.html#init-org.apache.samza.config.Config-org.apache.samza.task.TaskContext-">init</a> method in <code>WikipediaStatsAggregator</code> to initialize the store.</p>
-<div class="highlight"><pre><code class="java"><span class="nd">@Override</span>
+<figure class="highlight"><pre><code class="language-java" data-lang="java"><span></span><span class="nd">@Override</span>
<span class="kd">public</span> <span class="kt">void</span> <span class="nf">init</span><span class="o">(</span><span class="n">Config</span> <span class="n">config</span><span class="o">,</span> <span class="n">TaskContext</span> <span class="n">context</span><span class="o">)</span> <span class="o">{</span>
<span class="n">store</span> <span class="o">=</span> <span class="o">(</span><span class="n">KeyValueStore</span><span class="o"><</span><span class="n">String</span><span class="o">,</span> <span class="n">Integer</span><span class="o">>)</span> <span class="n">context</span><span class="o">.</span><span class="na">getStore</span><span class="o">(</span><span class="s">"wikipedia-stats"</span><span class="o">);</span>
-<span class="o">}</span></code></pre></div>
+<span class="o">}</span></code></pre></figure>
<p>Update and persist the counter in the <code>apply</code> method.</p>
-<div class="highlight"><pre><code class="java"><span class="n">Integer</span> <span class="n">editsAllTime</span> <span class="o">=</span> <span class="n">store</span><span class="o">.</span><span class="na">get</span><span class="o">(</span><span class="s">"count-edits-all-time"</span><span class="o">);</span>
+<figure class="highlight"><pre><code class="language-java" data-lang="java"><span></span><span class="n">Integer</span> <span class="n">editsAllTime</span> <span class="o">=</span> <span class="n">store</span><span class="o">.</span><span class="na">get</span><span class="o">(</span><span class="s">"count-edits-all-time"</span><span class="o">);</span>
<span class="k">if</span> <span class="o">(</span><span class="n">editsAllTime</span> <span class="o">==</span> <span class="kc">null</span><span class="o">)</span> <span class="n">editsAllTime</span> <span class="o">=</span> <span class="mi">0</span><span class="o">;</span>
<span class="n">editsAllTime</span><span class="o">++;</span>
<span class="n">store</span><span class="o">.</span><span class="na">put</span><span class="o">(</span><span class="s">"count-edits-all-time"</span><span class="o">,</span> <span class="n">editsAllTime</span><span class="o">);</span>
-<span class="n">stats</span><span class="o">.</span><span class="na">totalEdits</span> <span class="o">=</span> <span class="n">editsAllTime</span><span class="o">;</span></code></pre></div>
+<span class="n">stats</span><span class="o">.</span><span class="na">totalEdits</span> <span class="o">=</span> <span class="n">editsAllTime</span><span class="o">;</span></code></pre></figure>
<p>Finally, update the <code>MyWikipediaApplication#formatOutput</code> method to include the total counter in its <code>WikipediaStatsOutput</code>.</p>
@@ -498,29 +495,29 @@ stores.wikipedia-stats.msg.serde<span cl
<p>As with the key-value store, we must first define the metrics reporters in the config file.</p>
-<div class="highlight"><pre><code class="bash">metrics.reporters<span class="o">=</span>snapshot,jmx
+<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span></span>metrics.reporters<span class="o">=</span>snapshot,jmx
metrics.reporter.snapshot.class<span class="o">=</span>org.apache.samza.metrics.reporter.MetricsSnapshotReporterFactory
metrics.reporter.snapshot.stream<span class="o">=</span>kafka.metrics
-metrics.reporter.jmx.class<span class="o">=</span>org.apache.samza.metrics.reporter.JmxReporterFactory</code></pre></div>
+metrics.reporter.jmx.class<span class="o">=</span>org.apache.samza.metrics.reporter.JmxReporterFactory</code></pre></figure>
<p>The above properties define 2 metrics reporters. The first emits metrics to a <em>metrics</em> topic on the <em>kafka</em> system. The second reporter emits metrics to JMX.</p>
<p>In the WikipediaStatsAggregator, declare a counter member variable.</p>
-<div class="highlight"><pre><code class="java"><span class="kd">private</span> <span class="n">Counter</span> <span class="n">repeatEdits</span><span class="o">;</span></code></pre></div>
+<figure class="highlight"><pre><code class="language-java" data-lang="java"><span></span><span class="kd">private</span> <span class="n">Counter</span> <span class="n">repeatEdits</span><span class="o">;</span></code></pre></figure>
<p>Then add the following to the <code>WikipediaStatsAggregator#init</code> method to initialize the counter.</p>
-<div class="highlight"><pre><code class="java"><span class="n">repeatEdits</span> <span class="o">=</span> <span class="n">context</span><span class="o">.</span><span class="na">getMetricsRegistry</span><span class="o">().</span><span class="na">newCounter</span><span class="o">(</span><span class="s">"edit-counters"</span><span class="o">,</span> <span class="s">"repeat-edits"</span><span class="o">);</span></code></pre></div>
+<figure class="highlight"><pre><code class="language-java" data-lang="java"><span></span><span class="n">repeatEdits</span> <span class="o">=</span> <span class="n">context</span><span class="o">.</span><span class="na">getMetricsRegistry</span><span class="o">().</span><span class="na">newCounter</span><span class="o">(</span><span class="s">"edit-counters"</span><span class="o">,</span> <span class="s">"repeat-edits"</span><span class="o">);</span></code></pre></figure>
<p>Update and persist the counter from the <code>apply</code> method.</p>
-<div class="highlight"><pre><code class="java"><span class="kt">boolean</span> <span class="n">newTitle</span> <span class="o">=</span> <span class="n">stats</span><span class="o">.</span><span class="na">titles</span><span class="o">.</span><span class="na">add</span><span class="o">((</span><span class="n">String</span><span class="o">)</span> <span class="n">edit</span><span class="o">.</span><span class="na">get</span><span class="o">(</span><span class="s">"title"</span><span class="o">));</span>
+<figure class="highlight"><pre><code class="language-java" data-lang="java"><span></span><span class="kt">boolean</span> <span class="n">newTitle</span> <span class="o">=</span> <span class="n">stats</span><span class="o">.</span><span class="na">titles</span><span class="o">.</span><span class="na">add</span><span class="o">((</span><span class="n">String</span><span class="o">)</span> <span class="n">edit</span><span class="o">.</span><span class="na">get</span><span class="o">(</span><span class="s">"title"</span><span class="o">));</span>
<span class="k">if</span> <span class="o">(!</span><span class="n">newTitle</span><span class="o">)</span> <span class="o">{</span>
<span class="n">repeatEdits</span><span class="o">.</span><span class="na">inc</span><span class="o">();</span>
<span class="n">log</span><span class="o">.</span><span class="na">info</span><span class="o">(</span><span class="s">"Frequent edits for title: {}"</span><span class="o">,</span> <span class="n">edit</span><span class="o">.</span><span class="na">get</span><span class="o">(</span><span class="s">"title"</span><span class="o">));</span>
-<span class="o">}</span></code></pre></div>
+<span class="o">}</span></code></pre></figure>
<h4 id="run-and-view-plan">Run and View Plan</h4>
@@ -548,7 +545,7 @@ metrics.reporter.jmx.class<span class="o
<script>
$( document ).ready(function() {
- if ( $.fn.urlExists( "/learn/tutorials/0.14/hello-samza-high-level-code.html" ) ) {
+ if ( $.fn.urlExists( "/learn/tutorials/0.14/hello-samza-high-level-code" ) ) {
$("#switch-version-button").addClass("fa fa-history masthead-icon");
}
});
Modified: samza/site/learn/tutorials/latest/hello-samza-high-level-yarn.html
URL: http://svn.apache.org/viewvc/samza/site/learn/tutorials/latest/hello-samza-high-level-yarn.html?rev=1832281&r1=1832280&r2=1832281&view=diff
==============================================================================
--- samza/site/learn/tutorials/latest/hello-samza-high-level-yarn.html (original)
+++ samza/site/learn/tutorials/latest/hello-samza-high-level-yarn.html Fri May 25 22:28:38 2018
@@ -44,9 +44,7 @@
<!-- this icon only shows in versioned pages -->
-
-
- <a href="http://samza.apache.org/learn/tutorials/0.14/hello-samza-high-level-yarn.html"><i id="switch-version-button"></i></a>
+ <a href="http://samza.apache.org/learn/tutorials/0.14/hello-samza-high-level-yarn"><i id="switch-version-button"></i></a>
<!-- links for the navigation bar -->
@@ -155,9 +153,8 @@
<p>Check out the hello-samza project:</p>
-<div class="highlight"><pre><code class="bash">git clone https://git.apache.org/samza-hello-samza.git hello-samza
-<span class="nb">cd </span>hello-samza
-git checkout latest</code></pre></div>
+<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span></span>git clone https://git.apache.org/samza-hello-samza.git hello-samza
+<span class="nb">cd</span> hello-samza</code></pre></figure>
<p>This project contains everything you’ll need to run your first Samza application.</p>
@@ -165,7 +162,7 @@ git checkout latest</code></pre></div>
<p>A Samza grid usually comprises three different systems: <a href="http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/YARN.html">YARN</a>, <a href="http://kafka.apache.org/">Kafka</a>, and <a href="http://zookeeper.apache.org/">ZooKeeper</a>. The hello-samza project comes with a script called “grid” to help you setup these systems. Start by running:</p>
-<div class="highlight"><pre><code class="bash">./bin/grid bootstrap</code></pre></div>
+<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span></span>./bin/grid bootstrap</code></pre></figure>
<p>This command will download, install, and start ZooKeeper, Kafka, and YARN. It will also check out the latest version of Samza and build it. All package files will be put in a sub-directory called “deploy” inside hello-samza’s root folder.</p>
@@ -179,19 +176,19 @@ git checkout latest</code></pre></div>
<p>NOTE: if you are building from the latest branch of hello-samza project, make sure that you run the following step from your local Samza project first:</p>
-<div class="highlight"><pre><code class="bash">./gradlew publishToMavenLocal</code></pre></div>
+<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span></span>./gradlew publishToMavenLocal</code></pre></figure>
<p>Then, you can continue w/ the following command in hello-samza project:</p>
-<div class="highlight"><pre><code class="bash">mvn clean package
+<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span></span>mvn clean package
mkdir -p deploy/samza
-tar -xvf ./target/hello-samza-0.14.1-SNAPSHOT-dist.tar.gz -C deploy/samza</code></pre></div>
+tar -xvf ./target/hello-samza-0.14.1-dist.tar.gz -C deploy/samza</code></pre></figure>
<h3 id="run-a-samza-application">Run a Samza Application</h3>
<p>After you’ve built your Samza package, you can start the app on the grid using the run-app.sh script.</p>
-<div class="highlight"><pre><code class="bash">./deploy/samza/bin/run-app.sh --config-factory<span class="o">=</span>org.apache.samza.config.factories.PropertiesConfigFactory --config-path<span class="o">=</span>file://<span class="nv">$PWD</span>/deploy/samza/config/wikipedia-application.properties</code></pre></div>
+<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span></span>./deploy/samza/bin/run-app.sh --config-factory<span class="o">=</span>org.apache.samza.config.factories.PropertiesConfigFactory --config-path<span class="o">=</span>file://<span class="nv">$PWD</span>/deploy/samza/config/wikipedia-application.properties</code></pre></figure>
<p>The app will do all of the following:</p>
@@ -206,14 +203,14 @@ tar -xvf ./target/hello-samza-0.14.1-SNA
<p>Give the job a minute to startup, and then tail the Kafka topic:</p>
-<div class="highlight"><pre><code class="bash">./deploy/kafka/bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic wikipedia-stats</code></pre></div>
+<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span></span>./deploy/kafka/bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic wikipedia-stats</code></pre></figure>
<p>The messages in the stats topic look like this:</p>
-<div class="highlight"><pre><code class="json"><span class="p">{</span><span class="nt">"is-talk"</span><span class="p">:</span><span class="mi">2</span><span class="p">,</span><span class="nt">"bytes-added"</span><span class="p">:</span><span class="mi">5276</span><span class="p">,</span><span class="nt">"edits"</span><span class="p">:</span><span class="mi">13</span><span class="p">,</span><span class="nt">"unique-titles"</span><span class="p">:</span><span class="mi">13</span><span class="p">}</span>
+<figure class="highlight"><pre><code class="language-json" data-lang="json"><span></span><span class="p">{</span><span class="nt">"is-talk"</span><span class="p">:</span><span class="mi">2</span><span class="p">,</span><span class="nt">"bytes-added"</span><span class="p">:</span><span class="mi">5276</span><span class="p">,</span><span class="nt">"edits"</span><span class="p">:</span><span class="mi">13</span><span class="p">,</span><span class="nt">"unique-titles"</span><span class="p">:</span><span class="mi">13</span><span class="p">}</span>
<span class="p">{</span><span class="nt">"is-bot-edit"</span><span class="p">:</span><span class="mi">1</span><span class="p">,</span><span class="nt">"is-talk"</span><span class="p">:</span><span class="mi">3</span><span class="p">,</span><span class="nt">"bytes-added"</span><span class="p">:</span><span class="mi">4211</span><span class="p">,</span><span class="nt">"edits"</span><span class="p">:</span><span class="mi">30</span><span class="p">,</span><span class="nt">"unique-titles"</span><span class="p">:</span><span class="mi">30</span><span class="p">,</span><span class="nt">"is-unpatrolled"</span><span class="p">:</span><span class="mi">1</span><span class="p">,</span><span class="nt">"is-new"</span><span class="p">:</span><span class="mi">2</span><span class="p">,</span><span class="nt">"is-minor"</span><span class="p">:</span><span class="mi">7</span><span class="p">}</span>
<span class="p">{</span><span class="nt">"bytes-added"</span><span class="p">:</span><span class="mi">3180</span><span class="p">,</span><span class="nt">"edits"</span><span class="p">:</span><span class="mi">19</span><span class="p">,</span><span class="nt">"unique-titles"</span><span class="p">:</span><span class="mi">19</span><span class="p">,</span><span class="nt">"is-unpatrolled"</span><span class="p">:</span><span class="mi">1</span><span class="p">,</span><span class="nt">"is-new"</span><span class="p">:</span><span class="mi">1</span><span class="p">,</span><span class="nt">"is-minor"</span><span class="p">:</span><span class="mi">3</span><span class="p">}</span>
-<span class="p">{</span><span class="nt">"bytes-added"</span><span class="p">:</span><span class="mi">2218</span><span class="p">,</span><span class="nt">"edits"</span><span class="p">:</span><span class="mi">18</span><span class="p">,</span><span class="nt">"unique-titles"</span><span class="p">:</span><span class="mi">18</span><span class="p">,</span><span class="nt">"is-unpatrolled"</span><span class="p">:</span><span class="mi">2</span><span class="p">,</span><span class="nt">"is-new"</span><span class="p">:</span><span class="mi">2</span><span class="p">,</span><span class="nt">"is-minor"</span><span class="p">:</span><span class="mi">3</span><span class="p">}</span></code></pre></div>
+<span class="p">{</span><span class="nt">"bytes-added"</span><span class="p">:</span><span class="mi">2218</span><span class="p">,</span><span class="nt">"edits"</span><span class="p">:</span><span class="mi">18</span><span class="p">,</span><span class="nt">"unique-titles"</span><span class="p">:</span><span class="mi">18</span><span class="p">,</span><span class="nt">"is-unpatrolled"</span><span class="p">:</span><span class="mi">2</span><span class="p">,</span><span class="nt">"is-new"</span><span class="p">:</span><span class="mi">2</span><span class="p">,</span><span class="nt">"is-minor"</span><span class="p">:</span><span class="mi">3</span><span class="p">}</span></code></pre></figure>
<p>Pretty neat, right? Now, check out the YARN UI again (<a href="http://localhost:8088">http://localhost:8088</a>). This time around, you’ll see your Samza job is running!</p>
@@ -221,7 +218,7 @@ tar -xvf ./target/hello-samza-0.14.1-SNA
<p>Each application goes through an execution planner and you can visualize the execution plan after starting the job by opening the following file in a browser</p>
-<div class="highlight"><pre><code class="bash">deploy/samza/bin/plan.html</code></pre></div>
+<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span></span>deploy/samza/bin/plan.html</code></pre></figure>
<p>This plan will make more sense after the <a href="hello-samza-high-level-code.html">code walkthrough</a>. For now, just take note that this visualization is available and it is useful for visibility into the structure of the application. For this tutorial, the plan should look something like this:</p>
@@ -231,11 +228,11 @@ tar -xvf ./target/hello-samza-0.14.1-SNA
<p>To shutdown the app, use the same <em>run-app.sh</em> script with an extra <em>–operation=kill</em> argument</p>
-<div class="highlight"><pre><code class="bash">./deploy/samza/bin/run-app.sh --config-factory<span class="o">=</span>org.apache.samza.config.factories.PropertiesConfigFactory --config-path<span class="o">=</span>file://<span class="nv">$PWD</span>/deploy/samza/config/wikipedia-application.properties --operation<span class="o">=</span><span class="nb">kill</span></code></pre></div>
+<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span></span>./deploy/samza/bin/run-app.sh --config-factory<span class="o">=</span>org.apache.samza.config.factories.PropertiesConfigFactory --config-path<span class="o">=</span>file://<span class="nv">$PWD</span>/deploy/samza/config/wikipedia-application.properties --operation<span class="o">=</span><span class="nb">kill</span></code></pre></figure>
<p>After you’re done, you can clean everything up using the same grid script.</p>
-<div class="highlight"><pre><code class="bash">./bin/grid stop all</code></pre></div>
+<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span></span>./bin/grid stop all</code></pre></figure>
<p>Congratulations! You’ve now setup a local grid that includes YARN, Kafka, and ZooKeeper, and run a Samza application on it. Curious how this application was built? See the <a href="hello-samza-high-level-code.html">code walk-through</a>.</p>
@@ -255,7 +252,7 @@ tar -xvf ./target/hello-samza-0.14.1-SNA
<script>
$( document ).ready(function() {
- if ( $.fn.urlExists( "/learn/tutorials/0.14/hello-samza-high-level-yarn.html" ) ) {
+ if ( $.fn.urlExists( "/learn/tutorials/0.14/hello-samza-high-level-yarn" ) ) {
$("#switch-version-button").addClass("fa fa-history masthead-icon");
}
});
Modified: samza/site/learn/tutorials/latest/hello-samza-high-level-zk.html
URL: http://svn.apache.org/viewvc/samza/site/learn/tutorials/latest/hello-samza-high-level-zk.html?rev=1832281&r1=1832280&r2=1832281&view=diff
==============================================================================
--- samza/site/learn/tutorials/latest/hello-samza-high-level-zk.html (original)
+++ samza/site/learn/tutorials/latest/hello-samza-high-level-zk.html Fri May 25 22:28:38 2018
@@ -44,9 +44,7 @@
<!-- this icon only shows in versioned pages -->
-
-
- <a href="http://samza.apache.org/learn/tutorials/0.14/hello-samza-high-level-zk.html"><i id="switch-version-button"></i></a>
+ <a href="http://samza.apache.org/learn/tutorials/0.14/hello-samza-high-level-zk"><i id="switch-version-button"></i></a>
<!-- links for the navigation bar -->
@@ -154,9 +152,8 @@ In this tutorial, we will learn how to r
<p>Let’s get started by cloning the hello-samza project</p>
-<div class="highlight"><pre><code class="bash">git clone https://git.apache.org/samza-hello-samza.git hello-samza
-<span class="nb">cd </span>hello-samza
-git checkout latest</code></pre></div>
+<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span></span>git clone https://git.apache.org/samza-hello-samza.git hello-samza
+<span class="nb">cd</span> hello-samza</code></pre></figure>
<p>The project comes up with numerous examples and for this tutorial, we will pick the Wikipedia application.</p>
@@ -164,7 +161,7 @@ git checkout latest</code></pre></div>
<p>For our Wikipedia application, we require two systems: <a href="http://kafka.apache.org/">Kafka</a> and <a href="http://zookeeper.apache.org/">ZooKeeper</a>. The hello-samza project comes with a script called “grid” to help with the environment setup</p>
-<div class="highlight"><pre><code class="bash">./bin/grid standalone</code></pre></div>
+<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span></span>./bin/grid standalone</code></pre></figure>
<p>This command will download, install, and start ZooKeeper and Kafka. It will also check out the latest version of Samza and build it. All package files will be put in a sub-directory called “deploy” inside hello-samza’s root folder.</p>
@@ -174,13 +171,13 @@ git checkout latest</code></pre></div>
<p>NOTE: if you are building from the latest branch of hello-samza project and want to use your local copy of samza, make sure that you run the following step from your local Samza project first</p>
-<div class="highlight"><pre><code class="bash">./gradlew publishToMavenLocal</code></pre></div>
+<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span></span>./gradlew publishToMavenLocal</code></pre></figure>
<p>With the environment setup complete, let us move on to building the hello-samza project. Execute the following commands:</p>
-<div class="highlight"><pre><code class="bash">mvn clean package
+<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span></span>mvn clean package
mkdir -p deploy/samza
-tar -xvf ./target/hello-samza-0.14.1-SNAPSHOT-dist.tar.gz -C deploy/samza</code></pre></div>
+tar -xvf ./target/hello-samza-0.14.1-dist.tar.gz -C deploy/samza</code></pre></figure>
<p>We are now all set to deploy the application locally.</p>
@@ -188,7 +185,7 @@ tar -xvf ./target/hello-samza-0.14.1-SNA
<p>In order to run the application, we will use the <em>run-wikipedia-zk-application</em> script.</p>
-<div class="highlight"><pre><code class="bash">./deploy/samza/bin/run-wikipedia-zk-application.sh</code></pre></div>
+<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span></span>./deploy/samza/bin/run-wikipedia-zk-application.sh</code></pre></figure>
<p>The above command executes the helper script which invokes the <em>WikipediaZkLocalApplication</em> main class with the appropriate job configurations as command line arguments. The main class is an application wrapper
that initializes the application and passes it to the local runner for execution. It is blocking and waits for the <em>LocalApplicationRunner</em> to finish.</p>
@@ -198,14 +195,14 @@ and uses the <em>LocalApplicationRunner<
<p>Getting back to our example, the application consumes a feed of real-time edits from Wikipedia, and produces them to a Kafka topic called “wikipedia-stats”. Give the job a minute to startup, and then tail the Kafka topic. To do so, run the following command:</p>
-<div class="highlight"><pre><code class="bash">./deploy/kafka/bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic wikipedia-stats</code></pre></div>
+<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span></span>./deploy/kafka/bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic wikipedia-stats</code></pre></figure>
<p>The messages in the stats topic should look like the sample below:</p>
-<div class="highlight"><pre><code class="json"><span class="p">{</span><span class="nt">"is-talk"</span><span class="p">:</span><span class="mi">2</span><span class="p">,</span><span class="nt">"bytes-added"</span><span class="p">:</span><span class="mi">5276</span><span class="p">,</span><span class="nt">"edits"</span><span class="p">:</span><span class="mi">13</span><span class="p">,</span><span class="nt">"unique-titles"</span><span class="p">:</span><span class="mi">13</span><span class="p">}</span>
+<figure class="highlight"><pre><code class="language-json" data-lang="json"><span></span><span class="p">{</span><span class="nt">"is-talk"</span><span class="p">:</span><span class="mi">2</span><span class="p">,</span><span class="nt">"bytes-added"</span><span class="p">:</span><span class="mi">5276</span><span class="p">,</span><span class="nt">"edits"</span><span class="p">:</span><span class="mi">13</span><span class="p">,</span><span class="nt">"unique-titles"</span><span class="p">:</span><span class="mi">13</span><span class="p">}</span>
<span class="p">{</span><span class="nt">"is-bot-edit"</span><span class="p">:</span><span class="mi">1</span><span class="p">,</span><span class="nt">"is-talk"</span><span class="p">:</span><span class="mi">3</span><span class="p">,</span><span class="nt">"bytes-added"</span><span class="p">:</span><span class="mi">4211</span><span class="p">,</span><span class="nt">"edits"</span><span class="p">:</span><span class="mi">30</span><span class="p">,</span><span class="nt">"unique-titles"</span><span class="p">:</span><span class="mi">30</span><span class="p">,</span><span class="nt">"is-unpatrolled"</span><span class="p">:</span><span class="mi">1</span><span class="p">,</span><span class="nt">"is-new"</span><span class="p">:</span><span class="mi">2</span><span class="p">,</span><span class="nt">"is-minor"</span><span class="p">:</span><span class="mi">7</span><span class="p">}</span>
<span class="p">{</span><span class="nt">"bytes-added"</span><span class="p">:</span><span class="mi">3180</span><span class="p">,</span><span class="nt">"edits"</span><span class="p">:</span><span class="mi">19</span><span class="p">,</span><span class="nt">"unique-titles"</span><span class="p">:</span><span class="mi">19</span><span class="p">,</span><span class="nt">"is-unpatrolled"</span><span class="p">:</span><span class="mi">1</span><span class="p">,</span><span class="nt">"is-new"</span><span class="p">:</span><span class="mi">1</span><span class="p">,</span><span class="nt">"is-minor"</span><span class="p">:</span><span class="mi">3</span><span class="p">}</span>
-<span class="p">{</span><span class="nt">"bytes-added"</span><span class="p">:</span><span class="mi">2218</span><span class="p">,</span><span class="nt">"edits"</span><span class="p">:</span><span class="mi">18</span><span class="p">,</span><span class="nt">"unique-titles"</span><span class="p">:</span><span class="mi">18</span><span class="p">,</span><span class="nt">"is-unpatrolled"</span><span class="p">:</span><span class="mi">2</span><span class="p">,</span><span class="nt">"is-new"</span><span class="p">:</span><span class="mi">2</span><span class="p">,</span><span class="nt">"is-minor"</span><span class="p">:</span><span class="mi">3</span><span class="p">}</span></code></pre></div>
+<span class="p">{</span><span class="nt">"bytes-added"</span><span class="p">:</span><span class="mi">2218</span><span class="p">,</span><span class="nt">"edits"</span><span class="p">:</span><span class="mi">18</span><span class="p">,</span><span class="nt">"unique-titles"</span><span class="p">:</span><span class="mi">18</span><span class="p">,</span><span class="nt">"is-unpatrolled"</span><span class="p">:</span><span class="mi">2</span><span class="p">,</span><span class="nt">"is-new"</span><span class="p">:</span><span class="mi">2</span><span class="p">,</span><span class="nt">"is-minor"</span><span class="p">:</span><span class="mi">3</span><span class="p">}</span></code></pre></figure>
<p>Excellent! Now that the job is running, open the <em>plan.html</em> file under <em>deploy/samza/bin</em> directory to take a look at the execution plan for the Wikipedia application.
The execution plan is a colorful graphic representing various stages of your application and how they are connected. Here is a sample plan visualization:</p>
@@ -217,7 +214,7 @@ The execution plan is a colorful graphic
<p>The Wikipedia application can be shutdown by terminating the <em>run-wikipedia-zk-application</em> script.
We can use the <em>grid</em> script to tear down the local environment (<a href="http://kafka.apache.org/">Kafka</a> and <a href="http://zookeeper.apache.org/">Zookeeper</a>).</p>
-<div class="highlight"><pre><code class="bash">bin/grid stop all</code></pre></div>
+<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span></span>bin/grid stop all</code></pre></figure>
<p>Congratulations! You’ve now successfully run a Samza application using ZooKeeper deployment model. Next up, check out the <a href="/startup/preview/">deployment-models</a> and <a href="/startup/preview.html">high level API</a> pages.</p>
@@ -237,7 +234,7 @@ We can use the <em>grid</em> script to t
<script>
$( document ).ready(function() {
- if ( $.fn.urlExists( "/learn/tutorials/0.14/hello-samza-high-level-zk.html" ) ) {
+ if ( $.fn.urlExists( "/learn/tutorials/0.14/hello-samza-high-level-zk" ) ) {
$("#switch-version-button").addClass("fa fa-history masthead-icon");
}
});
Modified: samza/site/learn/tutorials/latest/index.html
URL: http://svn.apache.org/viewvc/samza/site/learn/tutorials/latest/index.html?rev=1832281&r1=1832280&r2=1832281&view=diff
==============================================================================
--- samza/site/learn/tutorials/latest/index.html (original)
+++ samza/site/learn/tutorials/latest/index.html Fri May 25 22:28:38 2018
@@ -44,9 +44,7 @@
<!-- this icon only shows in versioned pages -->
-
-
- <a href="http://samza.apache.org/learn/tutorials/0.14/index.html"><i id="switch-version-button"></i></a>
+ <a href="http://samza.apache.org/learn/tutorials/0.14/"><i id="switch-version-button"></i></a>
<!-- links for the navigation bar -->
@@ -200,7 +198,7 @@
<script>
$( document ).ready(function() {
- if ( $.fn.urlExists( "/learn/tutorials/0.14/index.html" ) ) {
+ if ( $.fn.urlExists( "/learn/tutorials/0.14/" ) ) {
$("#switch-version-button").addClass("fa fa-history masthead-icon");
}
});
Modified: samza/site/learn/tutorials/latest/remote-debugging-samza.html
URL: http://svn.apache.org/viewvc/samza/site/learn/tutorials/latest/remote-debugging-samza.html?rev=1832281&r1=1832280&r2=1832281&view=diff
==============================================================================
--- samza/site/learn/tutorials/latest/remote-debugging-samza.html (original)
+++ samza/site/learn/tutorials/latest/remote-debugging-samza.html Fri May 25 22:28:38 2018
@@ -44,9 +44,7 @@
<!-- this icon only shows in versioned pages -->
-
-
- <a href="http://samza.apache.org/learn/tutorials/0.14/remote-debugging-samza.html"><i id="switch-version-button"></i></a>
+ <a href="http://samza.apache.org/learn/tutorials/0.14/remote-debugging-samza"><i id="switch-version-button"></i></a>
<!-- links for the navigation bar -->
@@ -153,22 +151,22 @@
<p>Start by checking out Samza, so we have access to the source.</p>
-<div class="highlight"><pre><code class="bash">git clone http://git-wip-us.apache.org/repos/asf/samza.git</code></pre></div>
+<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span></span>git clone http://git-wip-us.apache.org/repos/asf/samza.git</code></pre></figure>
<p>Next, grab hello-samza.</p>
-<div class="highlight"><pre><code class="bash">git clone https://git.apache.org/samza-hello-samza.git</code></pre></div>
+<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span></span>git clone https://git.apache.org/samza-hello-samza.git</code></pre></figure>
<h3 id="setup-the-environment">Setup the Environment</h3>
<p>Now, let’s setup the Eclipse project files.</p>
-<div class="highlight"><pre><code class="bash"><span class="nb">cd </span>samza
-./gradlew eclipse</code></pre></div>
+<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span></span><span class="nb">cd</span> samza
+./gradlew eclipse</code></pre></figure>
<p>Let’s also release Samza to Maven’s local repository, so hello-samza has access to the JARs that it needs.</p>
-<div class="highlight"><pre><code class="bash">./gradlew -PscalaVersion<span class="o">=</span>2.10 clean publishToMavenLocal</code></pre></div>
+<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span></span>./gradlew -PscalaVersion<span class="o">=</span><span class="m">2</span>.10 clean publishToMavenLocal</code></pre></figure>
<p>Next, open Eclipse, and import the Samza source code into your workspace: “File” > “Import” > “Existing Projects into Workspace” > “Browse”. Select ‘samza’ folder, and hit ‘finish’.</p>
@@ -176,7 +174,7 @@
<p>Now, go back to the hello-samza project, and edit ./src/main/config/wikipedia-feed.properties to add the following line:</p>
-<div class="highlight"><pre><code class="jproperties"><span class="na">task.opts</span><span class="o">=</span><span class="s">-agentlib:jdwp=transport=dt_socket,address=localhost:9009,server=y,suspend=y</span></code></pre></div>
+<figure class="highlight"><pre><code class="language-jproperties" data-lang="jproperties"><span></span><span class="na">task.opts</span><span class="o">=</span><span class="s">-agentlib:jdwp=transport=dt_socket,address=localhost:9009,server=y,suspend=y</span></code></pre></figure>
<p>The <a href="../../documentation/latest/jobs/configuration-table.html">task.opts</a> configuration parameter is a way to override Java parameters at runtime for your Samza containers. In this example, we’re setting the agentlib parameter to enable remote debugging on localhost, port 9009. In a more realistic environment, you might also set Java heap settings (-Xmx, -Xms, etc), as well as garbage collection and logging settings.</p>
@@ -186,18 +184,18 @@
<p>Now that the Samza job has been setup to enable remote debugging when a Samza container starts, let’s start the ZooKeeper, Kafka, and YARN.</p>
-<div class="highlight"><pre><code class="bash">bin/grid</code></pre></div>
+<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span></span>bin/grid</code></pre></figure>
<p>If you get a complaint that JAVA_HOME is not set, then you’ll need to set it. This can be done on OSX by running:</p>
-<div class="highlight"><pre><code class="bash"><span class="nb">export </span><span class="nv">JAVA_HOME</span><span class="o">=</span><span class="k">$(</span>/usr/libexec/java_home<span class="k">)</span></code></pre></div>
+<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span></span><span class="nb">export</span> <span class="nv">JAVA_HOME</span><span class="o">=</span><span class="k">$(</span>/usr/libexec/java_home<span class="k">)</span></code></pre></figure>
<p>Once the grid starts, you can start the wikipedia-feed Samza job.</p>
-<div class="highlight"><pre><code class="bash">mvn clean package
+<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span></span>mvn clean package
mkdir -p deploy/samza
-tar -xvf ./target/hello-samza-0.14.0-dist.tar.gz -C deploy/samza
-deploy/samza/bin/run-job.sh --config-factory<span class="o">=</span>org.apache.samza.config.factories.PropertiesConfigFactory --config-path<span class="o">=</span>file://<span class="nv">$PWD</span>/deploy/samza/config/wikipedia-feed.properties</code></pre></div>
+tar -xvf ./target/hello-samza-0.14.1-dist.tar.gz -C deploy/samza
+deploy/samza/bin/run-job.sh --config-factory<span class="o">=</span>org.apache.samza.config.factories.PropertiesConfigFactory --config-path<span class="o">=</span>file://<span class="nv">$PWD</span>/deploy/samza/config/wikipedia-feed.properties</code></pre></figure>
<p>When the wikipedia-feed job starts up, a single Samza container will be created to process all incoming messages. This is the container that we’ll want to connect to from the remote debugger.</p>
@@ -227,7 +225,7 @@ deploy/samza/bin/run-job.sh --config-fac
<script>
$( document ).ready(function() {
- if ( $.fn.urlExists( "/learn/tutorials/0.14/remote-debugging-samza.html" ) ) {
+ if ( $.fn.urlExists( "/learn/tutorials/0.14/remote-debugging-samza" ) ) {
$("#switch-version-button").addClass("fa fa-history masthead-icon");
}
});
Modified: samza/site/learn/tutorials/latest/run-hello-samza-without-internet.html
URL: http://svn.apache.org/viewvc/samza/site/learn/tutorials/latest/run-hello-samza-without-internet.html?rev=1832281&r1=1832280&r2=1832281&view=diff
==============================================================================
--- samza/site/learn/tutorials/latest/run-hello-samza-without-internet.html (original)
+++ samza/site/learn/tutorials/latest/run-hello-samza-without-internet.html Fri May 25 22:28:38 2018
@@ -44,9 +44,7 @@
<!-- this icon only shows in versioned pages -->
-
-
- <a href="http://samza.apache.org/learn/tutorials/0.14/run-hello-samza-without-internet.html"><i id="switch-version-button"></i></a>
+ <a href="http://samza.apache.org/learn/tutorials/0.14/run-hello-samza-without-internet"><i id="switch-version-button"></i></a>
<!-- links for the navigation bar -->
@@ -153,10 +151,10 @@
<p>Ping irc.wikimedia.org. Sometimes the firewall in your company blocks this service.</p>
-<div class="highlight"><pre><code class="bash">telnet irc.wikimedia.org 6667</code></pre></div>
+<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span></span>telnet irc.wikimedia.org <span class="m">6667</span></code></pre></figure>
<p>You should see something like this:</p>
-<div class="highlight"><pre><code class="language-text" data-lang="text">Trying 208.80.152.178...
+<div class="highlight"><pre><code class="language-text" data-lang="text"><span></span>Trying 208.80.152.178...
Connected to ekrem.wikimedia.org.
Escape character is '^]'.
NOTICE AUTH :*** Processing connection to irc.pmtpa.wikimedia.org
@@ -170,15 +168,15 @@ NOTICE AUTH :*** Found your hostname
<p>We provide an alternative to get wikipedia feed data. Instead of running</p>
-<div class="highlight"><pre><code class="bash">deploy/samza/bin/run-job.sh --config-factory<span class="o">=</span>org.apache.samza.config.factories.PropertiesConfigFactory --config-path<span class="o">=</span>file://<span class="nv">$PWD</span>/deploy/samza/config/wikipedia-feed.properties</code></pre></div>
+<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span></span>deploy/samza/bin/run-job.sh --config-factory<span class="o">=</span>org.apache.samza.config.factories.PropertiesConfigFactory --config-path<span class="o">=</span>file://<span class="nv">$PWD</span>/deploy/samza/config/wikipedia-feed.properties</code></pre></figure>
<p>You will run</p>
-<div class="highlight"><pre><code class="bash">bin/produce-wikipedia-raw-data.sh</code></pre></div>
+<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span></span>bin/produce-wikipedia-raw-data.sh</code></pre></figure>
<p>This script will read wikipedia feed data from local file and produce them to the Kafka broker. By default, it produces to localhost:9092 as the Kafka broker and uses localhost:2181 as zookeeper. You can overwrite them:</p>
-<div class="highlight"><pre><code class="bash">bin/produce-wikipedia-raw-data.sh -b yourKafkaBrokerAddress -z yourZookeeperAddress</code></pre></div>
+<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span></span>bin/produce-wikipedia-raw-data.sh -b yourKafkaBrokerAddress -z yourZookeeperAddress</code></pre></figure>
<p>Now you can go back to Generate Wikipedia Statistics section in <a href="../../../startup/hello-samza/latest/">Hello Samza</a> and follow the remaining steps.</p>
@@ -186,7 +184,7 @@ NOTICE AUTH :*** Found your hostname
<p>The goal of</p>
-<div class="highlight"><pre><code class="bash">deploy/samza/bin/run-job.sh --config-factory<span class="o">=</span>org.apache.samza.config.factories.PropertiesConfigFactory --config-path<span class="o">=</span>file://<span class="nv">$PWD</span>/deploy/samza/config/wikipedia-feed.properties</code></pre></div>
+<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span></span>deploy/samza/bin/run-job.sh --config-factory<span class="o">=</span>org.apache.samza.config.factories.PropertiesConfigFactory --config-path<span class="o">=</span>file://<span class="nv">$PWD</span>/deploy/samza/config/wikipedia-feed.properties</code></pre></figure>
<p>is to deploy a Samza job which listens to wikipedia API, receives the feed in realtime and produces the feed to the Kafka topic wikipedia-raw. The alternative in this tutorial is reading local wikipedia feed in an infinite loop and producing the data to Kafka wikipedia-raw. The follow-up job, wikipedia-parser is getting data from Kafka topic wikipedia-raw, so as long as we have correct data in Kafka topic wikipedia-raw, we are fine. All Samza jobs are connected by the Kafka and do not depend on each other.</p>
@@ -206,7 +204,7 @@ NOTICE AUTH :*** Found your hostname
<script>
$( document ).ready(function() {
- if ( $.fn.urlExists( "/learn/tutorials/0.14/run-hello-samza-without-internet.html" ) ) {
+ if ( $.fn.urlExists( "/learn/tutorials/0.14/run-hello-samza-without-internet" ) ) {
$("#switch-version-button").addClass("fa fa-history masthead-icon");
}
});