You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@samza.apache.org by cr...@apache.org on 2013/08/13 04:10:46 UTC
svn commit: r1513322 [2/2] - in /incubator/samza/site: contribute/
learn/documentation/0.7.0/api/ learn/documentation/0.7.0/api/javadocs/
learn/documentation/0.7.0/api/javadocs/org/apache/samza/
learn/documentation/0.7.0/api/javadocs/org/apache/samza/c...
Modified: incubator/samza/site/learn/documentation/0.7.0/api/javadocs/org/apache/samza/util/BlockingEnvelopeMap.html
URL: http://svn.apache.org/viewvc/incubator/samza/site/learn/documentation/0.7.0/api/javadocs/org/apache/samza/util/BlockingEnvelopeMap.html?rev=1513322&r1=1513321&r2=1513322&view=diff
==============================================================================
--- incubator/samza/site/learn/documentation/0.7.0/api/javadocs/org/apache/samza/util/BlockingEnvelopeMap.html (original)
+++ incubator/samza/site/learn/documentation/0.7.0/api/javadocs/org/apache/samza/util/BlockingEnvelopeMap.html Tue Aug 13 02:10:44 2013
@@ -2,7 +2,7 @@
<!--NewPage-->
<HTML>
<HEAD>
-<!-- Generated by javadoc (build 1.6.0_51) on Mon Aug 12 09:56:51 PDT 2013 -->
+<!-- Generated by javadoc (build 1.6.0_51) on Mon Aug 12 19:09:46 PDT 2013 -->
<TITLE>
BlockingEnvelopeMap (samza-api 0.7.0 API)
</TITLE>
Modified: incubator/samza/site/learn/documentation/0.7.0/api/javadocs/org/apache/samza/util/Clock.html
URL: http://svn.apache.org/viewvc/incubator/samza/site/learn/documentation/0.7.0/api/javadocs/org/apache/samza/util/Clock.html?rev=1513322&r1=1513321&r2=1513322&view=diff
==============================================================================
--- incubator/samza/site/learn/documentation/0.7.0/api/javadocs/org/apache/samza/util/Clock.html (original)
+++ incubator/samza/site/learn/documentation/0.7.0/api/javadocs/org/apache/samza/util/Clock.html Tue Aug 13 02:10:44 2013
@@ -2,7 +2,7 @@
<!--NewPage-->
<HTML>
<HEAD>
-<!-- Generated by javadoc (build 1.6.0_51) on Mon Aug 12 09:56:51 PDT 2013 -->
+<!-- Generated by javadoc (build 1.6.0_51) on Mon Aug 12 19:09:46 PDT 2013 -->
<TITLE>
Clock (samza-api 0.7.0 API)
</TITLE>
Modified: incubator/samza/site/learn/documentation/0.7.0/api/javadocs/org/apache/samza/util/NoOpMetricsRegistry.html
URL: http://svn.apache.org/viewvc/incubator/samza/site/learn/documentation/0.7.0/api/javadocs/org/apache/samza/util/NoOpMetricsRegistry.html?rev=1513322&r1=1513321&r2=1513322&view=diff
==============================================================================
--- incubator/samza/site/learn/documentation/0.7.0/api/javadocs/org/apache/samza/util/NoOpMetricsRegistry.html (original)
+++ incubator/samza/site/learn/documentation/0.7.0/api/javadocs/org/apache/samza/util/NoOpMetricsRegistry.html Tue Aug 13 02:10:44 2013
@@ -2,7 +2,7 @@
<!--NewPage-->
<HTML>
<HEAD>
-<!-- Generated by javadoc (build 1.6.0_51) on Mon Aug 12 09:56:51 PDT 2013 -->
+<!-- Generated by javadoc (build 1.6.0_51) on Mon Aug 12 19:09:46 PDT 2013 -->
<TITLE>
NoOpMetricsRegistry (samza-api 0.7.0 API)
</TITLE>
Modified: incubator/samza/site/learn/documentation/0.7.0/api/javadocs/org/apache/samza/util/SinglePartitionSystemAdmin.html
URL: http://svn.apache.org/viewvc/incubator/samza/site/learn/documentation/0.7.0/api/javadocs/org/apache/samza/util/SinglePartitionSystemAdmin.html?rev=1513322&r1=1513321&r2=1513322&view=diff
==============================================================================
--- incubator/samza/site/learn/documentation/0.7.0/api/javadocs/org/apache/samza/util/SinglePartitionSystemAdmin.html (original)
+++ incubator/samza/site/learn/documentation/0.7.0/api/javadocs/org/apache/samza/util/SinglePartitionSystemAdmin.html Tue Aug 13 02:10:44 2013
@@ -2,7 +2,7 @@
<!--NewPage-->
<HTML>
<HEAD>
-<!-- Generated by javadoc (build 1.6.0_51) on Mon Aug 12 09:56:51 PDT 2013 -->
+<!-- Generated by javadoc (build 1.6.0_51) on Mon Aug 12 19:09:46 PDT 2013 -->
<TITLE>
SinglePartitionSystemAdmin (samza-api 0.7.0 API)
</TITLE>
Modified: incubator/samza/site/learn/documentation/0.7.0/api/javadocs/org/apache/samza/util/package-frame.html
URL: http://svn.apache.org/viewvc/incubator/samza/site/learn/documentation/0.7.0/api/javadocs/org/apache/samza/util/package-frame.html?rev=1513322&r1=1513321&r2=1513322&view=diff
==============================================================================
--- incubator/samza/site/learn/documentation/0.7.0/api/javadocs/org/apache/samza/util/package-frame.html (original)
+++ incubator/samza/site/learn/documentation/0.7.0/api/javadocs/org/apache/samza/util/package-frame.html Tue Aug 13 02:10:44 2013
@@ -2,7 +2,7 @@
<!--NewPage-->
<HTML>
<HEAD>
-<!-- Generated by javadoc (build 1.6.0_51) on Mon Aug 12 09:56:51 PDT 2013 -->
+<!-- Generated by javadoc (build 1.6.0_51) on Mon Aug 12 19:09:47 PDT 2013 -->
<TITLE>
org.apache.samza.util (samza-api 0.7.0 API)
</TITLE>
Modified: incubator/samza/site/learn/documentation/0.7.0/api/javadocs/org/apache/samza/util/package-summary.html
URL: http://svn.apache.org/viewvc/incubator/samza/site/learn/documentation/0.7.0/api/javadocs/org/apache/samza/util/package-summary.html?rev=1513322&r1=1513321&r2=1513322&view=diff
==============================================================================
--- incubator/samza/site/learn/documentation/0.7.0/api/javadocs/org/apache/samza/util/package-summary.html (original)
+++ incubator/samza/site/learn/documentation/0.7.0/api/javadocs/org/apache/samza/util/package-summary.html Tue Aug 13 02:10:44 2013
@@ -2,7 +2,7 @@
<!--NewPage-->
<HTML>
<HEAD>
-<!-- Generated by javadoc (build 1.6.0_51) on Mon Aug 12 09:56:51 PDT 2013 -->
+<!-- Generated by javadoc (build 1.6.0_51) on Mon Aug 12 19:09:47 PDT 2013 -->
<TITLE>
org.apache.samza.util (samza-api 0.7.0 API)
</TITLE>
Modified: incubator/samza/site/learn/documentation/0.7.0/api/javadocs/org/apache/samza/util/package-tree.html
URL: http://svn.apache.org/viewvc/incubator/samza/site/learn/documentation/0.7.0/api/javadocs/org/apache/samza/util/package-tree.html?rev=1513322&r1=1513321&r2=1513322&view=diff
==============================================================================
--- incubator/samza/site/learn/documentation/0.7.0/api/javadocs/org/apache/samza/util/package-tree.html (original)
+++ incubator/samza/site/learn/documentation/0.7.0/api/javadocs/org/apache/samza/util/package-tree.html Tue Aug 13 02:10:44 2013
@@ -2,7 +2,7 @@
<!--NewPage-->
<HTML>
<HEAD>
-<!-- Generated by javadoc (build 1.6.0_51) on Mon Aug 12 09:56:51 PDT 2013 -->
+<!-- Generated by javadoc (build 1.6.0_51) on Mon Aug 12 19:09:47 PDT 2013 -->
<TITLE>
org.apache.samza.util Class Hierarchy (samza-api 0.7.0 API)
</TITLE>
Modified: incubator/samza/site/learn/documentation/0.7.0/api/javadocs/overview-frame.html
URL: http://svn.apache.org/viewvc/incubator/samza/site/learn/documentation/0.7.0/api/javadocs/overview-frame.html?rev=1513322&r1=1513321&r2=1513322&view=diff
==============================================================================
--- incubator/samza/site/learn/documentation/0.7.0/api/javadocs/overview-frame.html (original)
+++ incubator/samza/site/learn/documentation/0.7.0/api/javadocs/overview-frame.html Tue Aug 13 02:10:44 2013
@@ -2,7 +2,7 @@
<!--NewPage-->
<HTML>
<HEAD>
-<!-- Generated by javadoc (build 1.6.0_51) on Mon Aug 12 09:56:51 PDT 2013 -->
+<!-- Generated by javadoc (build 1.6.0_51) on Mon Aug 12 19:09:47 PDT 2013 -->
<TITLE>
Overview List (samza-api 0.7.0 API)
</TITLE>
Modified: incubator/samza/site/learn/documentation/0.7.0/api/javadocs/overview-summary.html
URL: http://svn.apache.org/viewvc/incubator/samza/site/learn/documentation/0.7.0/api/javadocs/overview-summary.html?rev=1513322&r1=1513321&r2=1513322&view=diff
==============================================================================
--- incubator/samza/site/learn/documentation/0.7.0/api/javadocs/overview-summary.html (original)
+++ incubator/samza/site/learn/documentation/0.7.0/api/javadocs/overview-summary.html Tue Aug 13 02:10:44 2013
@@ -2,7 +2,7 @@
<!--NewPage-->
<HTML>
<HEAD>
-<!-- Generated by javadoc (build 1.6.0_51) on Mon Aug 12 09:56:51 PDT 2013 -->
+<!-- Generated by javadoc (build 1.6.0_51) on Mon Aug 12 19:09:47 PDT 2013 -->
<TITLE>
Overview (samza-api 0.7.0 API)
</TITLE>
Modified: incubator/samza/site/learn/documentation/0.7.0/api/javadocs/overview-tree.html
URL: http://svn.apache.org/viewvc/incubator/samza/site/learn/documentation/0.7.0/api/javadocs/overview-tree.html?rev=1513322&r1=1513321&r2=1513322&view=diff
==============================================================================
--- incubator/samza/site/learn/documentation/0.7.0/api/javadocs/overview-tree.html (original)
+++ incubator/samza/site/learn/documentation/0.7.0/api/javadocs/overview-tree.html Tue Aug 13 02:10:44 2013
@@ -2,7 +2,7 @@
<!--NewPage-->
<HTML>
<HEAD>
-<!-- Generated by javadoc (build 1.6.0_51) on Mon Aug 12 09:56:51 PDT 2013 -->
+<!-- Generated by javadoc (build 1.6.0_51) on Mon Aug 12 19:09:47 PDT 2013 -->
<TITLE>
Class Hierarchy (samza-api 0.7.0 API)
</TITLE>
Modified: incubator/samza/site/learn/documentation/0.7.0/api/javadocs/serialized-form.html
URL: http://svn.apache.org/viewvc/incubator/samza/site/learn/documentation/0.7.0/api/javadocs/serialized-form.html?rev=1513322&r1=1513321&r2=1513322&view=diff
==============================================================================
--- incubator/samza/site/learn/documentation/0.7.0/api/javadocs/serialized-form.html (original)
+++ incubator/samza/site/learn/documentation/0.7.0/api/javadocs/serialized-form.html Tue Aug 13 02:10:44 2013
@@ -2,7 +2,7 @@
<!--NewPage-->
<HTML>
<HEAD>
-<!-- Generated by javadoc (build 1.6.0_51) on Mon Aug 12 09:56:51 PDT 2013 -->
+<!-- Generated by javadoc (build 1.6.0_51) on Mon Aug 12 19:09:47 PDT 2013 -->
<TITLE>
Serialized Form (samza-api 0.7.0 API)
</TITLE>
Modified: incubator/samza/site/learn/documentation/0.7.0/api/overview.html
URL: http://svn.apache.org/viewvc/incubator/samza/site/learn/documentation/0.7.0/api/overview.html?rev=1513322&r1=1513321&r2=1513322&view=diff
==============================================================================
--- incubator/samza/site/learn/documentation/0.7.0/api/overview.html (original)
+++ incubator/samza/site/learn/documentation/0.7.0/api/overview.html Tue Aug 13 02:10:44 2013
@@ -66,13 +66,13 @@
<h2>API Overview</h2>
<p>When writing a stream processor for Samza, you must implement the StreamTask interface:</p>
-<div class="highlight"><pre><code class="text language-text" data-lang="text">/** User processing tasks implement this. */
+<div class="highlight"><pre><code class="text language-text">/** User processing tasks implement this. */
public interface StreamTask {
void process(MessageEnvelope envelope, MessageCollector collector, TaskCoordinator coordinator);
}
</code></pre></div>
<p>When Samza runs your task, the process method will be called once for each message that Samza receives from your task's input streams. The envelope contains three things of importance: the message, the key), and the stream that the message came from:</p>
-<div class="highlight"><pre><code class="text language-text" data-lang="text">/** This class is given to a StreamTask once for each message that it receives. */
+<div class="highlight"><pre><code class="text language-text">/** This class is given to a StreamTask once for each message that it receives. */
public interface MessageEnvelope {
/** A deserialized message. */
<M> M getMessage();
@@ -85,7 +85,7 @@ public interface MessageEnvelope {
}
</code></pre></div>
<p>Notice that the getStream() method returns a Stream object, not a String, as you might expect. This is because a Samza Stream actually consists of a name, a system, and a stream. The name is what you call the stream in your Samza configuration file. The system is the name of the cluster that the stream came from (e.g. kafka-aggreate-tracking, databus, etc). The system name is also defined in your Samza configuration file. Lastly, the actual stream is available. For Kafka, this would be the Kafka topic's name.</p>
-<div class="highlight"><pre><code class="text language-text" data-lang="text">/** A name/system/stream tuple that represents a Samza stream. */
+<div class="highlight"><pre><code class="text language-text">/** A name/system/stream tuple that represents a Samza stream. */
public class Stream {
/** The name of the stream, if the stream is defined in a Samza job's
configuration. If not, this is null. */
@@ -100,7 +100,7 @@ public class Stream {
}
</code></pre></div>
<p>To make this a bit clearer, let me show you an example. A Samza job's configuration might have:</p>
-<div class="highlight"><pre><code class="text language-text" data-lang="text"># the stream
+<div class="highlight"><pre><code class="text language-text"># the stream
streams.page-view-event.stream=PageViewEvent
streams.page-view-event.system=kafka
streams.page-view-event.serde=json
@@ -114,7 +114,7 @@ systems.kafka.samza.producer.factory=sam
<p>I this example, getName would return page-view-event, getSystem would return kafka, and getStream would return PageViewEvent. If you've got more than one input stream feeding into your StreamTask, you can use the getStream() object to determine what kind of message you've received.</p>
<p>What about sending messages? If you take a look at the process() method in StreamTask, you'll see that you get a MessageCollector.</p>
-<div class="highlight"><pre><code class="text language-text" data-lang="text">/** When a task wishes to send a message, it uses this class. */
+<div class="highlight"><pre><code class="text language-text">/** When a task wishes to send a message, it uses this class. */
public interface MessageCollector {
void send(KeyedMessageEnvelope envelope);
}
@@ -122,13 +122,13 @@ public interface MessageCollector {
<!-- TODO I think we're getting rid of KeyedMessageEnvelope in Jay's API change for state management. -->
<p>The collector takes KeyedMessageEnvelope, which extends the normal MessageEnvelope to allow tasks to supply a partition key when sending the message. The partition key, if supplied, is used to determine which partition of a stream a message is destined for.</p>
-<div class="highlight"><pre><code class="text language-text" data-lang="text">/** A message envelope that has a key. */
+<div class="highlight"><pre><code class="text language-text">/** A message envelope that has a key. */
public interface KeyedMessageEnvelope extends MessageEnvelope {
<K> K getKey();
}
</code></pre></div>
<p>And, putting it all together:</p>
-<div class="highlight"><pre><code class="text language-text" data-lang="text">class MyStreamerTask extends StreamTask {
+<div class="highlight"><pre><code class="text language-text">class MyStreamerTask extends StreamTask {
def process(envelope: MessageEnvelope, collector: MessageCollector, coordinator: TaskCoordinator) {
val msg = envelope.getMessage[GenericRecord]
collector.send(new OutgoingMessageEnvelope(new Stream("kafka", "SomeTopicPartitionedByMemberId"), msg.get("member_id"), msg))
Modified: incubator/samza/site/learn/documentation/0.7.0/container/checkpointing.html
URL: http://svn.apache.org/viewvc/incubator/samza/site/learn/documentation/0.7.0/container/checkpointing.html?rev=1513322&r1=1513321&r2=1513322&view=diff
==============================================================================
--- incubator/samza/site/learn/documentation/0.7.0/container/checkpointing.html (original)
+++ incubator/samza/site/learn/documentation/0.7.0/container/checkpointing.html Tue Aug 13 02:10:44 2013
@@ -74,7 +74,7 @@
<p>If a StreamConsumer is reading messages for a TaskRunner, and the TaskRunner stops for some reason (due to hardware failure, re-deployment, or whatever), the StreamConsumer should start where it left off when the TaskRunner starts back up again. We're able to do this because the Kafka broker is buffering messages on a remote server (the broker). Since the messages are available when we come back, we can just start from our last offset, and continue moving forward, without losing data.</p>
<p>The TaskRunner supports this ability using something called a CheckpointManager.</p>
-<div class="highlight"><pre><code class="text language-text" data-lang="text">public interface CheckpointManager {
+<div class="highlight"><pre><code class="text language-text">public interface CheckpointManager {
public void writeCheckpoint(Partition partition, Checkpoint checkpoint);
public Checkpoint readLastCheckpoint(Partition partition);
Modified: incubator/samza/site/learn/documentation/0.7.0/container/event-loop.html
URL: http://svn.apache.org/viewvc/incubator/samza/site/learn/documentation/0.7.0/container/event-loop.html?rev=1513322&r1=1513321&r2=1513322&view=diff
==============================================================================
--- incubator/samza/site/learn/documentation/0.7.0/container/event-loop.html (original)
+++ incubator/samza/site/learn/documentation/0.7.0/container/event-loop.html Tue Aug 13 02:10:44 2013
@@ -81,7 +81,7 @@
<h3>Lifecycle Listeners</h3>
<p>Sometimes, it's useful to receive notifications when a specific event happens in the TaskRunner. For example, you might want to reset some context in the container whenever a new message arrives. To accomplish this, Samza provides a TaskLifecycleListener interface, that can be wired into the TaskRunner through configuration.</p>
-<div class="highlight"><pre><code class="text language-text" data-lang="text">/**
+<div class="highlight"><pre><code class="text language-text">/**
* Used to get before/after notifications before initializing/closing all tasks
* in a given container (JVM/process).
*/
Modified: incubator/samza/site/learn/documentation/0.7.0/container/jmx.html
URL: http://svn.apache.org/viewvc/incubator/samza/site/learn/documentation/0.7.0/container/jmx.html?rev=1513322&r1=1513321&r2=1513322&view=diff
==============================================================================
--- incubator/samza/site/learn/documentation/0.7.0/container/jmx.html (original)
+++ incubator/samza/site/learn/documentation/0.7.0/container/jmx.html Tue Aug 13 02:10:44 2013
@@ -66,7 +66,7 @@
<h2>JMX</h2>
<p>The Samza TaskRunner (and YARN Application Master) will turn on JMX using a randomly selected port, since Samza is meant to be run in a distributed environment, and it's unknown which ports will be available prior to runtime. The port will be output in the TaskRunner's logs with a line like this:</p>
-<div class="highlight"><pre><code class="text language-text" data-lang="text">2013-07-05 20:42:36 JmxServer [INFO] According to InetAddress.getLocalHost.getHostName we are Chriss-MacBook-Pro.local
+<div class="highlight"><pre><code class="text language-text">2013-07-05 20:42:36 JmxServer [INFO] According to InetAddress.getLocalHost.getHostName we are Chriss-MacBook-Pro.local
2013-07-05 20:42:36 JmxServer [INFO] Started JmxServer port=64905 url=service:jmx:rmi:///jndi/rmi://Chriss-MacBook-Pro.local:64905/jmxrmi
</code></pre></div>
<p>Any metrics that are registered in the TaskRunner will be visible through JMX. To toggle JMX, see the <a href="../jobs/configuration.html">Configuration</a> section.</p>
Modified: incubator/samza/site/learn/documentation/0.7.0/container/metrics.html
URL: http://svn.apache.org/viewvc/incubator/samza/site/learn/documentation/0.7.0/container/metrics.html?rev=1513322&r1=1513321&r2=1513322&view=diff
==============================================================================
--- incubator/samza/site/learn/documentation/0.7.0/container/metrics.html (original)
+++ incubator/samza/site/learn/documentation/0.7.0/container/metrics.html Tue Aug 13 02:10:44 2013
@@ -66,7 +66,7 @@
<h2>Metrics</h2>
<p>Samza also provides a metrics library that the TaskRunner uses. It allows a StreamTask to create counters and gauges. The TaskRunner then writes those metrics to metrics infrastructure through a MetricsReporter implementation.</p>
-<div class="highlight"><pre><code class="text language-text" data-lang="text">public class MyJavaStreamerTask implements StreamTask, InitableTask {
+<div class="highlight"><pre><code class="text language-text">public class MyJavaStreamerTask implements StreamTask, InitableTask {
private static final Counter messageCount;
public void init(Config config, TaskContextPartition context) {
@@ -81,7 +81,7 @@
}
</code></pre></div>
<p>Samza's metrics design is very similar to Coda Hale's <a href="https://github.com/codahale/metrics">metrics</a> library. It has two important interfaces:</p>
-<div class="highlight"><pre><code class="text language-text" data-lang="text">public interface MetricsRegistry {
+<div class="highlight"><pre><code class="text language-text">public interface MetricsRegistry {
Counter newCounter(String group, String name);
<T> Gauge<T> newGauge(String group, String name, T value);
Modified: incubator/samza/site/learn/documentation/0.7.0/container/state-management.html
URL: http://svn.apache.org/viewvc/incubator/samza/site/learn/documentation/0.7.0/container/state-management.html?rev=1513322&r1=1513321&r2=1513322&view=diff
==============================================================================
--- incubator/samza/site/learn/documentation/0.7.0/container/state-management.html (original)
+++ incubator/samza/site/learn/documentation/0.7.0/container/state-management.html Tue Aug 13 02:10:44 2013
@@ -108,7 +108,7 @@
<h3>Usage</h3>
<p>To declare a new store for usage you add the following to your job config:</p>
-<div class="highlight"><pre><code class="text language-text" data-lang="text"># Use the key-value store implementation for
+<div class="highlight"><pre><code class="text language-text"># Use the key-value store implementation for
stores.my-store.factory=samza.storage.kv.KeyValueStorageEngineFactory
# Log changes to the store to a stream
stores.my-store.changelog=my-stream-name
@@ -118,7 +118,7 @@ stores.my-store.serde=string
stores.my-store.system=kafka
</code></pre></div>
<p>Example code:</p>
-<div class="highlight"><pre><code class="text language-text" data-lang="text">public class MyStatefulTask implements StreamTask, InitableTask {
+<div class="highlight"><pre><code class="text language-text">public class MyStatefulTask implements StreamTask, InitableTask {
private KeyValueStore<String, String> store;
public void init(Config config, TaskContextPartition context) {
@@ -132,7 +132,7 @@ stores.my-store.system=kafka
}
</code></pre></div>
<p>This shows the put() API, but KeyValueStore gives a fairly general key-value interface:</p>
-<div class="highlight"><pre><code class="text language-text" data-lang="text">public interface KeyValueStore<K, V> {
+<div class="highlight"><pre><code class="text language-text">public interface KeyValueStore<K, V> {
V get(K key);
void put(K key, V value);
void putAll(List<Entry<K,V>> entries);
@@ -144,7 +144,7 @@ stores.my-store.system=kafka
<h3>Implementing Storage Engines</h3>
<p>The above code shows usage of the key-value storage engine, but it is not too hard to implement an alternate storage engine. To do so, you implement methods to restore the contents of the store from a stream, flush any cached content on commit, and close the store:</p>
-<div class="highlight"><pre><code class="text language-text" data-lang="text">public interface StorageEngine {
+<div class="highlight"><pre><code class="text language-text">public interface StorageEngine {
void restore(StreamConsumer consumer);
void flush();
void close();
Modified: incubator/samza/site/learn/documentation/0.7.0/container/streams.html
URL: http://svn.apache.org/viewvc/incubator/samza/site/learn/documentation/0.7.0/container/streams.html?rev=1513322&r1=1513321&r2=1513322&view=diff
==============================================================================
--- incubator/samza/site/learn/documentation/0.7.0/container/streams.html (original)
+++ incubator/samza/site/learn/documentation/0.7.0/container/streams.html Tue Aug 13 02:10:44 2013
@@ -66,7 +66,7 @@
<h2>Streams</h2>
<p>The <a href="task-runner.html">TaskRunner</a> reads and writes messages using the StreamConsumer and StreamProducer interfaces.</p>
-<div class="highlight"><pre><code class="text language-text" data-lang="text">public interface StreamConsumer {
+<div class="highlight"><pre><code class="text language-text">public interface StreamConsumer {
StreamConsumerMessageEnvelope getMessage();
void close();
Modified: incubator/samza/site/learn/documentation/0.7.0/container/task-runner.html
URL: http://svn.apache.org/viewvc/incubator/samza/site/learn/documentation/0.7.0/container/task-runner.html?rev=1513322&r1=1513321&r2=1513322&view=diff
==============================================================================
--- incubator/samza/site/learn/documentation/0.7.0/container/task-runner.html (original)
+++ incubator/samza/site/learn/documentation/0.7.0/container/task-runner.html Tue Aug 13 02:10:44 2013
@@ -85,7 +85,7 @@
<h3>Tasks and Partitions</h3>
<p>When the TaskRunner starts, it creates an instance of the StreamTask that you've written. If the StreamTask implements the InitableTask interface, the TaskRunner will also call the init() method.</p>
-<div class="highlight"><pre><code class="text language-text" data-lang="text">public interface InitableTask {
+<div class="highlight"><pre><code class="text language-text">public interface InitableTask {
void init(Config config, TaskContextPartition context);
}
</code></pre></div>
Modified: incubator/samza/site/learn/documentation/0.7.0/container/windowing.html
URL: http://svn.apache.org/viewvc/incubator/samza/site/learn/documentation/0.7.0/container/windowing.html?rev=1513322&r1=1513321&r2=1513322&view=diff
==============================================================================
--- incubator/samza/site/learn/documentation/0.7.0/container/windowing.html (original)
+++ incubator/samza/site/learn/documentation/0.7.0/container/windowing.html Tue Aug 13 02:10:44 2013
@@ -68,7 +68,7 @@
<p>Referring back to the, "count PageViewEvent by member ID," example in the <a href="../introduction/architecture.html">Architecture</a> section, one thing that we left out was what we do with the counts. Let's say that the Samza job wants to update the member ID counts in a database once every minute. Here's how it would work. The Samza job that does the counting would keep a Map<Integer, Integer> in memory, which maps member IDs to page view counts. Every time a message arrives, the job would take the member ID in the PageViewEvent, and use it to increment the member ID's count in the in-memory map. Then, once a minute, the StreamTask would update the database (total<em>count += current</em>count) for every member ID in the map, and then reset the count map.</p>
<p>Windowing is how we achieve this. If a StreamTask implements the WindowableTask interface, the TaskRunner will call the window() method on the task over a configured interval.</p>
-<div class="highlight"><pre><code class="text language-text" data-lang="text">public interface WindowableTask {
+<div class="highlight"><pre><code class="text language-text">public interface WindowableTask {
void window(MessageCollector collector, TaskCoordinator coordinator);
}
</code></pre></div>
Modified: incubator/samza/site/learn/documentation/0.7.0/jobs/configuration.html
URL: http://svn.apache.org/viewvc/incubator/samza/site/learn/documentation/0.7.0/jobs/configuration.html?rev=1513322&r1=1513321&r2=1513322&view=diff
==============================================================================
--- incubator/samza/site/learn/documentation/0.7.0/jobs/configuration.html (original)
+++ incubator/samza/site/learn/documentation/0.7.0/jobs/configuration.html Tue Aug 13 02:10:44 2013
@@ -66,7 +66,7 @@
<h2>Configuration</h2>
<p>All Samza jobs have a configuration file that defines the job. A very basic configuration file looks like this:</p>
-<div class="highlight"><pre><code class="text language-text" data-lang="text"># Job
+<div class="highlight"><pre><code class="text language-text"># Job
job.factory.class=samza.job.local.LocalJobFactory
job.name=hello-world
Modified: incubator/samza/site/learn/documentation/0.7.0/jobs/job-runner.html
URL: http://svn.apache.org/viewvc/incubator/samza/site/learn/documentation/0.7.0/jobs/job-runner.html?rev=1513322&r1=1513321&r2=1513322&view=diff
==============================================================================
--- incubator/samza/site/learn/documentation/0.7.0/jobs/job-runner.html (original)
+++ incubator/samza/site/learn/documentation/0.7.0/jobs/job-runner.html Tue Aug 13 02:10:44 2013
@@ -66,19 +66,19 @@
<h2>JobRunner</h2>
<p>Samza jobs are started using a script called run-job.sh.</p>
-<div class="highlight"><pre><code class="text language-text" data-lang="text">samza-example/target/bin/run-job.sh \
+<div class="highlight"><pre><code class="text language-text">samza-example/target/bin/run-job.sh \
--config-factory=samza.config.factories.PropertiesConfigFactory \
--config-path=file://$PWD/config/hello-world.properties
</code></pre></div>
<p>You provide two parameters to the run-job.sh script. One is the config location, and the other is a factory class that is used to read your configuration file. The run-job.sh script is actually executing a Samza class called JobRunner. The JobRunner uses your ConfigFactory to get a Config object from the config path.</p>
-<div class="highlight"><pre><code class="text language-text" data-lang="text">public interface ConfigFactory {
+<div class="highlight"><pre><code class="text language-text">public interface ConfigFactory {
Config getConfig(URI configUri);
}
</code></pre></div>
<p>The Config object is just a wrapper around Map<String, String>, with some nice helper methods. Out of the box, Samza ships with the PropertiesConfigFactory, but developers can implement any kind of ConfigFactory they wish.</p>
<p>Once the JobRunner gets your configuration, it gives your configuration to the StreamJobFactory class defined by the "job.factory" property. Samza ships with two job factory implementations: LocalJobFactory and YarnJobFactory. The StreamJobFactory's responsibility is to give the JobRunner a job that it can run.</p>
-<div class="highlight"><pre><code class="text language-text" data-lang="text">public interface StreamJob {
+<div class="highlight"><pre><code class="text language-text">public interface StreamJob {
StreamJob submit();
StreamJob kill();
Modified: incubator/samza/site/learn/documentation/0.7.0/jobs/logging.html
URL: http://svn.apache.org/viewvc/incubator/samza/site/learn/documentation/0.7.0/jobs/logging.html?rev=1513322&r1=1513321&r2=1513322&view=diff
==============================================================================
--- incubator/samza/site/learn/documentation/0.7.0/jobs/logging.html (original)
+++ incubator/samza/site/learn/documentation/0.7.0/jobs/logging.html Tue Aug 13 02:10:44 2013
@@ -70,7 +70,7 @@
<h3>Log4j</h3>
<p>The <a href="/startup/hello-samza/0.7.0">hello-samza</a> project shows how to use <a href="http://logging.apache.org/log4j/1.2/">log4j</a> with Samza. To turn on log4j logging, you just need to make sure slf4j-log4j12 is in your Samza TaskRunner's classpath. In Maven, this can be done by adding the following dependency to your Samza package project.</p>
-<div class="highlight"><pre><code class="text language-text" data-lang="text"><dependency>
+<div class="highlight"><pre><code class="text language-text"><dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-log4j12</artifactId>
<scope>runtime</scope>
@@ -82,7 +82,7 @@
<h4>log4j.xml</h4>
<p>Samza's <a href="packaging.html">run-class.sh</a> script will automatically set the following setting if log4j.xml exists in your <a href="packaging.html">Samza package's</a> lib directory.</p>
-<div class="highlight"><pre><code class="text language-text" data-lang="text">-Dlog4j.configuration=file:$base_dir/lib/log4j.xml
+<div class="highlight"><pre><code class="text language-text">-Dlog4j.configuration=file:$base_dir/lib/log4j.xml
</code></pre></div>
<!-- TODO add notes showing how to use task.opts for gc logging
#### task.opts
@@ -95,7 +95,7 @@
<h3>Garbage Collection Logging</h3>
<p>Samza's will automatically set the following garbage collection logging setting, and will output it to <em>$SAMZA</em>_<em>LOG</em>_<em>DIR</em>/gc.log.</p>
-<div class="highlight"><pre><code class="text language-text" data-lang="text">-XX:+PrintGCDateStamps -Xloggc:$SAMZA_LOG_DIR/gc.log
+<div class="highlight"><pre><code class="text language-text">-XX:+PrintGCDateStamps -Xloggc:$SAMZA_LOG_DIR/gc.log
</code></pre></div>
<h4>Rotation</h4>
Modified: incubator/samza/site/learn/documentation/0.7.0/jobs/packaging.html
URL: http://svn.apache.org/viewvc/incubator/samza/site/learn/documentation/0.7.0/jobs/packaging.html?rev=1513322&r1=1513321&r2=1513322&view=diff
==============================================================================
--- incubator/samza/site/learn/documentation/0.7.0/jobs/packaging.html (original)
+++ incubator/samza/site/learn/documentation/0.7.0/jobs/packaging.html Tue Aug 13 02:10:44 2013
@@ -66,13 +66,13 @@
<h2>Packaging</h2>
<p>The <a href="job-runner.html">JobRunner</a> page talks about run-job.sh, and how it's used to start a job either locally (LocalJobFactory) or with YARN (YarnJobFactory). In the diagram that shows the execution flow, it also shows a run-task.sh script. This script, along with a run-am.sh script, are what Samza actually calls to execute its code.</p>
-<div class="highlight"><pre><code class="text language-text" data-lang="text">bin/run-am.sh
+<div class="highlight"><pre><code class="text language-text">bin/run-am.sh
bin/run-task.sh
</code></pre></div>
<p>The run-task.sh script is responsible for starting the TaskRunner. The run-am.sh script is responsible for starting Samza's application master for YARN. Thus, the run-am.sh script is only used by the YarnJob, but both YarnJob and ProcessJob use run-task.sh.</p>
<p>Typically, these two scripts are bundled into a tar.gz file that has a structure like this:</p>
-<div class="highlight"><pre><code class="text language-text" data-lang="text">bin/run-am.sh
+<div class="highlight"><pre><code class="text language-text">bin/run-am.sh
bin/run-class.sh
bin/run-job.sh
bin/run-task.sh
Modified: incubator/samza/site/learn/documentation/0.7.0/jobs/yarn-jobs.html
URL: http://svn.apache.org/viewvc/incubator/samza/site/learn/documentation/0.7.0/jobs/yarn-jobs.html?rev=1513322&r1=1513321&r2=1513322&view=diff
==============================================================================
--- incubator/samza/site/learn/documentation/0.7.0/jobs/yarn-jobs.html (original)
+++ incubator/samza/site/learn/documentation/0.7.0/jobs/yarn-jobs.html Tue Aug 13 02:10:44 2013
@@ -68,7 +68,7 @@
<p>When you define job.factory.class=samza.job.yarn.YarnJobFactory in your job's configuration, Samza will use YARN to execute your job. The YarnJobFactory will use the YARN_HOME environment variable on the machine that run-job.sh is executed on to get the appropriate YARN configuration, which will define where the YARN resource manager is. The YarnJob will work with the resource manager to get your job started on the YARN cluster.</p>
<p>If you want to use YARN to run your Samza job, you'll also need to define the location of your Samza job's package. For example, you might say:</p>
-<div class="highlight"><pre><code class="text language-text" data-lang="text">yarn.package.path=http://my.http.server/jobs/ingraphs-package-0.0.55.tgz
+<div class="highlight"><pre><code class="text language-text">yarn.package.path=http://my.http.server/jobs/ingraphs-package-0.0.55.tgz
</code></pre></div>
<p>This .tgz file follows the conventions outlined on the <a href="packaging.html">Packaging</a> page (it has bin/run-am.sh and bin/run-task.sh). YARN NodeManagers will take responsibility for downloading this .tgz file on the appropriate machines, and untar'ing them. From there, YARN will execute run-am.sh or run-task.sh for the Samza Application Master, and TaskRunner, respectively.</p>
Modified: incubator/samza/site/learn/documentation/0.7.0/operations/kafka.html
URL: http://svn.apache.org/viewvc/incubator/samza/site/learn/documentation/0.7.0/operations/kafka.html?rev=1513322&r1=1513321&r2=1513322&view=diff
==============================================================================
--- incubator/samza/site/learn/documentation/0.7.0/operations/kafka.html (original)
+++ incubator/samza/site/learn/documentation/0.7.0/operations/kafka.html Tue Aug 13 02:10:44 2013
@@ -74,7 +74,7 @@
<h3>Auto-Create Topics</h3>
<p>Kafka brokers should be configured to automatically create topics. Without this, it's going to be very cumbersome to run Samze jobs, since jobs will write to arbitrary (and sometimes new) topics.</p>
-<div class="highlight"><pre><code class="text language-text" data-lang="text">auto.create.topics.enable=true
+<div class="highlight"><pre><code class="text language-text">auto.create.topics.enable=true
</code></pre></div>
</div>
Modified: incubator/samza/site/startup/download/index.html
URL: http://svn.apache.org/viewvc/incubator/samza/site/startup/download/index.html?rev=1513322&r1=1513321&r2=1513322&view=diff
==============================================================================
--- incubator/samza/site/startup/download/index.html (original)
+++ incubator/samza/site/startup/download/index.html Tue Aug 13 02:10:44 2013
@@ -129,7 +129,7 @@ Snapshot builds are available in the Apa
<h3>Checking out and Building</h3>
<p>If you're interested in working on Samza, or building the JARs from scratch, then you'll need to checkout and build the code. Samza does not have a binary release at this time. To check out and build Samza, run these commands.</p>
-<div class="highlight"><pre><code class="text language-text" data-lang="text">git clone http://git-wip-us.apache.org/repos/asf/incubator-samza.git
+<div class="highlight"><pre><code class="text language-text">git clone http://git-wip-us.apache.org/repos/asf/incubator-samza.git
cd incubator-samza
./gradlew clean build
</code></pre></div>
Modified: incubator/samza/site/startup/hello-samza/0.7.0/index.html
URL: http://svn.apache.org/viewvc/incubator/samza/site/startup/hello-samza/0.7.0/index.html?rev=1513322&r1=1513321&r2=1513322&view=diff
==============================================================================
--- incubator/samza/site/startup/hello-samza/0.7.0/index.html (original)
+++ incubator/samza/site/startup/hello-samza/0.7.0/index.html Tue Aug 13 02:10:44 2013
@@ -72,19 +72,19 @@
<h3>Get the Code</h3>
<p>You'll need to check out and publish Samza, since it's not available in a Maven repository right now.</p>
-<div class="highlight"><pre><code class="text language-text" data-lang="text">git clone http://git-wip-us.apache.org/repos/asf/incubator-samza.git
+<div class="highlight"><pre><code class="text language-text">git clone http://git-wip-us.apache.org/repos/asf/incubator-samza.git
cd incubator-samza
./gradlew -PscalaVersion=2.8.1 clean publishToMavenLocal
</code></pre></div>
<p>Next, check out the hello-samza project.</p>
-<div class="highlight"><pre><code class="text language-text" data-lang="text">git clone git://github.com/linkedin/hello-samza.git
+<div class="highlight"><pre><code class="text language-text">git clone git://github.com/linkedin/hello-samza.git
</code></pre></div>
<p>This project contains everything you'll need to run your first Samza jobs.</p>
<h3>Start a Grid</h3>
<p>A Samza grid usually comprises three different systems: <a href="http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/YARN.html">YARN</a>, <a href="http://kafka.apache.org/">Kafka</a>, and <a href="http://zookeeper.apache.org/">ZooKeeper</a>. The hello-samza project comes with a script called "grid" to help you setup these systems. Start by running:</p>
-<div class="highlight"><pre><code class="text language-text" data-lang="text">bin/grid
+<div class="highlight"><pre><code class="text language-text">bin/grid
</code></pre></div>
<p>This command will download, install, and start ZooKeeper, Kafka, and YARN. All package files will be put in a sub-directory called "deploy" inside hello-samza's root folder.</p>
@@ -93,34 +93,34 @@ cd incubator-samza
<h3>Build a Samza Job Package</h3>
<p>Before you can run a Samza job, you need to build a package for it. This package is what YARN uses to deploy your jobs on the grid.</p>
-<div class="highlight"><pre><code class="text language-text" data-lang="text">mvn clean package
+<div class="highlight"><pre><code class="text language-text">mvn clean package
mkdir -p deploy/samza
tar -xvf ./samza-job-package/target/samza-job-package-0.7.0-dist.tar.gz -C deploy/samza
</code></pre></div>
<h3>Run a Samza Job</h3>
<p>After you've built your Samza package, you can start a job on the grid using the run-job.sh script.</p>
-<div class="highlight"><pre><code class="text language-text" data-lang="text">deploy/samza/bin/run-job.sh --config-factory=samza.config.factories.PropertiesConfigFactory --config-path=file://$PWD/deploy/samza/config/wikipedia-feed.properties
+<div class="highlight"><pre><code class="text language-text">deploy/samza/bin/run-job.sh --config-factory=org.apache.samza.config.factories.PropertiesConfigFactory --config-path=file://$PWD/deploy/samza/config/wikipedia-feed.properties
</code></pre></div>
<p>The job will consume a feed of real-time edits from Wikipedia, and produce them to a Kafka topic called "wikipedia-raw". Give the job a minute to startup, and then tail the Kafka topic:</p>
-<div class="highlight"><pre><code class="text language-text" data-lang="text">deploy/kafka/bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic wikipedia-raw
+<div class="highlight"><pre><code class="text language-text">deploy/kafka/bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic wikipedia-raw
</code></pre></div>
<p>Pretty neat, right? Now, check out the YARN UI again (<a href="http://localhost:8088">http://localhost:8088</a>). This time around, you'll see your Samza job is running!</p>
<h3>Generate Wikipedia Statistics</h3>
<p>Let's calculate some statistics based on the messages in the wikipedia-raw topic. Start two more jobs:</p>
-<div class="highlight"><pre><code class="text language-text" data-lang="text">deploy/samza/bin/run-job.sh --config-factory=samza.config.factories.PropertiesConfigFactory --config-path=file://$PWD/deploy/samza/config/wikipedia-parser.properties
-deploy/samza/bin/run-job.sh --config-factory=samza.config.factories.PropertiesConfigFactory --config-path=file://$PWD/deploy/samza/config/wikipedia-stats.properties
+<div class="highlight"><pre><code class="text language-text">deploy/samza/bin/run-job.sh --config-factory=org.apache.samza.config.factories.PropertiesConfigFactory --config-path=file://$PWD/deploy/samza/config/wikipedia-parser.properties
+deploy/samza/bin/run-job.sh --config-factory=org.apache.samza.config.factories.PropertiesConfigFactory --config-path=file://$PWD/deploy/samza/config/wikipedia-stats.properties
</code></pre></div>
<p>The first job (wikipedia-parser) parses the messages in wikipedia-raw, and extracts information about the size of the edit, who made the change, etc. You can take a look at its output with:</p>
-<div class="highlight"><pre><code class="text language-text" data-lang="text">deploy/kafka/bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic wikipedia-edits
+<div class="highlight"><pre><code class="text language-text">deploy/kafka/bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic wikipedia-edits
</code></pre></div>
<p>The last job (wikipedia-stats) reads messages from the wikipedia-edits topic, and calculates counts, every ten seconds, for all edits that were made during that window. It outputs these counts to the wikipedia-stats topic.</p>
-<div class="highlight"><pre><code class="text language-text" data-lang="text">deploy/kafka/bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic wikipedia-stats
+<div class="highlight"><pre><code class="text language-text">deploy/kafka/bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic wikipedia-stats
</code></pre></div>
<p>The messages in the stats topic look like this:</p>
-<div class="highlight"><pre><code class="text language-text" data-lang="text">{"is-talk":2,"bytes-added":5276,"edits":13,"unique-titles":13}
+<div class="highlight"><pre><code class="text language-text">{"is-talk":2,"bytes-added":5276,"edits":13,"unique-titles":13}
{"is-bot-edit":1,"is-talk":3,"bytes-added":4211,"edits":30,"unique-titles":30,"is-unpatrolled":1,"is-new":2,"is-minor":7}
{"bytes-added":3180,"edits":19,"unique-titles":19,"is-unpatrolled":1,"is-new":1,"is-minor":3}
{"bytes-added":2218,"edits":18,"unique-titles":18,"is-unpatrolled":2,"is-new":2,"is-minor":3}
@@ -130,7 +130,7 @@ deploy/samza/bin/run-job.sh --config-fac
<h3>Shutdown</h3>
<p>After you're done, you can clean everything up using the same grid script.</p>
-<div class="highlight"><pre><code class="text language-text" data-lang="text">bin/grid stop yarn
+<div class="highlight"><pre><code class="text language-text">bin/grid stop yarn
bin/grid stop kafka
bin/grid stop zookeeper
</code></pre></div>