You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@samza.apache.org by ma...@apache.org on 2014/06/09 22:35:58 UTC
svn commit: r1601502 [3/3] - in /incubator/samza/site: ./ css/
img/0.7.0/learn/documentation/container/ learn/documentation/0.7.0/
learn/documentation/0.7.0/api/ learn/documentation/0.7.0/comparisons/
learn/documentation/0.7.0/container/ learn/document...
Modified: incubator/samza/site/learn/documentation/0.7.0/jobs/logging.html
URL: http://svn.apache.org/viewvc/incubator/samza/site/learn/documentation/0.7.0/jobs/logging.html?rev=1601502&r1=1601501&r2=1601502&view=diff
==============================================================================
--- incubator/samza/site/learn/documentation/0.7.0/jobs/logging.html (original)
+++ incubator/samza/site/learn/documentation/0.7.0/jobs/logging.html Mon Jun 9 20:35:57 2014
@@ -76,7 +76,7 @@
<h3>Log4j</h3>
-<p>The <a href="/startup/hello-samza/0.7.0">hello-samza</a> project shows how to use <a href="http://logging.apache.org/log4j/1.2/">log4j</a> with Samza. To turn on log4j logging, you just need to make sure slf4j-log4j12 is in your Samza TaskRunner's classpath. In Maven, this can be done by adding the following dependency to your Samza package project.</p>
+<p>The <a href="/startup/hello-samza/0.7.0">hello-samza</a> project shows how to use <a href="http://logging.apache.org/log4j/1.2/">log4j</a> with Samza. To turn on log4j logging, you just need to make sure slf4j-log4j12 is in your SamzaContainer's classpath. In Maven, this can be done by adding the following dependency to your Samza package project.</p>
<div class="highlight"><pre><code class="text language-text" data-lang="text"><dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-log4j12</artifactId>
@@ -86,7 +86,7 @@
</code></pre></div>
<p>If you're not using Maven, just make sure that slf4j-log4j12 ends up in your Samza package's lib directory.</p>
-<h4>log4j.xml</h4>
+<h4>Log4j configuration</h4>
<p>Samza's <a href="packaging.html">run-class.sh</a> script will automatically set the following setting if log4j.xml exists in your <a href="packaging.html">Samza package's</a> lib directory.</p>
<div class="highlight"><pre><code class="text language-text" data-lang="text">-Dlog4j.configuration=file:$base_dir/lib/log4j.xml
@@ -105,9 +105,7 @@
</code></pre></div>
<p>Setting up a file-based appender is recommended as a better alternative to using standard out. Standard out log files (see below) don't roll, and can get quite large if used for logging.</p>
-<!-- TODO add notes showing how to use task.opts for gc logging
-#### task.opts
--->
+<p><strong>NOTE:</strong> If you use the task.opts configuration property, the log configuration is disrupted. This is a known bug; please see <a href="https://issues.apache.org/jira/browse/SAMZA-109">SAMZA-109</a> for a workaround.</p>
<h3>Log Directory</h3>
Modified: incubator/samza/site/learn/documentation/0.7.0/jobs/packaging.html
URL: http://svn.apache.org/viewvc/incubator/samza/site/learn/documentation/0.7.0/jobs/packaging.html?rev=1601502&r1=1601501&r2=1601502&view=diff
==============================================================================
--- incubator/samza/site/learn/documentation/0.7.0/jobs/packaging.html (original)
+++ incubator/samza/site/learn/documentation/0.7.0/jobs/packaging.html Mon Jun 9 20:35:57 2014
@@ -76,7 +76,7 @@
<div class="highlight"><pre><code class="text language-text" data-lang="text">bin/run-am.sh
bin/run-container.sh
</code></pre></div>
-<p>The run-container.sh script is responsible for starting the TaskRunner. The run-am.sh script is responsible for starting Samza's application master for YARN. Thus, the run-am.sh script is only used by the YarnJob, but both YarnJob and ProcessJob use run-container.sh.</p>
+<p>The run-container.sh script is responsible for starting the <a href="../container/samza-container.html">SamzaContainer</a>. The run-am.sh script is responsible for starting Samza's application master for YARN. Thus, the run-am.sh script is only used by the YarnJob, but both YarnJob and ProcessJob use run-container.sh.</p>
<p>Typically, these two scripts are bundled into a tar.gz file that has a structure like this:</p>
<div class="highlight"><pre><code class="text language-text" data-lang="text">bin/run-am.sh
Modified: incubator/samza/site/learn/documentation/0.7.0/jobs/yarn-jobs.html
URL: http://svn.apache.org/viewvc/incubator/samza/site/learn/documentation/0.7.0/jobs/yarn-jobs.html?rev=1601502&r1=1601501&r2=1601502&view=diff
==============================================================================
--- incubator/samza/site/learn/documentation/0.7.0/jobs/yarn-jobs.html (original)
+++ incubator/samza/site/learn/documentation/0.7.0/jobs/yarn-jobs.html Mon Jun 9 20:35:57 2014
@@ -72,12 +72,14 @@
<div class="content">
<h2>YARN Jobs</h2>
-<p>When you define job.factory.class=samza.job.yarn.YarnJobFactory in your job's configuration, Samza will use YARN to execute your job. The YarnJobFactory will use the YARN_HOME environment variable on the machine that run-job.sh is executed on to get the appropriate YARN configuration, which will define where the YARN resource manager is. The YarnJob will work with the resource manager to get your job started on the YARN cluster.</p>
+<p>When you define job.factory.class=org.apache.samza.job.yarn.YarnJobFactory in your job's configuration, Samza will use YARN to execute your job. The YarnJobFactory will use the YARN_HOME environment variable on the machine that run-job.sh is executed on to get the appropriate YARN configuration, which will define where the YARN resource manager is. The YarnJob will work with the resource manager to get your job started on the YARN cluster.</p>
<p>If you want to use YARN to run your Samza job, you'll also need to define the location of your Samza job's package. For example, you might say:</p>
<div class="highlight"><pre><code class="text language-text" data-lang="text">yarn.package.path=http://my.http.server/jobs/ingraphs-package-0.0.55.tgz
</code></pre></div>
-<p>This .tgz file follows the conventions outlined on the <a href="packaging.html">Packaging</a> page (it has bin/run-am.sh and bin/run-container.sh). YARN NodeManagers will take responsibility for downloading this .tgz file on the appropriate machines, and untar'ing them. From there, YARN will execute run-am.sh or run-container.sh for the Samza Application Master, and TaskRunner, respectively.</p>
+<p>This .tgz file follows the conventions outlined on the <a href="packaging.html">Packaging</a> page (it has bin/run-am.sh and bin/run-container.sh). YARN NodeManagers will take responsibility for downloading this .tgz file on the appropriate machines, and untar'ing them. From there, YARN will execute run-am.sh or run-container.sh for the Samza Application Master, and SamzaContainer, respectively.</p>
+
+<!-- TODO document yarn.container.count and other key configs -->
<h2><a href="logging.html">Logging »</a></h2>
Modified: incubator/samza/site/learn/documentation/0.7.0/yarn/application-master.html
URL: http://svn.apache.org/viewvc/incubator/samza/site/learn/documentation/0.7.0/yarn/application-master.html?rev=1601502&r1=1601501&r2=1601502&view=diff
==============================================================================
--- incubator/samza/site/learn/documentation/0.7.0/yarn/application-master.html (original)
+++ incubator/samza/site/learn/documentation/0.7.0/yarn/application-master.html Mon Jun 9 20:35:57 2014
@@ -76,7 +76,7 @@
<h3>Integration</h3>
-<p>Samza's main integration with YARN comes in the form of a Samza ApplicationMaster. This is the chunk of code responsible for managing a Samza job in a YARN grid. It decides what to do when a stream processor fails, which machines a Samza job's <a href="../container/task-runner.html">TaskRunner</a> should run on, and so on.</p>
+<p>Samza's main integration with YARN comes in the form of a Samza ApplicationMaster. This is the chunk of code responsible for managing a Samza job in a YARN grid. It decides what to do when a stream processor fails, which machines a Samza job's <a href="../container/samza-container.html">containers</a> should run on, and so on.</p>
<p>When the Samza ApplicationMaster starts up, it does the following:</p>
@@ -96,11 +96,11 @@
<h3>Fault Tolerance</h3>
-<p>Whenever a container is allocated, the AM will work with the YARN NM to start a TaskRunner (with appropriate partitions assigned to it) in the container. If a container fails with a non-zero return code, the AM will request a new container, and restart the TaskRunner. If a TaskRunner fails too many times, too quickly, the ApplicationMaster will fail the whole Samza job with a non-zero return code. See the yarn.countainer.retry.count and yarn.container.retry.window.ms <a href="../jobs/configuration.html">configuration</a> parameters for details.</p>
+<p>Whenever a container is allocated, the AM will work with the YARN NM to start a SamzaContainer (with appropriate partitions assigned to it) in the container. If a container fails with a non-zero return code, the AM will request a new container, and restart the SamzaContainer. If a SamzaContainer fails too many times, too quickly, the ApplicationMaster will fail the whole Samza job with a non-zero return code. See the yarn.countainer.retry.count and yarn.container.retry.window.ms <a href="../jobs/configuration.html">configuration</a> parameters for details.</p>
<p>When the AM receives a reboot signal from YARN, it will throw a SamzaException. This will trigger a clean and successful shutdown of the AM (YARN won't think the AM failed).</p>
-<p>If the AM, itself, fails, YARN will handle restarting the AM. When the AM is restarted, all containers that were running will be killed, and the AM will start from scratch. The same list of operations, shown above, will be executed. The AM will request new containers for its TaskRunners, and proceed as though it has just started for the first time. YARN has a yarn.resourcemanager.am.max-retries configuration parameter that's defined in <a href="http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-common/yarn-default.xml">yarn-site.xml</a>. This configuration defaults to 1, which means that, by default, a single AM failure will cause your Samza job to stop running.</p>
+<p>If the AM, itself, fails, YARN will handle restarting the AM. When the AM is restarted, all containers that were running will be killed, and the AM will start from scratch. The same list of operations, shown above, will be executed. The AM will request new containers for its SamzaContainers, and proceed as though it has just started for the first time. YARN has a yarn.resourcemanager.am.max-retries configuration parameter that's defined in <a href="http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-common/yarn-default.xml">yarn-site.xml</a>. This configuration defaults to 1, which means that, by default, a single AM failure will cause your Samza job to stop running.</p>
<h3>Dashboard</h3>
@@ -115,7 +115,7 @@
<p>You can find this dashboard by going to your YARN grid's ResourceManager page (usually something like <a href="http://localhost:8088/cluster">http://localhost:8088/cluster</a>), and clicking on the "ApplicationMaster" link of a running Samza job.</p>
-<p><img src="/img/0.7.0/learn/documentation/yarn/samza-am-dashboard.png" alt="diagram"></p>
+<p><img src="/img/0.7.0/learn/documentation/yarn/samza-am-dashboard.png" alt="Screenshot of ApplicationMaster dashboard" class="diagram-large"></p>
<h3>Security</h3>
Modified: incubator/samza/site/learn/documentation/0.7.0/yarn/isolation.html
URL: http://svn.apache.org/viewvc/incubator/samza/site/learn/documentation/0.7.0/yarn/isolation.html?rev=1601502&r1=1601501&r2=1601502&view=diff
==============================================================================
--- incubator/samza/site/learn/documentation/0.7.0/yarn/isolation.html (original)
+++ incubator/samza/site/learn/documentation/0.7.0/yarn/isolation.html Mon Jun 9 20:35:57 2014
@@ -82,7 +82,7 @@
<p>YARN will automatically enforce memory limits for all containers that it executes. All containers must have a max-memory size defined when they're created. If the sum of all memory usage for processes associated with a single YARN container exceeds this maximum, YARN will kill the container.</p>
-<p>Samza supports memory limits using the yarn.container.memory.mb and yarn.am.container.memory.mb configuration parameters. Keep in mind that this is simply the amount of memory YARN will allow a Samza <a href="../container/task-runner.html">TaskRunner</a> or <a href="application-master.html">ApplicationMaster</a> to have. You'll still need to configure your heap settings appropriately using task.opts, when using Java (the default is -Xmx160M). See the <a href="../jobs/configuration.html">Configuration</a> and <a href="../jobs/packaging.html">Packaging</a> pages for details.</p>
+<p>Samza supports memory limits using the yarn.container.memory.mb and yarn.am.container.memory.mb configuration parameters. Keep in mind that this is simply the amount of memory YARN will allow a <a href="../container/samza-container.html">SamzaContainer</a> or <a href="application-master.html">ApplicationMaster</a> to have. You'll still need to configure your heap settings appropriately using task.opts, when using Java (the default is -Xmx160M). See the <a href="../jobs/configuration.html">Configuration</a> and <a href="../jobs/packaging.html">Packaging</a> pages for details.</p>
<h3>CPU</h3>
Modified: incubator/samza/site/sitemap.xml
URL: http://svn.apache.org/viewvc/incubator/samza/site/sitemap.xml?rev=1601502&r1=1601501&r2=1601502&view=diff
==============================================================================
--- incubator/samza/site/sitemap.xml (original)
+++ incubator/samza/site/sitemap.xml Mon Jun 9 20:35:57 2014
@@ -4,7 +4,7 @@
<url>
<loc>http://samza.incubator.apache.org/</loc>
- <lastmod>2014-05-23</lastmod>
+ <lastmod>2014-06-09</lastmod>
<changefreq>daily</changefreq>
<priority>1.0</priority>
</url>
@@ -14,301 +14,301 @@
<url>
<loc>http://samza.incubator.apache.org/community/committers.html</loc>
- <lastmod>2014-05-23</lastmod>
+ <lastmod>2014-06-09</lastmod>
</url>
<url>
<loc>http://samza.incubator.apache.org/community/irc.html</loc>
- <lastmod>2014-05-23</lastmod>
+ <lastmod>2014-06-09</lastmod>
</url>
<url>
<loc>http://samza.incubator.apache.org/community/mailing-lists.html</loc>
- <lastmod>2014-05-23</lastmod>
+ <lastmod>2014-06-09</lastmod>
</url>
<url>
<loc>http://samza.incubator.apache.org/contribute/code.html</loc>
- <lastmod>2014-05-23</lastmod>
+ <lastmod>2014-06-09</lastmod>
</url>
<url>
<loc>http://samza.incubator.apache.org/contribute/coding-guide.html</loc>
- <lastmod>2014-05-23</lastmod>
+ <lastmod>2014-06-09</lastmod>
</url>
<url>
<loc>http://samza.incubator.apache.org/contribute/disclaimer.html</loc>
- <lastmod>2014-05-23</lastmod>
+ <lastmod>2014-06-09</lastmod>
</url>
<url>
<loc>http://samza.incubator.apache.org/contribute/projects.html</loc>
- <lastmod>2014-05-23</lastmod>
+ <lastmod>2014-06-09</lastmod>
</url>
<url>
<loc>http://samza.incubator.apache.org/contribute/rules.html</loc>
- <lastmod>2014-05-23</lastmod>
+ <lastmod>2014-06-09</lastmod>
</url>
<url>
<loc>http://samza.incubator.apache.org/contribute/seps.html</loc>
- <lastmod>2014-05-23</lastmod>
+ <lastmod>2014-06-09</lastmod>
</url>
<url>
<loc>http://samza.incubator.apache.org/index.html</loc>
- <lastmod>2014-05-23</lastmod>
+ <lastmod>2014-06-09</lastmod>
</url>
<url>
<loc>http://samza.incubator.apache.org/learn/documentation/0.7.0/api/overview.html</loc>
- <lastmod>2014-05-23</lastmod>
+ <lastmod>2014-06-09</lastmod>
</url>
<url>
<loc>http://samza.incubator.apache.org/learn/documentation/0.7.0/comparisons/introduction.html</loc>
- <lastmod>2014-05-23</lastmod>
+ <lastmod>2014-06-09</lastmod>
</url>
<url>
<loc>http://samza.incubator.apache.org/learn/documentation/0.7.0/comparisons/mupd8.html</loc>
- <lastmod>2014-05-23</lastmod>
+ <lastmod>2014-06-09</lastmod>
</url>
<url>
<loc>http://samza.incubator.apache.org/learn/documentation/0.7.0/comparisons/storm.html</loc>
- <lastmod>2014-05-23</lastmod>
+ <lastmod>2014-06-09</lastmod>
</url>
<url>
<loc>http://samza.incubator.apache.org/learn/documentation/0.7.0/container/checkpointing.html</loc>
- <lastmod>2014-05-23</lastmod>
+ <lastmod>2014-06-09</lastmod>
</url>
<url>
<loc>http://samza.incubator.apache.org/learn/documentation/0.7.0/container/event-loop.html</loc>
- <lastmod>2014-05-23</lastmod>
+ <lastmod>2014-06-09</lastmod>
</url>
<url>
- <loc>http://samza.incubator.apache.org/learn/documentation/0.7.0/container/index.html</loc>
- <lastmod>2014-05-23</lastmod>
+ <loc>http://samza.incubator.apache.org/learn/documentation/0.7.0/container/jmx.html</loc>
+ <lastmod>2014-06-09</lastmod>
</url>
<url>
- <loc>http://samza.incubator.apache.org/learn/documentation/0.7.0/container/jmx.html</loc>
- <lastmod>2014-05-23</lastmod>
+ <loc>http://samza.incubator.apache.org/learn/documentation/0.7.0/container/metrics.html</loc>
+ <lastmod>2014-06-09</lastmod>
</url>
<url>
- <loc>http://samza.incubator.apache.org/learn/documentation/0.7.0/container/metrics.html</loc>
- <lastmod>2014-05-23</lastmod>
+ <loc>http://samza.incubator.apache.org/learn/documentation/0.7.0/container/samza-container.html</loc>
+ <lastmod>2014-06-09</lastmod>
</url>
<url>
- <loc>http://samza.incubator.apache.org/learn/documentation/0.7.0/container/state-management.html</loc>
- <lastmod>2014-05-23</lastmod>
+ <loc>http://samza.incubator.apache.org/learn/documentation/0.7.0/container/serialization.html</loc>
+ <lastmod>2014-06-09</lastmod>
</url>
<url>
- <loc>http://samza.incubator.apache.org/learn/documentation/0.7.0/container/streams.html</loc>
- <lastmod>2014-05-23</lastmod>
+ <loc>http://samza.incubator.apache.org/learn/documentation/0.7.0/container/state-management.html</loc>
+ <lastmod>2014-06-09</lastmod>
</url>
<url>
- <loc>http://samza.incubator.apache.org/learn/documentation/0.7.0/container/task-runner.html</loc>
- <lastmod>2014-05-23</lastmod>
+ <loc>http://samza.incubator.apache.org/learn/documentation/0.7.0/container/streams.html</loc>
+ <lastmod>2014-06-09</lastmod>
</url>
<url>
<loc>http://samza.incubator.apache.org/learn/documentation/0.7.0/container/windowing.html</loc>
- <lastmod>2014-05-23</lastmod>
+ <lastmod>2014-06-09</lastmod>
</url>
<url>
<loc>http://samza.incubator.apache.org/learn/documentation/0.7.0/index.html</loc>
- <lastmod>2014-05-23</lastmod>
+ <lastmod>2014-06-09</lastmod>
</url>
<url>
<loc>http://samza.incubator.apache.org/learn/documentation/0.7.0/introduction/architecture.html</loc>
- <lastmod>2014-05-23</lastmod>
+ <lastmod>2014-06-09</lastmod>
</url>
<url>
<loc>http://samza.incubator.apache.org/learn/documentation/0.7.0/introduction/background.html</loc>
- <lastmod>2014-05-23</lastmod>
+ <lastmod>2014-06-09</lastmod>
</url>
<url>
<loc>http://samza.incubator.apache.org/learn/documentation/0.7.0/introduction/concepts.html</loc>
- <lastmod>2014-05-23</lastmod>
+ <lastmod>2014-06-09</lastmod>
</url>
<url>
<loc>http://samza.incubator.apache.org/learn/documentation/0.7.0/jobs/configuration.html</loc>
- <lastmod>2014-05-23</lastmod>
+ <lastmod>2014-06-09</lastmod>
</url>
<url>
<loc>http://samza.incubator.apache.org/learn/documentation/0.7.0/jobs/job-runner.html</loc>
- <lastmod>2014-05-23</lastmod>
+ <lastmod>2014-06-09</lastmod>
</url>
<url>
<loc>http://samza.incubator.apache.org/learn/documentation/0.7.0/jobs/logging.html</loc>
- <lastmod>2014-05-23</lastmod>
+ <lastmod>2014-06-09</lastmod>
</url>
<url>
<loc>http://samza.incubator.apache.org/learn/documentation/0.7.0/jobs/packaging.html</loc>
- <lastmod>2014-05-23</lastmod>
+ <lastmod>2014-06-09</lastmod>
</url>
<url>
<loc>http://samza.incubator.apache.org/learn/documentation/0.7.0/jobs/yarn-jobs.html</loc>
- <lastmod>2014-05-23</lastmod>
+ <lastmod>2014-06-09</lastmod>
</url>
<url>
<loc>http://samza.incubator.apache.org/learn/documentation/0.7.0/operations/kafka.html</loc>
- <lastmod>2014-05-23</lastmod>
+ <lastmod>2014-06-09</lastmod>
</url>
<url>
<loc>http://samza.incubator.apache.org/learn/documentation/0.7.0/operations/security.html</loc>
- <lastmod>2014-05-23</lastmod>
+ <lastmod>2014-06-09</lastmod>
</url>
<url>
<loc>http://samza.incubator.apache.org/learn/documentation/0.7.0/yarn/application-master.html</loc>
- <lastmod>2014-05-23</lastmod>
+ <lastmod>2014-06-09</lastmod>
</url>
<url>
<loc>http://samza.incubator.apache.org/learn/documentation/0.7.0/yarn/isolation.html</loc>
- <lastmod>2014-05-23</lastmod>
+ <lastmod>2014-06-09</lastmod>
</url>
<url>
<loc>http://samza.incubator.apache.org/learn/tutorials/0.7.0/deploy-samza-job-from-hdfs.html</loc>
- <lastmod>2014-05-23</lastmod>
+ <lastmod>2014-06-09</lastmod>
</url>
<url>
<loc>http://samza.incubator.apache.org/learn/tutorials/0.7.0/index.html</loc>
- <lastmod>2014-05-23</lastmod>
+ <lastmod>2014-06-09</lastmod>
</url>
<url>
<loc>http://samza.incubator.apache.org/learn/tutorials/0.7.0/remote-debugging-samza.html</loc>
- <lastmod>2014-05-23</lastmod>
+ <lastmod>2014-06-09</lastmod>
</url>
<url>
<loc>http://samza.incubator.apache.org/learn/tutorials/0.7.0/run-hello-samza-without-internet.html</loc>
- <lastmod>2014-05-23</lastmod>
+ <lastmod>2014-06-09</lastmod>
</url>
<url>
<loc>http://samza.incubator.apache.org/learn/tutorials/0.7.0/run-in-multi-node-yarn.html</loc>
- <lastmod>2014-05-23</lastmod>
+ <lastmod>2014-06-09</lastmod>
</url>
<url>
<loc>http://samza.incubator.apache.org/startup/download/index.html</loc>
- <lastmod>2014-05-23</lastmod>
+ <lastmod>2014-06-09</lastmod>
</url>
<url>
<loc>http://samza.incubator.apache.org/startup/hello-samza/0.7.0/index.html</loc>
- <lastmod>2014-05-23</lastmod>
+ <lastmod>2014-06-09</lastmod>
</url>