You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@samza.apache.org by ja...@apache.org on 2017/05/22 22:09:36 UTC
samza git commit: SAMZA-1280;
document for the general/universal resource localization in YARN
Repository: samza
Updated Branches:
refs/heads/master f758fb591 -> 2a75a209b
SAMZA-1280; document for the general/universal resource localization in YARN
This PR added a MD for localizing resource in Samza on YARN by configuring path, local.name, local.type and local.visibility, and also updated the configuration table and index table.
Author: Fred Ji <fj...@linkedin.com>
Reviewers: Navina Ramesh <na...@apache.org>
Closes #191 from fredji97/resourceLocalizationDoc
Project: http://git-wip-us.apache.org/repos/asf/samza/repo
Commit: http://git-wip-us.apache.org/repos/asf/samza/commit/2a75a209
Tree: http://git-wip-us.apache.org/repos/asf/samza/tree/2a75a209
Diff: http://git-wip-us.apache.org/repos/asf/samza/diff/2a75a209
Branch: refs/heads/master
Commit: 2a75a209b25fb68b8a9cb217fa909640ea1f1d98
Parents: f758fb5
Author: Fred Ji <fj...@linkedin.com>
Authored: Mon May 22 15:09:32 2017 -0700
Committer: vjagadish1989 <jv...@linkedin.com>
Committed: Mon May 22 15:09:32 2017 -0700
----------------------------------------------------------------------
docs/learn/documentation/versioned/index.html | 3 +-
.../versioned/jobs/configuration-table.html | 43 +++++++-
.../versioned/yarn/yarn-host-affinity.md | 2 +-
.../yarn/yarn-resource-localization.md | 108 +++++++++++++++++++
.../versioned/yarn/yarn-security.md | 2 +-
5 files changed, 154 insertions(+), 4 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/samza/blob/2a75a209/docs/learn/documentation/versioned/index.html
----------------------------------------------------------------------
diff --git a/docs/learn/documentation/versioned/index.html b/docs/learn/documentation/versioned/index.html
index e3cecc1..a710383 100644
--- a/docs/learn/documentation/versioned/index.html
+++ b/docs/learn/documentation/versioned/index.html
@@ -81,9 +81,10 @@ title: Documentation
<li><a href="yarn/application-master.html">Application Master</a></li>
<li><a href="yarn/isolation.html">Isolation</a></li>
<li><a href="yarn/yarn-host-affinity.html">Host Affinity & Yarn</a></li>
+ <li><a href="yarn/yarn-resource-localization.html">Resource Localization</a></li>
+ <li><a href="yarn/yarn-security.html">Yarn Security</a></li>
<li><a href="hdfs/producer.html">Writing to HDFS</a></li>
<li><a href="hdfs/consumer.html">Reading from HDFS</a></li>
- <li><a href="yarn/yarn-security.html">Yarn Security</a></li>
<!-- TODO write yarn pages
<li><a href="">Fault Tolerance</a></li>
<li><a href="">Security</a></li>
http://git-wip-us.apache.org/repos/asf/samza/blob/2a75a209/docs/learn/documentation/versioned/jobs/configuration-table.html
----------------------------------------------------------------------
diff --git a/docs/learn/documentation/versioned/jobs/configuration-table.html b/docs/learn/documentation/versioned/jobs/configuration-table.html
index fe1580f..f34146c 100644
--- a/docs/learn/documentation/versioned/jobs/configuration-table.html
+++ b/docs/learn/documentation/versioned/jobs/configuration-table.html
@@ -59,7 +59,7 @@
font-family: monospace;
}
- span.system, span.stream, span.store, span.serde, span.rewriter, span.listener, span.reporter {
+ span.system, span.stream, span.store, span.serde, span.rewriter, span.listener, span.reporter, span.resource {
padding: 1px;
margin: 1px;
border-width: 1px;
@@ -101,6 +101,11 @@
background-color: #dff;
border-color: #bdd;
}
+
+ span.resource {
+ background-color: #ded;
+ border-color: #bcb;
+ }
</style>
</head>
@@ -1935,6 +1940,42 @@
</tr>
<tr>
+ <td class="property" id="yarn-resources-resource-name-path">yarn.resources.<span class="resource">resource-name</span>.path</td>
+ <td class="default"></td>
+ <td class="description">
+ The path for localizing the resource for <span class="resource">resource-name</span>. The scheme (e.g. http, ftp, hdsf, file, etc) in the path should be configured in YARN core-site.xml as fs.<scheme>.impl and is associated with a <a href="https://hadoop.apache.org/docs/stable/api/index.html?org/apache/hadoop/fs/FileSystem.html">FileSystem</a>.
+ If defined, the resource will be localized in the Samza application directory before the Samza job runs. More details can be found <a href="../yarn/yarn-resource-localization.html">here</a>.
+ </td>
+ </tr>
+
+ <tr>
+ <td class="property" id="yarn-resources-resource-name-local-name">yarn.resources.<span class="resource">resource-name</span>.local.name</td>
+ <td class="default"><span class="resource">resource-name</span></td>
+ <td class="description">
+ The new local name for the resource after localization.
+ This configuration only applies when yarn.resources.<span class="resource">resource-name</span>.path is configured.
+ </td>
+ </tr>
+
+ <tr>
+ <td class="property" id="yarn-resources-resource-name-local-type">yarn.resources.<span class="resource">resource-name</span>.local.type</td>
+ <td class="default">FILE</td>
+ <td class="description">
+ The type for the resource after localization. It can be ARCHIVE (archived directory), FILE, or PATTERN (the entries extracted from the archive with the pattern).
+ This configuration only applies when yarn.resources.<span class="resource">resource-name</span>.path is configured.
+ </td>
+ </tr>
+
+ <tr>
+ <td class="property" id="yarn-resources-resource-name-local-visibility">yarn.resources.<span class="resource">resource-name</span>.local.visibility</td>
+ <td class="default">APPLICATION</td>
+ <td class="description">
+ The visibility for the resource after localization. It can be PUBLIC (visible to everyone), PRIVATE (visible to all Samza applications of the same account user as this application), or APPLICATION (visible to only this Samza application).
+ This configuration only applies when yarn.resources.<span class="resource">resource-name</span>.path is configured.
+ </td>
+ </tr>
+
+ <tr>
<th colspan="3" class="section" id="metrics"><a href="../container/metrics.html">Metrics</a></th>
</tr>
http://git-wip-us.apache.org/repos/asf/samza/blob/2a75a209/docs/learn/documentation/versioned/yarn/yarn-host-affinity.md
----------------------------------------------------------------------
diff --git a/docs/learn/documentation/versioned/yarn/yarn-host-affinity.md b/docs/learn/documentation/versioned/yarn/yarn-host-affinity.md
index 13bc9b9..14e10cc 100644
--- a/docs/learn/documentation/versioned/yarn/yarn-host-affinity.md
+++ b/docs/learn/documentation/versioned/yarn/yarn-host-affinity.md
@@ -119,4 +119,4 @@ As you have observed, host-affinity cannot be guaranteed all the time due to var
1. _When the number of containers and/or container-task assignment changes across successive application runs_ - We may be able to re-use local state for a subset of partitions. Currently, there is no logic in the Job Coordinator to handle partitioning of tasks among containers intelligently. Handling this is more involved as relates to [auto-scaling](https://issues.apache.org/jira/browse/SAMZA-336) of the containers. However, with [task-container mapping](https://issues.apache.org/jira/browse/SAMZA-906), this will work better for typical container count adjustments.
2. _When SystemStreamPartitionGrouper changes across successive application runs_ - When the grouper logic used to distribute the partitions across containers changes, the data in the Coordinator Stream (for changelog-task partition assignment etc) and the data stores becomes invalid. Thus, to be safe, we should flush out all state-related data from the Coordinator Stream. An alternative is to overwrite the Task-ChangelogPartition assignment message and the Container Locality message in the Coordinator Stream, before starting up the job again.
-## [Writing to HDFS »](../hdfs/producer.html)
\ No newline at end of file
+## [Resource Localization »](../operations/resource-localization.html)
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/samza/blob/2a75a209/docs/learn/documentation/versioned/yarn/yarn-resource-localization.md
----------------------------------------------------------------------
diff --git a/docs/learn/documentation/versioned/yarn/yarn-resource-localization.md b/docs/learn/documentation/versioned/yarn/yarn-resource-localization.md
new file mode 100644
index 0000000..a55670b
--- /dev/null
+++ b/docs/learn/documentation/versioned/yarn/yarn-resource-localization.md
@@ -0,0 +1,108 @@
+---
+layout: page
+title: YARN Resource Localization
+---
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+When Samza jobs run on YARN clusters, sometimes there are needs to preload some files or data (called as resources in this doc) before job starts, such as preparing the job package, fetching job certificate, or etc., Samza supports a general configuration way to localize difference resources.
+
+### Resource Localization Process
+
+For the Samza jobs running on YARN, the resource localization leverages the YARN node manager localization service. Here is a good [deep dive](https://hortonworks.com/blog/resource-localization-in-yarn-deep-dive/) from Horton Works on how localization works in YARN.
+
+Depending on where and how the resource comes from, fetching the resource is associated with a scheme in the path, such as `http`, `https`, `hdfs`, `ftp`, `file`, etc., which maps to a certain FileSystem for handling the localization.
+
+If there is an implementation of [FileSystem](https://hadoop.apache.org/docs/stable/api/index.html?org/apache/hadoop/fs/FileSystem.html) on YARN supporting a scheme, then that scheme can be used for resource localization.
+
+There are some predefined file systems in Hadoop or Samza, which are provided if you run Samza jobs on YARN:
+
+* `org.apache.samza.util.hadoop.HttpFileSystem`: used for fetching resources based on http, or https without client side authentication requirement.
+* `org.apache.hadoop.hdfs.DistributedFileSystem`: used for fetching resource from DFS system on Hadoop.
+* `org.apache.hadoop.fs.LocalFileSystem`: used for copying resources from local file system to the job directory.
+* `org.apache.hadoop.fs.ftp.FTPFileSystem`: used for fetching resources based on ftp.
+* ...
+
+If you would like to have your own file system, you need to implement a class which extends from `org.apache.hadoop.fs.FileSystem`.
+
+### Job Configuration
+With the configuration properly defined, the resources a job requiring from external or internal locations may be prepared automatically before it runs.
+
+For each resource with the name `<resourceName>` in the Samza job, the following set of job configurations are used when running on a YARN cluster. The first one which definiing resource path is required, but the others are optional and they have default values.
+
+1. `yarn.resources.<resourceName>.path`
+ * Required
+ * The path for fetching the resource for localization, e.g. http://hostname.com/packages/mySamzaJob
+2. `yarn.resources.<resourceName>.local.name`
+ * Optional
+ * The local name used for the localized resource.
+ * If not set, the default one will be `<resourceName>` from the config key.
+3. `yarn.resources.<resourceName>.local.type`
+ * Optional
+ * Localized resource type with valid values from: `ARCHIVE`, `FILE`, `PATTERN`.
+ * ARCHIVE: the localized resource will be an archived directory;
+ * FILE: the localized resource will be a file;
+ * PATTERN: the localized resource will be the entries extracted from the archive with the pattern.
+ * If not set, the default value is `FILE`.
+4. `yarn.resources.<resourceName>.local.visibility`
+ * Optional
+ * Localized resource visibility for the resource, and it can be a value from `PUBLIC`, `PRIVATE`, `APPLICATION`
+ * PUBLIC: visible to everyone
+ * PRIVATE: visible to just the account which runs the job
+ * APPLICATION: visible only to the specific application job which has the resource configuration
+ * If not set, the default value is `APPLICATION`
+
+It is up to you how to name the resource, but `<resourceName>` should be the same in the above configurations to apply to the same resource.
+
+### YARN Configuration
+Make sure the scheme used in the yarn.resources.<resourceName>.path is configured in YARN core-site.xml with a FileSystem implementation. For example, for scheme `http`, you should have the following property in YARN core-site.xml:
+
+{% highlight xml %}
+<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
+<configuration>
+ <property>
+ <name>fs.http.impl</name>
+ <value>org.apache.samza.util.hadoop.HttpFileSystem</value>
+ </property>
+</configuration>
+{% endhighlight %}
+
+You can override a behavior for a scheme by linking it to another file system. For example, you have a special need for localizing a resource for your job through http request, you may implement your own Http File System by extending [FileSystem](https://hadoop.apache.org/docs/stable/api/index.html?org/apache/hadoop/fs/FileSystem.html), and have the following configuration:
+
+{% highlight xml %}
+<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
+<configuration>
+ <property>
+ <name>fs.http.impl</name>
+ <value>com.myCompany.MyHttpFileSystem</value>
+ </property>
+</configuration>
+{% endhighlight %}
+
+If you are using other scheme which is not defined in Hadoop or Samza, for example, `yarn.resources.mySampleResource.path=myScheme://host.com/test`, in your job configuration, you may implement your own [FileSystem](https://hadoop.apache.org/docs/stable/api/index.html?org/apache/hadoop/fs/FileSystem.html) such as com.myCompany.MySchemeFileSystem and link it with your own scheme in yarn core-site.xml configuration.
+
+{% highlight xml %}
+<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
+<configuration>
+ <property>
+ <name>fs.myScheme.impl</name>
+ <value>com.myCompany.MySchemeFileSystem</value>
+ </property>
+</configuration>
+{% endhighlight %}
+
+## [Yarn Security »](../yarn/yarn-security.html)
http://git-wip-us.apache.org/repos/asf/samza/blob/2a75a209/docs/learn/documentation/versioned/yarn/yarn-security.md
----------------------------------------------------------------------
diff --git a/docs/learn/documentation/versioned/yarn/yarn-security.md b/docs/learn/documentation/versioned/yarn/yarn-security.md
index 8d164c6..7b66ed8 100644
--- a/docs/learn/documentation/versioned/yarn/yarn-security.md
+++ b/docs/learn/documentation/versioned/yarn/yarn-security.md
@@ -91,4 +91,4 @@ yarn.token.renewal.interval.seconds=86400
</property>
{% endhighlight %}
-## [Security »](../operations/security.html)
+## [Writing to HDFS »](../hdfs/producer.html)