You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-commits@hadoop.apache.org by wa...@apache.org on 2019/01/06 19:16:57 UTC
[hadoop] branch branch-3.2 updated (71bee05 -> cffe5c1)
This is an automated email from the ASF dual-hosted git repository.
wangda pushed a change to branch branch-3.2
in repository https://gitbox.apache.org/repos/asf/hadoop.git.
from 71bee05 Revert "HADOOP-15759. AliyunOSS: Update oss-sdk version to 3.0.0. Contributed by Jinhu Wu."
new 1ad1ab5 YARN-9141. [Submarine] JobStatus outputs with system UTC clock, not local clock. (Zac Zhou via wangda)
new cffe5c1 YARN-9160. [Submarine] Document 'PYTHONPATH' environment variable setting when using -localization options. (Zhankun Tang via wangda)
The 2 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.../hadoop/yarn/submarine/common/api/JobStatus.java | 4 ++--
.../src/site/markdown/QuickStart.md | 18 ++++++++++++++++++
2 files changed, 20 insertions(+), 2 deletions(-)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-commits-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-commits-help@hadoop.apache.org
[hadoop] 01/02: YARN-9141. [Submarine] JobStatus outputs with
system UTC clock, not local clock. (Zac Zhou via wangda)
Posted by wa...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
wangda pushed a commit to branch branch-3.2
in repository https://gitbox.apache.org/repos/asf/hadoop.git
commit 1ad1ab5d7a2a292039ccb06ae659f691e9ef63aa
Author: Wangda Tan <wa...@apache.org>
AuthorDate: Sun Jan 6 10:24:29 2019 -0800
YARN-9141. [Submarine] JobStatus outputs with system UTC clock, not local clock. (Zac Zhou via wangda)
Change-Id: Ib2b17cf7860ddc9188f06498086de3326bc09519
(cherry picked from commit 8f004fe4e0d479e26d0063427566edc1606bced5)
---
.../java/org/apache/hadoop/yarn/submarine/common/api/JobStatus.java | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-submarine/src/main/java/org/apache/hadoop/yarn/submarine/common/api/JobStatus.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-submarine/src/main/java/org/apache/hadoop/yarn/submarine/common/api/JobStatus.java
index 6e390f3..0ba9de9 100644
--- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-submarine/src/main/java/org/apache/hadoop/yarn/submarine/common/api/JobStatus.java
+++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-submarine/src/main/java/org/apache/hadoop/yarn/submarine/common/api/JobStatus.java
@@ -18,7 +18,7 @@
package org.apache.hadoop.yarn.submarine.common.api;
import java.io.PrintStream;
-import java.time.Instant;
+import java.time.LocalDateTime;
import java.util.List;
/**
@@ -34,7 +34,7 @@ public class JobStatus {
public void nicePrint(PrintStream out) {
out.println(
"Job Name=" + this.jobName + ", status=" + state.name() + " time="
- + Instant.now());
+ + LocalDateTime.now());
if (JobState.isFinal(this.state)) {
return;
}
---------------------------------------------------------------------
To unsubscribe, e-mail: common-commits-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-commits-help@hadoop.apache.org
[hadoop] 02/02: YARN-9160. [Submarine] Document 'PYTHONPATH'
environment variable setting when using -localization options. (Zhankun
Tang via wangda)
Posted by wa...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
wangda pushed a commit to branch branch-3.2
in repository https://gitbox.apache.org/repos/asf/hadoop.git
commit cffe5c1ba089cb6a720f479a40a8e300127caa44
Author: Wangda Tan <wa...@apache.org>
AuthorDate: Sun Jan 6 10:25:35 2019 -0800
YARN-9160. [Submarine] Document 'PYTHONPATH' environment variable setting when using -localization options. (Zhankun Tang via wangda)
Change-Id: I115b6710eb11501e21ab3e9e7cd1a87f322e5cf8
(cherry picked from commit 2c02aa6ec259128934cc5468cf66104a624d88a7)
---
.../src/site/markdown/QuickStart.md | 18 ++++++++++++++++++
1 file changed, 18 insertions(+)
diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-submarine/src/site/markdown/QuickStart.md b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-submarine/src/site/markdown/QuickStart.md
index 8e7a956..21db6bb 100644
--- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-submarine/src/site/markdown/QuickStart.md
+++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-submarine/src/site/markdown/QuickStart.md
@@ -87,6 +87,24 @@ usage: job run
-localization "/user/yarn/mydir3:/opt/mydir3"
-localization "./mydir1:."
```
+
+#### Notes:
+When using `localization` option to make a collection of dependency Python
+scripts available to entry python script in the container, you may also need to
+ set `PYTHONPATH` environment variable as below to avoid module import error
+reported from `entry_script.py`.
+
+```
+... job run
+ # the entry point
+ --localization entry_script.py:<path>/entry_script.py
+ # the dependency Python scripts of the entry point
+ --localization other_scripts_dir:<path>/other_scripts_dir
+ # the PYTHONPATH env to make dependency available to entry script
+ --env PYTHONPATH="<path>/other_scripts_dir"
+ --worker_launch_cmd "python <path>/entry_script.py ..."
+```
+
### Submarine Configuration
For submarine internal configuration, please create a `submarine.xml` which should be placed under `$HADOOP_CONF_DIR`.
---------------------------------------------------------------------
To unsubscribe, e-mail: common-commits-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-commits-help@hadoop.apache.org