You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@heron.apache.org by hu...@apache.org on 2018/05/04 22:30:40 UTC
[incubator-heron] branch huijunwu/submitremotejar updated (a06f51b
-> a35cfcb)
This is an automated email from the ASF dual-hosted git repository.
huijun pushed a change to branch huijunwu/submitremotejar
in repository https://gitbox.apache.org/repos/asf/incubator-heron.git.
from a06f51b revert comment
add cf976de move downloader registry map to yaml
add b710463 fix neng comment
add d56d91b fix path
add 499c978 fix type
new 3e8a22d Merge branch 'huijunw/refactordownloader' into huijunwu/submitremotejar
new a35cfcb fix downloader command
The 2 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
heron/config/src/yaml/conf/aurora/downloader.yaml | 6 +
.../config/src/yaml/conf/examples/downloader.yaml | 6 +
.../src/yaml/conf/kubernetes/downloader.yaml | 6 +
heron/config/src/yaml/conf/local/downloader.yaml | 6 +
heron/config/src/yaml/conf/localzk/downloader.yaml | 6 +
heron/config/src/yaml/conf/nomad/downloader.yaml | 6 +
heron/config/src/yaml/conf/sandbox/downloader.yaml | 6 +
.../src/yaml/conf/standalone/downloader.yaml | 6 +
heron/config/src/yaml/conf/yarn/downloader.yaml | 6 +
heron/downloaders/src/java/BUILD | 1 +
.../apache/heron/downloader/DownloadRunner.java | 154 ++++++++++++++++++++-
.../java/org/apache/heron/downloader/Registry.java | 35 ++---
.../org/apache/heron/downloader/RegistryTest.java | 30 +++-
.../org/apache/heron/spi/common/ConfigLoader.java | 2 +
.../java/org/apache/heron/spi/common/Context.java | 4 +
.../src/java/org/apache/heron/spi/common/Key.java | 23 +--
.../apache/heron/spi/common/ConfigLoaderTest.java | 5 +-
heron/tools/cli/src/python/submit.py | 18 ++-
scripts/packages/BUILD | 8 ++
tools/rules/heron_client.bzl | 6 +
20 files changed, 291 insertions(+), 49 deletions(-)
create mode 100644 heron/config/src/yaml/conf/aurora/downloader.yaml
create mode 100644 heron/config/src/yaml/conf/examples/downloader.yaml
create mode 100644 heron/config/src/yaml/conf/kubernetes/downloader.yaml
create mode 100644 heron/config/src/yaml/conf/local/downloader.yaml
create mode 100644 heron/config/src/yaml/conf/localzk/downloader.yaml
create mode 100644 heron/config/src/yaml/conf/nomad/downloader.yaml
create mode 100644 heron/config/src/yaml/conf/sandbox/downloader.yaml
create mode 100644 heron/config/src/yaml/conf/standalone/downloader.yaml
create mode 100644 heron/config/src/yaml/conf/yarn/downloader.yaml
--
To stop receiving notification emails like this one, please contact
huijun@apache.org.
[incubator-heron] 02/02: fix downloader command
Posted by hu...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
huijun pushed a commit to branch huijunwu/submitremotejar
in repository https://gitbox.apache.org/repos/asf/incubator-heron.git
commit a35cfcb8b87914ab8f815630b47df777a12bdd26
Author: Huijun Wu <hu...@twitter.com>
AuthorDate: Fri May 4 15:29:50 2018 -0700
fix downloader command
---
heron/tools/cli/src/python/submit.py | 18 +++++++++++++-----
1 file changed, 13 insertions(+), 5 deletions(-)
diff --git a/heron/tools/cli/src/python/submit.py b/heron/tools/cli/src/python/submit.py
index 5015145..9dc13cd 100644
--- a/heron/tools/cli/src/python/submit.py
+++ b/heron/tools/cli/src/python/submit.py
@@ -368,11 +368,17 @@ def submit_cpp(cl_args, unknown_args, tmp_dir):
return launch_topologies(cl_args, topology_file, tmp_dir)
-def download(uri):
- current_path = os.path.realpath(__file__)
- print current_path
+def download(uri, cluster):
tmp_dir = tempfile.mkdtemp()
- subprocess.call([current_path + "downloader.sh", uri, tmp_dir])
+ cmd_downloader = config.get_heron_bin_dir() + "/heron-downloader.sh"
+ cmd_uri = "-u " + uri
+ cmd_destination = "-f " + tmp_dir
+ cmd_heron_root = "-d " + config.get_heron_dir()
+ cmd_heron_config = "-p " + config.get_heron_cluster_conf_dir(cluster, config.get_heron_conf_dir())
+ cmd_mode = "-m local"
+ cmd = [cmd_downloader, cmd_uri, cmd_destination, cmd_heron_root, cmd_heron_config, cmd_mode]
+ Log.debug("download uri command: %s", cmd)
+ subprocess.call(cmd)
suffix = (".jar", ".tar", ".tar.gz", ".pex", ".dylib", ".so")
for f in os.listdir(tmp_dir):
if f.endswith(suffix):
@@ -400,7 +406,9 @@ def run(command, parser, cl_args, unknown_args):
topology_file = cl_args['topology-file-name']
if urlparse.urlparse(topology_file).scheme:
- topology_file = download(topology_file)
+ cl_args['topology-file-name'] = download(topology_file, cl_args['cluster'])
+ topology_file = cl_args['topology-file-name']
+ Log.debug("download uri to local file: %s", topology_file)
# check to see if the topology file exists
if not os.path.isfile(topology_file):
--
To stop receiving notification emails like this one, please contact
huijun@apache.org.
[incubator-heron] 01/02: Merge branch 'huijunw/refactordownloader'
into huijunwu/submitremotejar
Posted by hu...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
huijun pushed a commit to branch huijunwu/submitremotejar
in repository https://gitbox.apache.org/repos/asf/incubator-heron.git
commit 3e8a22d69701983d5a98ad35e2ee075a1935105d
Merge: a06f51b 499c978
Author: Huijun Wu <hu...@twitter.com>
AuthorDate: Fri May 4 14:58:52 2018 -0700
Merge branch 'huijunw/refactordownloader' into huijunwu/submitremotejar
heron/config/src/yaml/conf/aurora/downloader.yaml | 6 +
.../config/src/yaml/conf/examples/downloader.yaml | 6 +
.../src/yaml/conf/kubernetes/downloader.yaml | 6 +
heron/config/src/yaml/conf/local/downloader.yaml | 6 +
heron/config/src/yaml/conf/localzk/downloader.yaml | 6 +
heron/config/src/yaml/conf/nomad/downloader.yaml | 6 +
heron/config/src/yaml/conf/sandbox/downloader.yaml | 6 +
.../src/yaml/conf/standalone/downloader.yaml | 6 +
heron/config/src/yaml/conf/yarn/downloader.yaml | 6 +
heron/downloaders/src/java/BUILD | 1 +
.../apache/heron/downloader/DownloadRunner.java | 154 ++++++++++++++++++++-
.../java/org/apache/heron/downloader/Registry.java | 35 ++---
.../org/apache/heron/downloader/RegistryTest.java | 30 +++-
.../org/apache/heron/spi/common/ConfigLoader.java | 2 +
.../java/org/apache/heron/spi/common/Context.java | 4 +
.../src/java/org/apache/heron/spi/common/Key.java | 23 +--
.../apache/heron/spi/common/ConfigLoaderTest.java | 5 +-
scripts/packages/BUILD | 8 ++
tools/rules/heron_client.bzl | 6 +
19 files changed, 278 insertions(+), 44 deletions(-)
--
To stop receiving notification emails like this one, please contact
huijun@apache.org.