You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@heron.apache.org by sa...@apache.org on 2022/05/09 21:35:55 UTC
[incubator-heron] branch saadurrahman/3829-Deprecate-Apache-Aurora-dev created (now 8cce8b38d2b)
This is an automated email from the ASF dual-hosted git repository.
saadurrahman pushed a change to branch saadurrahman/3829-Deprecate-Apache-Aurora-dev
in repository https://gitbox.apache.org/repos/asf/incubator-heron.git
at 8cce8b38d2b [MetricsMgr] Switched test to using Kubernetes in URI.
This branch includes the following new commits:
new bfd151cf140 [Sched] Removed Aurora from schedulers.
new dcb68dafd35 [Docs] Removed Aurora.
new 0a56102b59a [SPI] Removed key for Aurora scheduler class.
new 825e818d923 [Config] Removed Aurora from configs.
new 5c67b2f10d2 [Sched] Base64 string conversion comment updated.
new c2efb67f8b1 [Config] ConfigReader test switched to using Kubernetes.
new 8cce8b38d2b [MetricsMgr] Switched test to using Kubernetes in URI.
The 7 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
[incubator-heron] 06/07: [Config] ConfigReader test switched to using Kubernetes.
Posted by sa...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
saadurrahman pushed a commit to branch saadurrahman/3829-Deprecate-Apache-Aurora-dev
in repository https://gitbox.apache.org/repos/asf/incubator-heron.git
commit c2efb67f8b1f476d0bc2c13294c552601c751147
Author: Saad Ur Rahman <sa...@apache.org>
AuthorDate: Mon May 9 17:31:53 2022 -0400
[Config] ConfigReader test switched to using Kubernetes.
---
.../tests/java/org/apache/heron/common/config/ConfigReaderTest.java | 2 +-
heron/common/tests/resources/defaults.yaml | 2 +-
2 files changed, 2 insertions(+), 2 deletions(-)
diff --git a/heron/common/tests/java/org/apache/heron/common/config/ConfigReaderTest.java b/heron/common/tests/java/org/apache/heron/common/config/ConfigReaderTest.java
index ad15f07edb0..b2cdd798792 100644
--- a/heron/common/tests/java/org/apache/heron/common/config/ConfigReaderTest.java
+++ b/heron/common/tests/java/org/apache/heron/common/config/ConfigReaderTest.java
@@ -43,7 +43,7 @@ public class ConfigReaderTest {
private void testProperty(Map<String, Object> props) {
Assert.assertEquals("role", props.get(ROLE_KEY));
Assert.assertEquals("environ", props.get(ENVIRON_KEY));
- Assert.assertEquals("org.apache.heron.scheduler.aurora.AuroraLauncher",
+ Assert.assertEquals("org.apache.heron.scheduler.kubernetes.KubernetesLauncher",
props.get(LAUNCHER_CLASS_KEY));
Assert.assertNull(props.get(USER_KEY));
}
diff --git a/heron/common/tests/resources/defaults.yaml b/heron/common/tests/resources/defaults.yaml
index e07336c1e96..d365784ac12 100644
--- a/heron/common/tests/resources/defaults.yaml
+++ b/heron/common/tests/resources/defaults.yaml
@@ -17,4 +17,4 @@
role: role
environ: environ
-heron.launcher.class: org.apache.heron.scheduler.aurora.AuroraLauncher
+heron.launcher.class: org.apache.heron.scheduler.kubernetes.KubernetesLauncher
[incubator-heron] 05/07: [Sched] Base64 string conversion comment updated.
Posted by sa...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
saadurrahman pushed a commit to branch saadurrahman/3829-Deprecate-Apache-Aurora-dev
in repository https://gitbox.apache.org/repos/asf/incubator-heron.git
commit 5c67b2f10d2a103458ebef7b4bfba9c21b484308
Author: Saad Ur Rahman <sa...@apache.org>
AuthorDate: Mon May 9 17:25:05 2022 -0400
[Sched] Base64 string conversion comment updated.
Replacement of "=" with "(61)" left in case it causes any issues in the future.
---
.../src/java/org/apache/heron/scheduler/utils/SchedulerUtils.java | 1 -
1 file changed, 1 deletion(-)
diff --git a/heron/scheduler-core/src/java/org/apache/heron/scheduler/utils/SchedulerUtils.java b/heron/scheduler-core/src/java/org/apache/heron/scheduler/utils/SchedulerUtils.java
index ad6ac5b5a9e..fe042c9b015 100644
--- a/heron/scheduler-core/src/java/org/apache/heron/scheduler/utils/SchedulerUtils.java
+++ b/heron/scheduler-core/src/java/org/apache/heron/scheduler/utils/SchedulerUtils.java
@@ -457,7 +457,6 @@ public final class SchedulerUtils {
* <br> 1. Convert it into Base64 format
* <br> 2. Add \" at the start and at the end
* <br> 3. replace "=" with "(61)" and "&equals;"
- * '=' can be parsed in a wrong way by some schedulers (aurora) hence it needs to be escaped.
*
* @return encoded string
*/
[incubator-heron] 04/07: [Config] Removed Aurora from configs.
Posted by sa...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
saadurrahman pushed a commit to branch saadurrahman/3829-Deprecate-Apache-Aurora-dev
in repository https://gitbox.apache.org/repos/asf/incubator-heron.git
commit 825e818d92376de5ada58869ffd0c12771474ed6
Author: Saad Ur Rahman <sa...@apache.org>
AuthorDate: Mon May 9 17:21:59 2022 -0400
[Config] Removed Aurora from configs.
---
heron/config/src/yaml/BUILD | 15 +-
heron/config/src/yaml/conf/aurora/README | 1 -
heron/config/src/yaml/conf/aurora/client.yaml | 23 --
heron/config/src/yaml/conf/aurora/downloader.yaml | 23 --
heron/config/src/yaml/conf/aurora/heron.aurora | 93 -------
.../src/yaml/conf/aurora/heron_internals.yaml | 296 ---------------------
.../config/src/yaml/conf/aurora/metrics_sinks.yaml | 257 ------------------
heron/config/src/yaml/conf/aurora/packing.yaml | 20 --
heron/config/src/yaml/conf/aurora/scheduler.yaml | 61 -----
heron/config/src/yaml/conf/aurora/stateful.yaml | 38 ---
heron/config/src/yaml/conf/aurora/statemgr.yaml | 49 ----
heron/config/src/yaml/conf/aurora/uploader.yaml | 22 --
heron/config/src/yaml/conf/examples/README.md | 3 -
.../src/yaml/conf/examples/aurora_scheduler.yaml | 29 --
14 files changed, 3 insertions(+), 927 deletions(-)
diff --git a/heron/config/src/yaml/BUILD b/heron/config/src/yaml/BUILD
index aba545eccdd..2cdcf392e35 100644
--- a/heron/config/src/yaml/BUILD
+++ b/heron/config/src/yaml/BUILD
@@ -15,13 +15,9 @@ filegroup(
filegroup(
name = "conf-yaml",
srcs = glob(
- ["conf/**/*.aurora"] + [
- "conf/**/*.hcl",
- ] + [
- "conf/**/*.sh",
- ] + [
- "conf/**/*.yaml",
- ],
+ ["conf/**/*.hcl"] +
+ ["conf/**/*.sh"] +
+ ["conf/**/*.yaml"],
),
)
@@ -35,11 +31,6 @@ filegroup(
srcs = glob(["conf/sandbox/*.yaml"]),
)
-filegroup(
- name = "conf-aurora-yaml",
- srcs = glob(["conf/aurora/*"]),
-)
-
filegroup(
name = "conf-slurm-yaml",
srcs = glob(["conf/slurm/*"]),
diff --git a/heron/config/src/yaml/conf/aurora/README b/heron/config/src/yaml/conf/aurora/README
deleted file mode 100644
index 4a92300fe61..00000000000
--- a/heron/config/src/yaml/conf/aurora/README
+++ /dev/null
@@ -1 +0,0 @@
-Config customization is required before using.
\ No newline at end of file
diff --git a/heron/config/src/yaml/conf/aurora/client.yaml b/heron/config/src/yaml/conf/aurora/client.yaml
deleted file mode 100644
index f14d8fdd968..00000000000
--- a/heron/config/src/yaml/conf/aurora/client.yaml
+++ /dev/null
@@ -1,23 +0,0 @@
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing,
-# software distributed under the License is distributed on an
-# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-# KIND, either express or implied. See the License for the
-# specific language governing permissions and limitations
-# under the License.
-
-# location of the core package
-heron.package.core.uri: "file:///vagrant/.herondata/dist/heron-core-release.tar.gz"
-
-# Whether role/env is required to submit a topology. Default value is False.
-heron.config.is.role.required: True
-heron.config.is.env.required: True
diff --git a/heron/config/src/yaml/conf/aurora/downloader.yaml b/heron/config/src/yaml/conf/aurora/downloader.yaml
deleted file mode 100644
index 68805525670..00000000000
--- a/heron/config/src/yaml/conf/aurora/downloader.yaml
+++ /dev/null
@@ -1,23 +0,0 @@
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing,
-# software distributed under the License is distributed on an
-# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-# KIND, either express or implied. See the License for the
-# specific language governing permissions and limitations
-# under the License.
-
-# downloader class for protocols
-heron.downloader.registry:
- http: org.apache.heron.downloader.HttpDownloader
- https: org.apache.heron.downloader.HttpDownloader
- distributedlog: org.apache.heron.downloader.DLDownloader
- file: org.apache.heron.downloader.FileDownloader
diff --git a/heron/config/src/yaml/conf/aurora/heron.aurora b/heron/config/src/yaml/conf/aurora/heron.aurora
deleted file mode 100644
index 1826e7faa61..00000000000
--- a/heron/config/src/yaml/conf/aurora/heron.aurora
+++ /dev/null
@@ -1,93 +0,0 @@
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing,
-# software distributed under the License is distributed on an
-# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-# KIND, either express or implied. See the License for the
-# specific language governing permissions and limitations
-# under the License.
-
-"""
-Launch the topology as a single aurora job with multiple instances.
-The heron-executor is responsible for starting a tmanager (container 0)
-and regular stmgr/metricsmgr/instances (container index > 0).
-"""
-
-heron_core_release_uri = '{{CORE_PACKAGE_URI}}'
-heron_topology_jar_uri = '{{TOPOLOGY_PACKAGE_URI}}'
-core_release_file = "heron-core.tar.gz"
-topology_package_file = "topology.tar.gz"
-
-# --- processes ---
-fetch_heron_system = Process(
- name = 'fetch_heron_system',
- cmdline = 'curl %s -o %s && tar zxf %s' % (heron_core_release_uri, core_release_file, core_release_file)
-)
-
-fetch_user_package = Process(
- name = 'fetch_user_package',
- cmdline = 'curl %s -o %s && tar zxf %s' % (heron_topology_jar_uri, topology_package_file, topology_package_file)
-)
-
-command_to_start_executor = \
- '{{EXECUTOR_BINARY}}' \
- ' --shard={{mesos.instance}}' \
- ' --server-port={{thermos.ports[port1]}}' \
- ' --tmanager-controller-port={{thermos.ports[port2]}}' \
- ' --tmanager-stats-port={{thermos.ports[port3]}}' \
- ' --shell-port={{thermos.ports[http]}}' \
- ' --metrics-manager-port={{thermos.ports[port4]}}' \
- ' --scheduler-port={{thermos.ports[scheduler]}}' \
- ' --metricscache-manager-server-port={{thermos.ports[metricscachemgr_serverport]}}' \
- ' --metricscache-manager-stats-port={{thermos.ports[metricscachemgr_statsport]}}' \
- ' --checkpoint-manager-port={{thermos.ports[ckptmgr_port]}}' \
- ' {{TOPOLOGY_ARGUMENTS}}'
-
-launch_heron_executor = Process(
- name = 'launch_heron_executor',
- cmdline = command_to_start_executor,
- max_failures = 1
-)
-
-discover_profiler_port = Process(
- name = 'discover_profiler_port',
- cmdline = 'echo {{thermos.ports[yourkit]}} > yourkit.port'
-)
-
-# --- tasks ---
-heron_task = SequentialTask(
- name = 'setup_and_run',
- processes = [fetch_heron_system, fetch_user_package, launch_heron_executor, discover_profiler_port],
- resources = Resources(cpu = '{{CPUS_PER_CONTAINER}}', ram = '{{RAM_PER_CONTAINER}}', disk = '{{DISK_PER_CONTAINER}}')
-)
-
-# -- jobs ---
-jobs = [
- Job(
- name = '{{TOPOLOGY_NAME}}',
- cluster = '{{CLUSTER}}',
- role = '{{ROLE}}',
- environment = '{{ENVIRON}}',
- service = True,
- task = heron_task,
- instances = '{{NUM_CONTAINERS}}',
- announce = Announcer(primary_port = 'http', portmap = {'admin': 'http', 'health': 'http'}),
- health_check_config = HealthCheckConfig(
- health_checker = HealthCheckerConfig(
- http = HttpHealthChecker(endpoint='/health', expected_response='ok', expected_response_code=200)
- ),
- initial_interval_secs=30,
- interval_secs=10,
- max_consecutive_failures=2,
- timeout_secs=5
- )
- )
-]
diff --git a/heron/config/src/yaml/conf/aurora/heron_internals.yaml b/heron/config/src/yaml/conf/aurora/heron_internals.yaml
deleted file mode 100644
index 22685e4e1a1..00000000000
--- a/heron/config/src/yaml/conf/aurora/heron_internals.yaml
+++ /dev/null
@@ -1,296 +0,0 @@
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing,
-# software distributed under the License is distributed on an
-# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-# KIND, either express or implied. See the License for the
-# specific language governing permissions and limitations
-# under the License.
-
-################################################################################
-# Default values for various configs used inside Heron.
-################################################################################
-# All the config associated with time is in the unit of milli-seconds,
-# unless otherwise specified.
-################################################################################
-# All the config associated with data size is in the unit of bytes, unless
-# otherwise specified.
-################################################################################
-
-################################################################################
-# System level configs
-################################################################################
-
-### heron.* configs are general configurations over all componenets
-
-# The relative path to the logging directory
-heron.logging.directory: "log-files"
-
-# The maximum log file size in MB
-heron.logging.maximum.size.mb: 100
-
-# The maximum number of log files
-heron.logging.maximum.files: 5
-
-# The interval in seconds after which to check if the tmanager location has been fetched or not
-heron.check.tmanager.location.interval.sec: 120
-
-# The interval in seconds to prune logging files in C++
-heron.logging.prune.interval.sec: 300
-
-# The interval in seconds to flush log files in C++
-heron.logging.flush.interval.sec: 10
-
-# The threshold level to log error
-heron.logging.err.threshold: 3
-
-# The interval in seconds for different components to export metrics to metrics manager
-heron.metrics.export.interval.sec: 60
-
-# The maximum count of exceptions in one MetricPublisherPublishMessage protobuf
-heron.metrics.max.exceptions.per.message.count: 1024
-
-################################################################################
-# Configs related to Stream Manager, starts with heron.streammgr.*
-################################################################################
-
-# The tuple cache (used for batching) can be drained in two ways:
-# (a) Time based
-# (b) size based
-
-# The frequency in ms to drain the tuple cache in stream manager
-heron.streammgr.cache.drain.frequency.ms: 10
-
-# The sized based threshold in MB for buffering data tuples
-# waiting for checkpoint markers before giving up
-heron.streammgr.stateful.buffer.size.mb: 100
-
-# The sized based threshold in MB for draining the tuple cache
-heron.streammgr.cache.drain.size.mb: 100
-
-# For efficient acknowledgements
-heron.streammgr.xormgr.rotatingmap.nbuckets: 3
-
-# The max number of messages in the memory pool for each message type
-heron.streammgr.mempool.max.message.number: 512
-
-# The reconnect interval to other stream managers in secs for stream manager client
-heron.streammgr.client.reconnect.interval.sec: 1
-
-# The reconnect interval to tamster in second for stream manager client
-heron.streammgr.client.reconnect.tmanager.interval.sec: 10
-
-# The max reconnect attempts to tmanager for stream manager client
-heron.streammgr.client.reconnect.tmanager.max.attempts: 30
-
-# The maximum packet size in MB of stream manager's network options
-heron.streammgr.network.options.maximum.packet.mb: 10
-
-# The interval in seconds to send heartbeat
-heron.streammgr.tmanager.heartbeat.interval.sec: 10
-
-# Maximum batch size in MB to read by stream manager from socket
-heron.streammgr.connection.read.batch.size.mb: 1
-
-# Maximum batch size in MB to write by stream manager to socket
-heron.streammgr.connection.write.batch.size.mb: 1
-
-# Number of times we should wait to see a buffer full while enqueueing data
-# before declaring start of back pressure
-heron.streammgr.network.backpressure.threshold: 3
-
-# High water mark on the num in MB that can be left outstanding on a connection
-heron.streammgr.network.backpressure.highwatermark.mb: 100
-
-# Low water mark on the num in MB that can be left outstanding on a connection
-heron.streammgr.network.backpressure.lowwatermark.mb: 50
-
-################################################################################
-# Configs related to Topology Manager, starts with heron.tmanager.*
-################################################################################
-
-# The maximum interval in minutes of metrics to be kept in tmanager
-heron.tmanager.metrics.collector.maximum.interval.min: 180
-
-# The maximum time to retry to establish the tmanager
-heron.tmanager.establish.retry.times: 30
-
-# The interval to retry to establish the tmanager
-heron.tmanager.establish.retry.interval.sec: 1
-
-# Maximum packet size in MB of tmanager's network options to connect to stream managers
-heron.tmanager.network.server.options.maximum.packet.mb: 16
-
-# Maximum packet size in MB of tmanager's network options to connect to scheduler
-heron.tmanager.network.controller.options.maximum.packet.mb: 1
-
-# Maximum packet size in MB of tmanager's network options for stat queries
-heron.tmanager.network.stats.options.maximum.packet.mb: 1
-
-# The interval for tmanager to purge metrics from socket
-heron.tmanager.metrics.collector.purge.interval.sec: 60
-
-# The maximum # of exceptions to be stored in tmetrics collector, to prevent potential OOM
-heron.tmanager.metrics.collector.maximum.exception: 256
-
-# Should the metrics reporter bind on all interfaces
-heron.tmanager.metrics.network.bindallinterfaces: False
-
-# The timeout in seconds for stream mgr, compared with (current time - last heartbeat time)
-heron.tmanager.stmgr.state.timeout.sec: 60
-
-################################################################################
-# Configs related to Metrics Manager, starts with heron.metricsmgr.*
-################################################################################
-
-# The size of packets to read from socket will be determined by the minimal of:
-# (a) time based
-# (b) size based
-
-# Time based, the maximum batch time in ms for metricsmgr to read from socket
-heron.metricsmgr.network.read.batch.time.ms: 16
-
-# Size based, the maximum batch size in bytes to read from socket
-heron.metricsmgr.network.read.batch.size.bytes: 32768
-
-# The size of packets to write to socket will be determined by the minimum of
-# (a) time based
-# (b) size based
-
-# Time based, the maximum batch time in ms for metricsmgr to write to socket
-heron.metricsmgr.network.write.batch.time.ms: 16
-
-# Size based, the maximum batch size in bytes to write to socket
-heron.metricsmgr.network.write.batch.size.bytes: 32768
-
-# The maximum socket's send buffer size in bytes
-heron.metricsmgr.network.options.socket.send.buffer.size.bytes: 6553600
-
-# The maximum socket's received buffer size in bytes of metricsmgr's network options
-heron.metricsmgr.network.options.socket.received.buffer.size.bytes: 8738000
-
-# The maximum packet size that metrics mgr can read
-heron.metricsmgr.network.options.maximum.packetsize.bytes: 1048576
-
-################################################################################
-# Configs related to Heron Instance, starts with heron.instance.*
-################################################################################
-
-# The maximum size of packets that instance can read
-heron.instance.network.options.maximum.packetsize.bytes: 10485760
-
-# The queue capacity (num of items) in bolt for buffer packets to read from stream manager
-heron.instance.internal.bolt.read.queue.capacity: 128
-
-# The queue capacity (num of items) in bolt for buffer packets to write to stream manager
-heron.instance.internal.bolt.write.queue.capacity: 128
-
-# The queue capacity (num of items) in spout for buffer packets to read from stream manager
-heron.instance.internal.spout.read.queue.capacity: 1024
-
-# The queue capacity (num of items) in spout for buffer packets to write to stream manager
-heron.instance.internal.spout.write.queue.capacity: 128
-
-# The queue capacity (num of items) for metrics packets to write to metrics manager
-heron.instance.internal.metrics.write.queue.capacity: 128
-
-# The size of packets read from stream manager will be determined by the minimal of
-# (a) time based
-# (b) size based
-
-# Time based, the maximum batch time in ms for instance to read from stream manager per attempt
-heron.instance.network.read.batch.time.ms: 16
-
-# Size based, the maximum batch size in bytes to read from stream manager
-heron.instance.network.read.batch.size.bytes: 32768
-
-# The size of packets written to stream manager will be determined by the minimum of
-# (a) time based
-# (b) size based
-
-# Time based, the maximum batch time in ms for instance to write to stream manager per attempt
-heron.instance.network.write.batch.time.ms: 16
-
-# Size based, the maximum batch size in bytes to write to stream manager
-heron.instance.network.write.batch.size.bytes: 32768
-
-# The maximum socket's send buffer size in bytes
-heron.instance.network.options.socket.send.buffer.size.bytes: 6553600
-
-# The maximum socket's received buffer size in bytes of instance's network options
-heron.instance.network.options.socket.received.buffer.size.bytes: 8738000
-
-# The maximum # of data tuple to batch in a HeronDataTupleSet protobuf
-heron.instance.set.data.tuple.capacity: 1024
-
-# The maximum size in bytes of data tuple to batch in a HeronDataTupleSet protobuf
-heron.instance.set.data.tuple.size.bytes: 8388608
-
-# The maximum # of control tuple to batch in a HeronControlTupleSet protobuf
-heron.instance.set.control.tuple.capacity: 1024
-
-# The maximum time in ms for a spout to do acknowledgement per attempt, the ack batch could
-# also break if there are no more ack tuples to process
-heron.instance.ack.batch.time.ms: 128
-
-# The maximum time in ms for an spout instance to emit tuples per attempt
-heron.instance.emit.batch.time.ms: 16
-
-# The maximum batch size in bytes for an spout to emit tuples per attempt
-heron.instance.emit.batch.size.bytes: 32768
-
-# The maximum time in ms for an bolt instance to execute tuples per attempt
-heron.instance.execute.batch.time.ms: 16
-
-# The maximum batch size in bytes for an bolt instance to execute tuples per attempt
-heron.instance.execute.batch.size.bytes: 32768
-
-# The time to wait before the instance exits forcibly when uncaught exception happens
-heron.instance.force.exit.timeout.ms: 2000
-
-# Interval in seconds to reconnect to the stream manager, including the request timeout in connecting
-heron.instance.reconnect.streammgr.interval.sec: 5
-heron.instance.reconnect.streammgr.times: 60
-
-# Interval in seconds to reconnect to the metrics manager, including the request timeout in connecting
-heron.instance.reconnect.metricsmgr.interval.sec: 5
-heron.instance.reconnect.metricsmgr.times: 60
-
-# The interval in second for an instance to sample its system metrics, for instance, CPU load.
-heron.instance.metrics.system.sample.interval.sec: 10
-
-# For efficient acknowledgement
-heron.instance.acknowledgement.nbuckets: 10
-
-################################################################################
-# For dynamically tuning the available sizes in the interval read & write queues
-# to provide high performance while avoiding GC issues
-################################################################################
-
-# The expected size on read queue in bolt
-heron.instance.tuning.expected.bolt.read.queue.size: 8
-
-# The expected size on write queue in bolt
-heron.instance.tuning.expected.bolt.write.queue.size: 8
-
-# The expected size on read queue in spout
-heron.instance.tuning.expected.spout.read.queue.size: 512
-
-# The exepected size on write queue in spout
-heron.instance.tuning.expected.spout.write.queue.size: 8
-
-# The expected size on metrics write queue
-heron.instance.tuning.expected.metrics.write.queue.size: 8
-
-heron.instance.tuning.current.sample.weight: 0.8
-
-# Interval in ms to tune the size of in & out data queue in instance
-heron.instance.tuning.interval.ms: 100
diff --git a/heron/config/src/yaml/conf/aurora/metrics_sinks.yaml b/heron/config/src/yaml/conf/aurora/metrics_sinks.yaml
deleted file mode 100644
index 6cac2f13c9e..00000000000
--- a/heron/config/src/yaml/conf/aurora/metrics_sinks.yaml
+++ /dev/null
@@ -1,257 +0,0 @@
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing,
-# software distributed under the License is distributed on an
-# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-# KIND, either express or implied. See the License for the
-# specific language governing permissions and limitations
-# under the License.
-
-########### These all have default values as shown
-
-# We would specify the unique sink-id first
-sinks:
- - file-sink
- - tmanager-sink
- - metricscache-sink
-
-########### Now we would specify the detailed configuration for every unique sink
-########### Syntax: sink-id: - option(s)
-
-########### option class is required as we need to instantiate a new instance by reflection
-########### option flush-frequency-ms is required to invoke flush() at interval
-########### option sink-restart-attempts, representsing # of times to restart a sink when it throws exceptions and dies.
-########### If this option is missed, default value 0 would be supplied; negative value represents to restart it forever.
-
-########### Other options would be constructed as an immutable map passed to IMetricsSink's init(Map conf) as argument,
-########### We would be able to fetch value by conf.get(options), for instance:
-########### We could get "org.apache.heron.metricsmgr.sink.FileSink" if conf.get("class") is called inside file-sink's instance
-
-### Config for file-sink
-file-sink:
- class: "org.apache.heron.metricsmgr.sink.FileSink"
- flush-frequency-ms: 60000 # 1 min
- sink-restart-attempts: -1 # Forever
- filename-output: "metrics.json" # File for metrics to write to
- file-maximum: 5 # maximum number of file saved in disk
-
-### Config for tmanager-sink
-tmanager-sink:
- class: "org.apache.heron.metricsmgr.sink.tmanager.TManagerSink"
- flush-frequency-ms: 60000
- sink-restart-attempts: -1 # Forever
- tmanager-location-check-interval-sec: 5
- tmanager-client:
- reconnect-interval-second: 5 # The re-connect interval to TManager from TManagerClient
- # The size of packets written to TManager will be determined by the minimal of: (a) time based (b) size based
- network-write-batch-size-bytes: 32768 # Size based, the maximum batch size in bytes to write to TManager
- network-write-batch-time-ms: 16 # Time based, the maximum batch time in ms for Metrics Manager to write to TManager per attempt
- network-read-batch-size-bytes: 32768 # Size based, the maximum batch size in bytes to write to TManager
- network-read-batch-time-ms: 16 # Time based, the maximum batch time in ms for Metrics Manager to write to TManager per attempt
- socket-send-buffer-size-bytes: 6553600 # The maximum socket's send buffer size in bytes
- socket-received-buffer-size-bytes: 8738000 # The maximum socket's received buffer size in bytes
- tmanager-metrics-type:
- "__emit-count": SUM
- "__execute-count": SUM
- "__fail-count": SUM
- "__ack-count": SUM
- "__complete-latency": AVG
- "__execute-latency": AVG
- "__process-latency": AVG
- "__jvm-uptime-secs": LAST
- "__jvm-process-cpu-load": LAST
- "__jvm-memory-used-mb": LAST
- "__jvm-memory-mb-total": LAST
- "__jvm-gc-collection-time-ms": LAST
- "__server/__time_spent_back_pressure_initiated": SUM
- "__time_spent_back_pressure_by_compid": SUM
-
-### Config for metricscache-sink
-metricscache-sink:
- class: "org.apache.heron.metricsmgr.sink.metricscache.MetricsCacheSink"
- flush-frequency-ms: 60000
- sink-restart-attempts: -1 # Forever
- metricscache-location-check-interval-sec: 5
- metricscache-client:
- reconnect-interval-second: 5 # The re-connect interval to TManager from TManagerClient
- # The size of packets written to TManager will be determined by the minimal of: (a) time based (b) size based
- network-write-batch-size-bytes: 32768 # Size based, the maximum batch size in bytes to write to TManager
- network-write-batch-time-ms: 16 # Time based, the maximum batch time in ms for Metrics Manager to write to TManager per attempt
- network-read-batch-size-bytes: 32768 # Size based, the maximum batch size in bytes to write to TManager
- network-read-batch-time-ms: 16 # Time based, the maximum batch time in ms for Metrics Manager to write to TManager per attempt
- socket-send-buffer-size-bytes: 6553600 # The maximum socket's send buffer size in bytes
- socket-received-buffer-size-bytes: 8738000 # The maximum socket's received buffer size in bytes
- metricscache-metrics-type:
- "__emit-count": SUM
- "__execute-count": SUM
- "__fail-count": SUM
- "__ack-count": SUM
- "__complete-latency": AVG
- "__execute-latency": AVG
- "__process-latency": AVG
- "__jvm-uptime-secs": LAST
- "__jvm-process-cpu-load": LAST
- "__jvm-memory-used-mb": LAST
- "__jvm-memory-mb-total": LAST
- "__jvm-gc-collection-time-ms": LAST
- "__server/__time_spent_back_pressure_initiated": SUM
- "__time_spent_back_pressure_by_compid": SUM
-
-### Config for prometheus-sink
-# prometheus-sink:
-# class: "org.apache.heron.metricsmgr.sink.PrometheusSink"
-# port: 8080 # The port on which to run (either port or port-file are mandatory)
-# path: /metrics # The path on which to publish the metrics (mandatory)
-# flat-metrics: true # By default the web-sink will publish a flat "name -> value" json map
-# include-topology-name: true # Include topology name in metric name (default false)
-# metrics-cache-max-size: 1000000 # Max number of metrics cached and published (default 1000000)
-# metrics-cache-ttl-sec: 600 # Time in seconds after which a metric that was collected will stopped being published (default 600)
-# rules:
-# # __jvm-peak-usage/G1-Survivor-Space-committed": "9",
-# - pattern: __jvm-(.+)/(.+)
-# name: jvm_$1_$2
-# attrNameSnakeCase: true
-# type: COUNTER
-# # "__execute-time-ns/pulsar-prod-4/default": "418764",
-# - pattern: __(?!jvm-+)(.+-count|.+-latency|.+-count|.+-time-ns)/(.+)/(.+)
-# name: $1
-# attrNameSnakeCase: true
-# type: COUNTER
-# labels:
-# dest_component: "$2"
-# context: "$3"
-# # "__execute-time-ns/pulsar-prod-4": "418764",
-# - pattern: __(?!jvm-+)(.+-count|.+-latency|.+-count|.+-time-ns)/(.+)
-# name: $1
-# attrNameSnakeCase: true
-# type: COUNTER
-# labels:
-# context: "$2"
-# # StreamManager
-# # "__client_stmgr-17/__bytes_to_stmgrs": "7841039",
-# - pattern: __(client_stmgr-.+)/__(.+_to_stmgrs)
-# name: $2
-# attrNameSnakeCase: true
-# type: COUNTER
-# labels:
-# dest_component: "$1"
-# # "__connection_buffer_by_instanceid/container_1_pulsar-prod-9_201/bytes": "0.000000",
-# - pattern: __(connection_buffer_by_instanceid)/container_(.+)_(.+)/(.+)
-# name: $1_$4
-# attrNameSnakeCase: true
-# type: COUNTER
-# labels:
-# dest_container: "$2"
-# dest_task: "$3"
-# # "__time_spent_back_pressure_by_compid/container_1_pulsar-prod-5_151": "0",
-# - pattern: __(time_spent_back_pressure_by_compid)/container_(.+)_(.+)
-# name: $1
-# attrNameSnakeCase: true
-# type: COUNTER
-# labels:
-# dest_container: "$2"
-# dest_task: "$3"
-# # PulsarSpoutMetrics of PulsarSpout 'PulsarSpoutMetrics/pulsar-prod-4-0/consumerThroughput'
-# - pattern: PulsarSpout/(.+)/(.+)
-# name: pulsar_spout_$2
-# attrNameSnakeCase: true
-# type: COUNTER
-# labels:
-# component: "$1"
-# - pattern: PulsarBolt/(.+)/(.+)
-# name: pulsar_bolt_$2
-# attrNameSnakeCase: true
-# type: COUNTER
-# labels:
-# component: "$1"
-# # name: "kafkaConsumer-request-size-max/consumer-node-metrics/client-id-spout/node-id-node-1"
-# - pattern: kafkaConsumer-(.+)/consumer-(node)-metrics/client-id-(.+)/node-id-(.+)
-# name: kafka_consumer_$2_$1
-# attrNameSnakeCase: true
-# type: COUNTER
-# labels:
-# client_id: "$3"
-# node_id: "$4"
-# # name: "kafkaConsumer-commit-rate/consumer-coordinator-metrics/client-id-spout"
-# - pattern: kafkaConsumer-(.+)/consumer-(coordinator)-metrics/client-id-(.+)
-# name: kafka_consumer_$2_$1
-# attrNameSnakeCase: true
-# type: COUNTER
-# labels:
-# client_id: "$3"
-# # name: "kafkaConsumer-records-lag-max/consumer-fetch-manager-metrics/client-id-spout/topic-nginx-lad-es/partition-1"
-# - pattern: kafkaConsumer-(.+)/consumer-(fetch-manager)-metrics/client-id-(.+)/topic-(.+)/partition-(.+)
-# name: kafka_consumer_$2_$1
-# attrNameSnakeCase: true
-# type: COUNTER
-# labels:
-# client_id: "$3"
-# topic: "$4"
-# partition: "$5"
-# # name: "kafkaConsumer-records-per-request-avg/consumer-fetch-manager-metrics/client-id-spout/topic-nginx-adp-cms-api"
-# - pattern: kafkaConsumer-(.+)/consumer-(fetch-manager)-metrics/client-id-(.+)/topic-(.+)
-# name: kafka_consumer_$2_$1
-# attrNameSnakeCase: true
-# type: COUNTER
-# labels:
-# client_id: "$3"
-# topic: "$4"
-# # name: "kafkaConsumer-bytes-consumed-total/consumer-fetch-manager-metrics/client-id-consumer-1"
-# - pattern: kafkaConsumer-(.+)/consumer-(feath-manager)-metrics/client-id-(.+)
-# name: kafka_consumer_$2_$1
-# attrNameSnakeCase: true
-# type: COUNTER
-# labels:
-# client_id: "$3"
-# - pattern: kafkaConsumer-(.+)/consumer-metrics/client-id-(.+)/node-id-(.+)
-# name: kafka_consumer_$1
-# attrNameSnakeCase: true
-# type: COUNTER
-# labels:
-# client_id: "$2"
-# node_id: "$3"
-# - pattern: kafkaConsumer-(.+)/consumer-metrics/client-id-(.+)
-# name: kafka_consumer_$1
-# attrNameSnakeCase: true
-# type: COUNTER
-# labels:
-# client_id: "$2"
-# - pattern: kafkaConsumer-(.+)/app-info/client-id-(.+)
-# name: kafka_consumer_$1
-# attrNameSnakeCase: true
-# type: COUNTER
-# labels:
-# client_id: "$2"
-# # kafkaOffset of KafkaSpout 'kafkaOffset/topicName/partition_2/spoutLag'
-# - pattern: kafkaOffset/(.+)/partition_([0-9]+)/(.+)
-# name: kafka_offset_$3
-# attrNameSnakeCase: true
-# type: COUNTER
-# labels:
-# topic: "$1"
-# partition: "$2"
-# # kafkaOffset of KafkaSpout 'kafkaOffset/topicName/totalSpoutLag'
-# - pattern: kafkaOffset/(.+)/(.+)
-# name: kafka_offset_$2
-# attrNameSnakeCase: true
-# type: COUNTER
-# labels:
-# topic: "$1"
-
-### Config for graphite-sink
-### Currently the graphite-sink is disabled
-# graphite-sink:
-# class: "org.apache.heron.metricsmgr.sink.GraphiteSink"
-# flush-frequency-ms: 60000
-# graphite_host: "127.0.0.1" # The host of graphite to be exported metrics to
-# graphite_port: 2004 # The port of graphite to be exported metrics to
-# metrics_prefix: "heron" # The prefix of every metrics
-# server_max_reconnect-attempts: 20 # The max reconnect attempts when failing to connect to graphite server
diff --git a/heron/config/src/yaml/conf/aurora/packing.yaml b/heron/config/src/yaml/conf/aurora/packing.yaml
deleted file mode 100644
index 4797a6a0218..00000000000
--- a/heron/config/src/yaml/conf/aurora/packing.yaml
+++ /dev/null
@@ -1,20 +0,0 @@
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing,
-# software distributed under the License is distributed on an
-# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-# KIND, either express or implied. See the License for the
-# specific language governing permissions and limitations
-# under the License.
-
-# packing algorithm for packing instances into containers
-heron.class.packing.algorithm: org.apache.heron.packing.roundrobin.RoundRobinPacking
-heron.class.repacking.algorithm: org.apache.heron.packing.roundrobin.RoundRobinPacking
diff --git a/heron/config/src/yaml/conf/aurora/scheduler.yaml b/heron/config/src/yaml/conf/aurora/scheduler.yaml
deleted file mode 100644
index a3a7a5a53de..00000000000
--- a/heron/config/src/yaml/conf/aurora/scheduler.yaml
+++ /dev/null
@@ -1,61 +0,0 @@
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing,
-# software distributed under the License is distributed on an
-# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-# KIND, either express or implied. See the License for the
-# specific language governing permissions and limitations
-# under the License.
-
-# scheduler class for distributing the topology for execution
-heron.class.scheduler: org.apache.heron.scheduler.aurora.AuroraScheduler
-
-# launcher class for submitting and launching the topology
-heron.class.launcher: org.apache.heron.scheduler.aurora.AuroraLauncher
-
-# location of java - pick it up from shell environment
-heron.directory.sandbox.java.home: /usr/lib/jvm/java-1.8.0-openjdk-amd64/
-
-# Invoke the IScheduler as a library directly
-heron.scheduler.is.service: False
-
-####################################################################
-# Following are Aurora-specific
-####################################################################
-# The maximum retry attempts when trying to kill an Aurora job
-heron.scheduler.job.max.kill.attempts: 5
-
-# The interval in ms between two retry-attempts to kill an Aurora job
-heron.scheduler.job.kill.retry.interval.ms: 2000
-
-# Aurora Controller Class
-# heron.class.scheduler.aurora.controller.cli: False
-
-####################################################################
-# Following are config for tunneling
-####################################################################
-# Whether we should attempt to tunnel if there is no direct access to a remote host (e.g. TManager)
-heron.scheduler.is.tunnel.needed: False
-
-# The connection timeout in ms when testing if we can connect to remote host
-heron.scheduler.tunnel.connection.timeout.ms: 1000
-
-# The number of retries when testing direct access to remote host
-heron.scheduler.tunnel.connection.retry.count: 2
-
-# The interval in ms between two retries when testing direct access to remote host
-heron.scheduler.tunnel.retry.interval.ms: 1000
-
-# The number of retries when verifying connectivity via tunnel
-heron.scheduler.tunnel.verify.count: 10
-
-# SSH tunnel host
-heron.scheduler.tunnel.host: "my.tunnel.host"
diff --git a/heron/config/src/yaml/conf/aurora/stateful.yaml b/heron/config/src/yaml/conf/aurora/stateful.yaml
deleted file mode 100644
index 76e1b0b542b..00000000000
--- a/heron/config/src/yaml/conf/aurora/stateful.yaml
+++ /dev/null
@@ -1,38 +0,0 @@
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing,
-# software distributed under the License is distributed on an
-# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-# KIND, either express or implied. See the License for the
-# specific language governing permissions and limitations
-# under the License.
-
-# indicate the type of storage for checkpoint is HDFS
-heron.statefulstorage.classname: "org.apache.heron.statefulstorage.hdfs.HDFSStorage"
-
-heron.statefulstorage.config:
- heron.statefulstorage.classpath: "$(hadoop --config /path/to/configs classpath)"
- heron.statefulstorage.hdfs.root.path: "/user/heron/checkpoints"
-
-# Following are configs for socket between ckptmgr and stateful storage
-heron.ckptmgr.network.write.batch.size.bytes: 32768
-
-heron.ckptmgr.network.write.batch.time.ms: 16
-
-heron.ckptmgr.network.read.batch.size.bytes: 32768
-
-heron.ckptmgr.network.read.batch.time.ms: 16
-
-heron.ckptmgr.network.options.socket.send.buffer.size.bytes: 655360
-
-heron.ckptmgr.network.options.socket.receive.buffer.size.bytes: 655360
-
-heron.ckptmgr.network.options.maximum.packetsize.bytes: 10485760
diff --git a/heron/config/src/yaml/conf/aurora/statemgr.yaml b/heron/config/src/yaml/conf/aurora/statemgr.yaml
deleted file mode 100644
index a2fe0feb761..00000000000
--- a/heron/config/src/yaml/conf/aurora/statemgr.yaml
+++ /dev/null
@@ -1,49 +0,0 @@
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing,
-# software distributed under the License is distributed on an
-# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-# KIND, either express or implied. See the License for the
-# specific language governing permissions and limitations
-# under the License.
-
-# local state manager class for managing state in a persistent fashion
-heron.class.state.manager: org.apache.heron.statemgr.localfs.LocalFileSystemStateManager
-
-# local state manager connection string
-heron.statemgr.connection.string: LOCALMODE
-
-# path of the root address to store the state in a local file system
-heron.statemgr.root.path: /vagrant/.herondata/repository/state/${CLUSTER}
-
-# create the sub directories, if needed
-heron.statemgr.localfs.is.initialize.file.tree: True
-
-####################################################################
-# Following are config for tunneling
-####################################################################
-# Whether we should attempt to tunnel if there is no direct access to a remote host (e.g. Zookeeper)
-heron.statemgr.is.tunnel.needed: False
-
-# The connection timeout in ms when testing if we can connect to remote host
-heron.statemgr.tunnel.connection.timeout.ms: 1000
-
-# The number of retries when testing direct access to remote host
-heron.statemgr.tunnel.connection.retry.count: 2
-
-# The interval in ms between two retries when testing direct access to remote host
-heron.statemgr.tunnel.retry.interval.ms: 1000
-
-# The number of retries when verifying connectivity via tunnel
-heron.statemgr.tunnel.verify.count: 10
-
-# SSH tunnel host
-heron.statemgr.tunnel.host: "my.tunnel.host"
diff --git a/heron/config/src/yaml/conf/aurora/uploader.yaml b/heron/config/src/yaml/conf/aurora/uploader.yaml
deleted file mode 100644
index 48a6ee48cd1..00000000000
--- a/heron/config/src/yaml/conf/aurora/uploader.yaml
+++ /dev/null
@@ -1,22 +0,0 @@
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing,
-# software distributed under the License is distributed on an
-# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-# KIND, either express or implied. See the License for the
-# specific language governing permissions and limitations
-# under the License.
-
-# uploader class for transferring the topology jar/tar files to storage
-heron.class.uploader: org.apache.heron.uploader.localfs.LocalFileSystemUploader
-
-# name of the directory to upload topologies for local file system uploader
-heron.uploader.localfs.file.system.directory: /vagrant/.herondata/repository/topologies/${CLUSTER}/${ROLE}/${TOPOLOGY}
\ No newline at end of file
diff --git a/heron/config/src/yaml/conf/examples/README.md b/heron/config/src/yaml/conf/examples/README.md
index 96c4e5d65d2..944ebb25ed3 100644
--- a/heron/config/src/yaml/conf/examples/README.md
+++ b/heron/config/src/yaml/conf/examples/README.md
@@ -21,9 +21,6 @@ for different components.
The files are
- * aurora_scheduler.yaml
- - this contains the configuration for aurora scheduler
-
* heron_internals.yaml
- this file contains several configuration elements that govern heron behavior
diff --git a/heron/config/src/yaml/conf/examples/aurora_scheduler.yaml b/heron/config/src/yaml/conf/examples/aurora_scheduler.yaml
deleted file mode 100644
index bef2cfca00a..00000000000
--- a/heron/config/src/yaml/conf/examples/aurora_scheduler.yaml
+++ /dev/null
@@ -1,29 +0,0 @@
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing,
-# software distributed under the License is distributed on an
-# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-# KIND, either express or implied. See the License for the
-# specific language governing permissions and limitations
-# under the License.
-
-# scheduler class for distributing the topology for execution
-heron.class.scheduler: org.apache.heron.scheduler.aurora.AuroraScheduler
-
-# launcher class for submitting and launching the topology
-heron.class.launcher: org.apache.heron.scheduler.aurora.AuroraLauncher
-
-# location of java - pick it up from shell environment
-heron.directory.sandbox.java.home: /usr/lib/jvm/java-1.8.0-openjdk-amd64/
-
-# Invoke the IScheduler as a library directly
-heron.scheduler.is.service: False
-
[incubator-heron] 01/07: [Sched] Removed Aurora from schedulers.
Posted by sa...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
saadurrahman pushed a commit to branch saadurrahman/3829-Deprecate-Apache-Aurora-dev
in repository https://gitbox.apache.org/repos/asf/incubator-heron.git
commit bfd151cf14034660ef7e5d767984d24f4f4a93a1
Author: Saad Ur Rahman <sa...@apache.org>
AuthorDate: Mon May 9 17:18:23 2022 -0400
[Sched] Removed Aurora from schedulers.
Apache Aurora source and tests removed from scheduler.
---
heron/schedulers/src/java/BUILD | 21 --
.../scheduler/aurora/AuroraCLIController.java | 214 ------------
.../heron/scheduler/aurora/AuroraContext.java | 53 ---
.../heron/scheduler/aurora/AuroraController.java | 44 ---
.../apache/heron/scheduler/aurora/AuroraField.java | 39 ---
.../aurora/AuroraHeronShellController.java | 132 --------
.../heron/scheduler/aurora/AuroraLauncher.java | 58 ----
.../heron/scheduler/aurora/AuroraScheduler.java | 299 -----------------
heron/schedulers/tests/java/BUILD | 19 --
.../scheduler/aurora/AuroraCLIControllerTest.java | 200 -----------
.../heron/scheduler/aurora/AuroraContextTest.java | 49 ---
.../heron/scheduler/aurora/AuroraLauncherTest.java | 80 -----
.../scheduler/aurora/AuroraSchedulerTest.java | 372 ---------------------
13 files changed, 1580 deletions(-)
diff --git a/heron/schedulers/src/java/BUILD b/heron/schedulers/src/java/BUILD
index b223c7bc440..caa66719e13 100644
--- a/heron/schedulers/src/java/BUILD
+++ b/heron/schedulers/src/java/BUILD
@@ -93,27 +93,6 @@ genrule(
cmd = "cp $< $@",
)
-java_library(
- name = "aurora-scheduler-java",
- srcs = glob(["**/aurora/*.java"]),
- resources = glob(["**/aurora/*.aurora"]),
- deps = scheduler_deps_files,
-)
-
-java_binary(
- name = "aurora-scheduler-unshaded",
- srcs = glob(["**/aurora/*.java"]),
- resources = glob(["**/aurora/*.aurora"]),
- deps = scheduler_deps_files,
-)
-
-genrule(
- name = "heron-aurora-scheduler",
- srcs = [":aurora-scheduler-unshaded_deploy.jar"],
- outs = ["heron-aurora-scheduler.jar"],
- cmd = "cp $< $@",
-)
-
java_library(
name = "null-scheduler-java",
srcs = glob(
diff --git a/heron/schedulers/src/java/org/apache/heron/scheduler/aurora/AuroraCLIController.java b/heron/schedulers/src/java/org/apache/heron/scheduler/aurora/AuroraCLIController.java
deleted file mode 100644
index 5cd710a7bc0..00000000000
--- a/heron/schedulers/src/java/org/apache/heron/scheduler/aurora/AuroraCLIController.java
+++ /dev/null
@@ -1,214 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-package org.apache.heron.scheduler.aurora;
-
-import java.util.ArrayList;
-import java.util.Arrays;
-import java.util.HashSet;
-import java.util.List;
-import java.util.Map;
-import java.util.Set;
-import java.util.logging.Logger;
-import java.util.stream.Collectors;
-
-import com.google.common.annotations.VisibleForTesting;
-
-import org.apache.heron.spi.packing.PackingPlan;
-import org.apache.heron.spi.utils.ShellUtils;
-
-/**
- * Implementation of AuroraController that shells out to the Aurora CLI to control the Aurora
- * scheduler workflow of a topology.
- */
-class AuroraCLIController implements AuroraController {
- private static final Logger LOG = Logger.getLogger(AuroraCLIController.class.getName());
-
- private final String jobSpec;
- private final boolean isVerbose;
- private String auroraFilename;
-
- AuroraCLIController(
- String jobName,
- String cluster,
- String role,
- String env,
- String auroraFilename,
- boolean isVerbose) {
- this.auroraFilename = auroraFilename;
- this.isVerbose = isVerbose;
- this.jobSpec = String.format("%s/%s/%s/%s", cluster, role, env, jobName);
- }
-
- @Override
- public boolean createJob(Map<AuroraField, String> bindings, Map<String, String> extra) {
- List<String> auroraCmd =
- new ArrayList<>(Arrays.asList("aurora", "job", "create", "--wait-until", "RUNNING"));
-
- for (AuroraField field : bindings.keySet()) {
- auroraCmd.add("--bind");
- auroraCmd.add(String.format("%s=%s", field, bindings.get(field)));
- }
-
- if (!extra.isEmpty()) {
- for (String field : extra.keySet()) {
- if (field.equals(AuroraContext.JOB_TEMPLATE)) {
- auroraFilename = auroraFilename.replace("heron.aurora", extra.get(field));
- } else {
- auroraCmd.add("--bind");
- auroraCmd.add(String.format("%s=%s", field, extra.get(field)));
- }
- }
- }
-
- auroraCmd.add(jobSpec);
- auroraCmd.add(auroraFilename);
-
- if (isVerbose) {
- auroraCmd.add("--verbose");
- }
-
- return runProcess(auroraCmd);
- }
-
- // Kill an aurora job
- @Override
- public boolean killJob() {
- List<String> auroraCmd = new ArrayList<>(Arrays.asList("aurora", "job", "killall"));
- auroraCmd.add(jobSpec);
-
- appendAuroraCommandOptions(auroraCmd, isVerbose);
-
- return runProcess(auroraCmd);
- }
-
- // Restart an aurora job
- @Override
- public boolean restart(Integer containerId) {
- List<String> auroraCmd = new ArrayList<>(Arrays.asList("aurora", "job", "restart"));
- if (containerId != null) {
- auroraCmd.add(String.format("%s/%d", jobSpec, containerId));
- } else {
- auroraCmd.add(jobSpec);
- }
-
- appendAuroraCommandOptions(auroraCmd, isVerbose);
-
- return runProcess(auroraCmd);
- }
-
- @Override
- public void removeContainers(Set<PackingPlan.ContainerPlan> containersToRemove) {
- String instancesToKill = getInstancesIdsToKill(containersToRemove);
- //aurora job kill <cluster>/<role>/<env>/<name>/<instance_ids>
- List<String> auroraCmd = new ArrayList<>(Arrays.asList(
- "aurora", "job", "kill", jobSpec + "/" + instancesToKill));
-
- appendAuroraCommandOptions(auroraCmd, isVerbose);
- LOG.info(String.format(
- "Killing %s aurora containers: %s", containersToRemove.size(), auroraCmd));
- if (!runProcess(auroraCmd)) {
- throw new RuntimeException("Failed to kill freed aurora instances: " + instancesToKill);
- }
- }
-
- private static final String ERR_PROMPT =
- "The topology can be in a strange stage. Please check carefully or redeploy the topology !!";
-
- @Override
- public Set<Integer> addContainers(Integer count) {
- //aurora job add <cluster>/<role>/<env>/<name>/<instance_id> <count>
- //clone instance 0
- List<String> auroraCmd = new ArrayList<>(Arrays.asList(
- "aurora", "job", "add", "--wait-until", "RUNNING",
- jobSpec + "/0", count.toString(), "--verbose"));
-
- LOG.info(String.format("Requesting %s new aurora containers %s", count, auroraCmd));
- StringBuilder stderr = new StringBuilder();
- if (!runProcess(auroraCmd, null, stderr)) {
- throw new RuntimeException(
- "Failed to create " + count + " new aurora instances. " + ERR_PROMPT);
- }
-
- if (stderr.length() <= 0) { // no container was added
- throw new RuntimeException("Empty output by Aurora. " + ERR_PROMPT);
- }
- return extractContainerIds(stderr.toString());
- }
-
- private Set<Integer> extractContainerIds(String auroraOutputStr) {
- String pattern = "Querying instance statuses: [";
- int idx1 = auroraOutputStr.indexOf(pattern);
- if (idx1 < 0) { // no container was added
- LOG.info("stdout & stderr by Aurora " + auroraOutputStr);
- return new HashSet<Integer>();
- }
- idx1 += pattern.length();
- int idx2 = auroraOutputStr.indexOf("]", idx1);
- String containerIdStr = auroraOutputStr.substring(idx1, idx2);
- LOG.info("container IDs returned by Aurora " + containerIdStr);
- return Arrays.asList(containerIdStr.split(", "))
- .stream().map(x->Integer.valueOf(x)).collect(Collectors.toSet());
- }
-
- // Utils method for unit tests
- @VisibleForTesting
- boolean runProcess(List<String> auroraCmd, StringBuilder stdout, StringBuilder stderr) {
- int status =
- ShellUtils.runProcess(auroraCmd.toArray(new String[auroraCmd.size()]),
- stderr != null ? stderr : new StringBuilder());
-
- if (status != 0) {
- LOG.severe(String.format(
- "Failed to run process. Command=%s, STDOUT=%s, STDERR=%s", auroraCmd, stdout, stderr));
- }
- return status == 0;
- }
-
- // Utils method for unit tests
- @VisibleForTesting
- boolean runProcess(List<String> auroraCmd) {
- return runProcess(auroraCmd, null, null);
- }
-
- private static String getInstancesIdsToKill(Set<PackingPlan.ContainerPlan> containersToRemove) {
- StringBuilder ids = new StringBuilder();
- for (PackingPlan.ContainerPlan containerPlan : containersToRemove) {
- if (ids.length() > 0) {
- ids.append(",");
- }
- ids.append(containerPlan.getId());
- }
- return ids.toString();
- }
-
- // Static method to append verbose and batching options if needed
- private static void appendAuroraCommandOptions(List<String> auroraCmd, boolean isVerbose) {
- // Append verbose if needed
- if (isVerbose) {
- auroraCmd.add("--verbose");
- }
-
- // Append batch size.
- // Note that we can not use "--no-batching" since "restart" command does not accept it.
- // So we play a small trick here by setting batch size Integer.MAX_VALUE.
- auroraCmd.add("--batch-size");
- auroraCmd.add(Integer.toString(Integer.MAX_VALUE));
- }
-}
diff --git a/heron/schedulers/src/java/org/apache/heron/scheduler/aurora/AuroraContext.java b/heron/schedulers/src/java/org/apache/heron/scheduler/aurora/AuroraContext.java
deleted file mode 100644
index d48cf8d7580..00000000000
--- a/heron/schedulers/src/java/org/apache/heron/scheduler/aurora/AuroraContext.java
+++ /dev/null
@@ -1,53 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-package org.apache.heron.scheduler.aurora;
-
-import java.io.File;
-
-import org.apache.heron.spi.common.Config;
-import org.apache.heron.spi.common.Context;
-
-public final class AuroraContext extends Context {
- public static final String JOB_LINK_TEMPLATE = "heron.scheduler.job.link.template";
- public static final String JOB_TEMPLATE = "heron.scheduler.job.template";
- public static final String JOB_MAX_KILL_ATTEMPTS = "heron.scheduler.job.max.kill.attempts";
- public static final String JOB_KILL_RETRY_INTERVAL_MS =
- "heron.scheduler.job.kill.retry.interval.ms";
-
- private AuroraContext() {
- }
-
- public static String getJobLinkTemplate(Config config) {
- return config.getStringValue(JOB_LINK_TEMPLATE);
- }
-
- public static String getHeronAuroraPath(Config config) {
- return config.getStringValue(JOB_TEMPLATE,
- new File(Context.heronConf(config), "heron.aurora").getPath());
- }
-
- public static int getJobMaxKillAttempts(Config config) {
- return config.getIntegerValue(JOB_MAX_KILL_ATTEMPTS, 5);
- }
-
- public static long getJobKillRetryIntervalMs(Config config) {
- return config.getLongValue(JOB_KILL_RETRY_INTERVAL_MS, 2000);
- }
-}
diff --git a/heron/schedulers/src/java/org/apache/heron/scheduler/aurora/AuroraController.java b/heron/schedulers/src/java/org/apache/heron/scheduler/aurora/AuroraController.java
deleted file mode 100644
index a70333908a0..00000000000
--- a/heron/schedulers/src/java/org/apache/heron/scheduler/aurora/AuroraController.java
+++ /dev/null
@@ -1,44 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-package org.apache.heron.scheduler.aurora;
-
-import java.util.Map;
-import java.util.Set;
-
-import org.apache.heron.spi.packing.PackingPlan;
-
-/**
- * Interface that defines how a client interacts with aurora to control the job lifecycle
- */
-public interface AuroraController {
-
- boolean createJob(Map<AuroraField, String> auroraProperties, Map<String, String> extra);
- boolean killJob();
-
- /**
- * Restarts a given container, or the entire job if containerId is null
- * @param containerId ID of container to restart, or entire job if null
- * @return the boolean return value
- */
- boolean restart(Integer containerId);
-
- void removeContainers(Set<PackingPlan.ContainerPlan> containersToRemove);
- Set<Integer> addContainers(Integer count);
-}
diff --git a/heron/schedulers/src/java/org/apache/heron/scheduler/aurora/AuroraField.java b/heron/schedulers/src/java/org/apache/heron/scheduler/aurora/AuroraField.java
deleted file mode 100644
index cc04ec10c98..00000000000
--- a/heron/schedulers/src/java/org/apache/heron/scheduler/aurora/AuroraField.java
+++ /dev/null
@@ -1,39 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-package org.apache.heron.scheduler.aurora;
-
-/**
- * Field names passed to aurora controllers during job creation
- */
-public enum AuroraField {
- CLUSTER,
- CORE_PACKAGE_URI,
- CPUS_PER_CONTAINER,
- DISK_PER_CONTAINER,
- ENVIRON,
- EXECUTOR_BINARY,
- NUM_CONTAINERS,
- RAM_PER_CONTAINER,
- ROLE,
- TIER,
- TOPOLOGY_ARGUMENTS,
- TOPOLOGY_NAME,
- TOPOLOGY_PACKAGE_URI
-}
diff --git a/heron/schedulers/src/java/org/apache/heron/scheduler/aurora/AuroraHeronShellController.java b/heron/schedulers/src/java/org/apache/heron/scheduler/aurora/AuroraHeronShellController.java
deleted file mode 100644
index aa6c708ca7a..00000000000
--- a/heron/schedulers/src/java/org/apache/heron/scheduler/aurora/AuroraHeronShellController.java
+++ /dev/null
@@ -1,132 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-package org.apache.heron.scheduler.aurora;
-
-import java.net.HttpURLConnection;
-import java.util.Map;
-import java.util.Set;
-import java.util.logging.Logger;
-
-import org.apache.heron.proto.system.PhysicalPlans.StMgr;
-import org.apache.heron.spi.common.Config;
-import org.apache.heron.spi.common.Context;
-import org.apache.heron.spi.packing.PackingPlan;
-import org.apache.heron.spi.statemgr.IStateManager;
-import org.apache.heron.spi.statemgr.SchedulerStateManagerAdaptor;
-import org.apache.heron.spi.utils.NetworkUtils;
-import org.apache.heron.spi.utils.ReflectionUtils;
-
-/**
- * Implementation of AuroraController that is a wrapper of AuroraCLIController.
- * The difference is `restart` command:
- * 1. restart whole topology: delegate to AuroraCLIController
- * 2. restart container 0: delegate to AuroraCLIController
- * 3. restart container x(x>0): call heron-shell endpoint `/killexecutor`
- * For backpressure, only containers with heron-stmgr may send out backpressure.
- * This class is to handle `restart backpressure containers inside container`,
- * while delegating to AuroraCLIController for all the other scenarios.
- */
-class AuroraHeronShellController implements AuroraController {
- private static final Logger LOG = Logger.getLogger(AuroraHeronShellController.class.getName());
-
- private final String topologyName;
- private final AuroraCLIController cliController;
- private final SchedulerStateManagerAdaptor stateMgrAdaptor;
-
- AuroraHeronShellController(String jobName, String cluster, String role, String env,
- String auroraFilename, boolean isVerbose, Config localConfig)
- throws ClassNotFoundException, InstantiationException, IllegalAccessException {
- this.topologyName = jobName;
- this.cliController =
- new AuroraCLIController(jobName, cluster, role, env, auroraFilename, isVerbose);
-
- Config config = Config.toClusterMode(localConfig);
- String stateMgrClass = Context.stateManagerClass(config);
- IStateManager stateMgr = ReflectionUtils.newInstance(stateMgrClass);
- stateMgr.initialize(config);
- stateMgrAdaptor = new SchedulerStateManagerAdaptor(stateMgr, 5000);
- }
-
- @Override
- public boolean createJob(Map<AuroraField, String> bindings, Map<String, String> extra) {
- return cliController.createJob(bindings, extra);
- }
-
- @Override
- public boolean killJob() {
- return cliController.killJob();
- }
-
- private StMgr searchContainer(Integer id) {
- String prefix = "stmgr-" + id;
- for (StMgr sm : stateMgrAdaptor.getPhysicalPlan(topologyName).getStmgrsList()) {
- if (sm.getId().equals(prefix)) {
- return sm;
- }
- }
- return null;
- }
-
- // Restart an aurora container
- @Override
- public boolean restart(Integer containerId) {
- // there is no backpressure for container 0, delegate to aurora client
- if (containerId == null || containerId == 0) {
- return cliController.restart(containerId);
- }
-
- if (stateMgrAdaptor == null) {
- LOG.warning("SchedulerStateManagerAdaptor not initialized");
- return false;
- }
-
- StMgr sm = searchContainer(containerId);
- if (sm == null) {
- LOG.warning("container not found in pplan " + containerId);
- return false;
- }
-
- String url = "http://" + sm.getHostName() + ":" + sm.getShellPort() + "/killexecutor";
- String payload = "secret=" + stateMgrAdaptor.getExecutionState(topologyName).getTopologyId();
- LOG.info("sending `kill container` to " + url + "; payload: " + payload);
-
- HttpURLConnection con = NetworkUtils.getHttpConnection(url);
- try {
- if (NetworkUtils.sendHttpPostRequest(con, "X", payload.getBytes())) {
- return NetworkUtils.checkHttpResponseCode(con, 200);
- } else { // if heron-shell command fails, delegate to aurora client
- LOG.info("heron-shell killexecutor failed; try aurora client ..");
- return cliController.restart(containerId);
- }
- } finally {
- con.disconnect();
- }
- }
-
- @Override
- public void removeContainers(Set<PackingPlan.ContainerPlan> containersToRemove) {
- cliController.removeContainers(containersToRemove);
- }
-
- @Override
- public Set<Integer> addContainers(Integer count) {
- return cliController.addContainers(count);
- }
-}
diff --git a/heron/schedulers/src/java/org/apache/heron/scheduler/aurora/AuroraLauncher.java b/heron/schedulers/src/java/org/apache/heron/scheduler/aurora/AuroraLauncher.java
deleted file mode 100644
index 023fcd98626..00000000000
--- a/heron/schedulers/src/java/org/apache/heron/scheduler/aurora/AuroraLauncher.java
+++ /dev/null
@@ -1,58 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-package org.apache.heron.scheduler.aurora;
-
-import org.apache.heron.scheduler.utils.LauncherUtils;
-import org.apache.heron.spi.common.Config;
-import org.apache.heron.spi.packing.PackingPlan;
-import org.apache.heron.spi.scheduler.ILauncher;
-import org.apache.heron.spi.scheduler.IScheduler;
-
-/**
- * Launch topology locally to Aurora.
- */
-
-public class AuroraLauncher implements ILauncher {
- private Config config;
- private Config runtime;
-
- @Override
- public void initialize(Config mConfig, Config mRuntime) {
- this.config = mConfig;
- this.runtime = mRuntime;
- }
-
- @Override
- public void close() {
-
- }
-
- @Override
- public boolean launch(PackingPlan packing) {
- LauncherUtils launcherUtils = LauncherUtils.getInstance();
- Config ytruntime = launcherUtils.createConfigWithPackingDetails(runtime, packing);
- return launcherUtils.onScheduleAsLibrary(config, ytruntime, getScheduler(), packing);
- }
-
- // Get AuroraScheduler
- protected IScheduler getScheduler() {
- return new AuroraScheduler();
- }
-}
diff --git a/heron/schedulers/src/java/org/apache/heron/scheduler/aurora/AuroraScheduler.java b/heron/schedulers/src/java/org/apache/heron/scheduler/aurora/AuroraScheduler.java
deleted file mode 100644
index 9c386f22a95..00000000000
--- a/heron/schedulers/src/java/org/apache/heron/scheduler/aurora/AuroraScheduler.java
+++ /dev/null
@@ -1,299 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-package org.apache.heron.scheduler.aurora;
-
-import java.util.ArrayList;
-import java.util.HashMap;
-import java.util.HashSet;
-import java.util.LinkedList;
-import java.util.List;
-import java.util.Map;
-import java.util.Scanner;
-import java.util.Set;
-import java.util.concurrent.ExecutionException;
-import java.util.logging.Level;
-import java.util.logging.Logger;
-
-import com.google.common.base.Optional;
-
-import org.apache.heron.api.generated.TopologyAPI;
-import org.apache.heron.api.utils.TopologyUtils;
-import org.apache.heron.api.utils.Utils;
-import org.apache.heron.proto.scheduler.Scheduler;
-import org.apache.heron.scheduler.UpdateTopologyManager;
-import org.apache.heron.scheduler.utils.Runtime;
-import org.apache.heron.scheduler.utils.SchedulerUtils;
-import org.apache.heron.spi.common.Config;
-import org.apache.heron.spi.common.Context;
-import org.apache.heron.spi.common.Key;
-import org.apache.heron.spi.common.TokenSub;
-import org.apache.heron.spi.packing.PackingPlan;
-import org.apache.heron.spi.packing.Resource;
-import org.apache.heron.spi.scheduler.IScalable;
-import org.apache.heron.spi.scheduler.IScheduler;
-
-public class AuroraScheduler implements IScheduler, IScalable {
- private static final Logger LOG = Logger.getLogger(AuroraLauncher.class.getName());
-
- private Config config;
- private Config runtime;
- private AuroraController controller;
- private UpdateTopologyManager updateTopologyManager;
-
- @Override
- public void initialize(Config mConfig, Config mRuntime) {
- this.config = Config.toClusterMode(mConfig);
- this.runtime = mRuntime;
- try {
- this.controller = getController();
- } catch (ClassNotFoundException | InstantiationException | IllegalAccessException e) {
- LOG.severe("AuroraController initialization failed " + e.getMessage());
- }
- this.updateTopologyManager =
- new UpdateTopologyManager(config, runtime, Optional.<IScalable>of(this));
- }
-
- /**
- * Get an AuroraController based on the config and runtime
- *
- * @return AuroraController
- */
- protected AuroraController getController()
- throws ClassNotFoundException, InstantiationException, IllegalAccessException {
- Boolean cliController = config.getBooleanValue(Key.AURORA_CONTROLLER_CLASS);
- Config localConfig = Config.toLocalMode(this.config);
- if (cliController) {
- return new AuroraCLIController(
- Runtime.topologyName(runtime),
- Context.cluster(localConfig),
- Context.role(localConfig),
- Context.environ(localConfig),
- AuroraContext.getHeronAuroraPath(localConfig),
- Context.verbose(localConfig));
- } else {
- return new AuroraHeronShellController(
- Runtime.topologyName(runtime),
- Context.cluster(localConfig),
- Context.role(localConfig),
- Context.environ(localConfig),
- AuroraContext.getHeronAuroraPath(localConfig),
- Context.verbose(localConfig),
- localConfig);
- }
- }
-
- @Override
- public void close() {
- if (updateTopologyManager != null) {
- updateTopologyManager.close();
- }
- }
-
- @Override
- public boolean onSchedule(PackingPlan packing) {
- if (packing == null || packing.getContainers().isEmpty()) {
- LOG.severe("No container requested. Can't schedule");
- return false;
- }
-
- LOG.info("Launching topology in aurora");
-
- // Align the cpu, RAM, disk to the maximal one, and set them to ScheduledResource
- PackingPlan updatedPackingPlan = packing.cloneWithHomogeneousScheduledResource();
- SchedulerUtils.persistUpdatedPackingPlan(Runtime.topologyName(runtime), updatedPackingPlan,
- Runtime.schedulerStateManagerAdaptor(runtime));
-
- // Use the ScheduledResource to create aurora properties
- // the ScheduledResource is guaranteed to be set after calling
- // cloneWithHomogeneousScheduledResource in the above code
- Resource containerResource =
- updatedPackingPlan.getContainers().iterator().next().getScheduledResource().get();
- Map<AuroraField, String> auroraProperties = createAuroraProperties(containerResource);
- Map<String, String> extraProperties = createExtraProperties(containerResource);
-
- return controller.createJob(auroraProperties, extraProperties);
- }
-
- @Override
- public List<String> getJobLinks() {
- List<String> jobLinks = new ArrayList<>();
-
- //Only the aurora job page is returned
- String jobLinkFormat = AuroraContext.getJobLinkTemplate(config);
- if (jobLinkFormat != null && !jobLinkFormat.isEmpty()) {
- String jobLink = TokenSub.substitute(config, jobLinkFormat);
- jobLinks.add(jobLink);
- }
-
- return jobLinks;
- }
-
- @Override
- public boolean onKill(Scheduler.KillTopologyRequest request) {
- // The aurora service can be unavailable or unstable for a while,
- // we will try to kill the job with multiple attempts
- int attempts = AuroraContext.getJobMaxKillAttempts(config);
- long retryIntervalMs = AuroraContext.getJobKillRetryIntervalMs(config);
- LOG.info("Will try " + attempts + " attempts at interval: " + retryIntervalMs + " ms");
-
- // First attempt
- boolean res = controller.killJob();
- attempts--;
-
- // Failure retry
- while (!res && attempts > 0) {
- LOG.warning("Failed to kill the topology. Will retry in " + retryIntervalMs + " ms...");
- Utils.sleep(retryIntervalMs);
-
- // Retry the killJob()
- res = controller.killJob();
- attempts--;
- }
-
- return res;
- }
-
- @Override
- public boolean onRestart(Scheduler.RestartTopologyRequest request) {
- Integer containerId = null;
- if (request.getContainerIndex() != -1) {
- containerId = request.getContainerIndex();
- }
- return controller.restart(containerId);
- }
-
- @Override
- public boolean onUpdate(Scheduler.UpdateTopologyRequest request) {
- try {
- updateTopologyManager.updateTopology(
- request.getCurrentPackingPlan(), request.getProposedPackingPlan());
- } catch (ExecutionException | InterruptedException e) {
- LOG.log(Level.SEVERE, "Could not update topology for request: " + request, e);
- return false;
- }
- return true;
- }
-
- private static final String CONFIRMED_YES = "y";
- boolean hasConfirmedWithUser(int newContainerCount) {
- LOG.info(String.format("After update there will be %d more containers. "
- + "Please make sure there are sufficient resources to update this job. "
- + "Continue update? [y/N]: ", newContainerCount));
- Scanner scanner = new Scanner(System.in);
- String userInput = scanner.nextLine();
- return CONFIRMED_YES.equalsIgnoreCase(userInput);
- }
-
- @Override
- public Set<PackingPlan.ContainerPlan> addContainers(
- Set<PackingPlan.ContainerPlan> containersToAdd) {
- Set<PackingPlan.ContainerPlan> remapping = new HashSet<>();
- if ("prompt".equalsIgnoreCase(Context.updatePrompt(config))
- && !hasConfirmedWithUser(containersToAdd.size())) {
- LOG.warning("Scheduler updated topology canceled.");
- return remapping;
- }
-
- // Do the actual containers adding
- LinkedList<Integer> newAddedContainerIds = new LinkedList<>(
- controller.addContainers(containersToAdd.size()));
- if (newAddedContainerIds.size() != containersToAdd.size()) {
- throw new RuntimeException(
- "Aurora returned different container count " + newAddedContainerIds.size()
- + "; input count was " + containersToAdd.size());
- }
- // Do the remapping:
- // use the `newAddedContainerIds` to replace the container id in the `containersToAdd`
- for (PackingPlan.ContainerPlan cp : containersToAdd) {
- PackingPlan.ContainerPlan newContainerPlan =
- new PackingPlan.ContainerPlan(
- newAddedContainerIds.pop(), cp.getInstances(),
- cp.getRequiredResource(), cp.getScheduledResource().orNull());
- remapping.add(newContainerPlan);
- }
- LOG.info("The remapping structure: " + remapping);
- return remapping;
- }
-
- @Override
- public void removeContainers(Set<PackingPlan.ContainerPlan> containersToRemove) {
- controller.removeContainers(containersToRemove);
- }
-
- protected Map<AuroraField, String> createAuroraProperties(Resource containerResource) {
- Map<AuroraField, String> auroraProperties = new HashMap<>();
-
- TopologyAPI.Topology topology = Runtime.topology(runtime);
-
- auroraProperties.put(AuroraField.EXECUTOR_BINARY,
- Context.executorBinary(config));
-
- List<String> topologyArgs = new ArrayList<>();
- SchedulerUtils.addExecutorTopologyArgs(topologyArgs, config, runtime);
- String args = String.join(" ", topologyArgs);
- auroraProperties.put(AuroraField.TOPOLOGY_ARGUMENTS, args);
-
- auroraProperties.put(AuroraField.CLUSTER, Context.cluster(config));
- auroraProperties.put(AuroraField.ENVIRON, Context.environ(config));
- auroraProperties.put(AuroraField.ROLE, Context.role(config));
- auroraProperties.put(AuroraField.TOPOLOGY_NAME, topology.getName());
-
- auroraProperties.put(AuroraField.CPUS_PER_CONTAINER,
- Double.toString(containerResource.getCpu()));
- auroraProperties.put(AuroraField.DISK_PER_CONTAINER,
- Long.toString(containerResource.getDisk().asBytes()));
- auroraProperties.put(AuroraField.RAM_PER_CONTAINER,
- Long.toString(containerResource.getRam().asBytes()));
-
- auroraProperties.put(AuroraField.NUM_CONTAINERS,
- Integer.toString(1 + TopologyUtils.getNumContainers(topology)));
-
- // Job configuration attribute 'production' is deprecated.
- // Use 'tier' attribute instead
- // See: http://aurora.apache.org/documentation/latest/reference/configuration/#job-objects
- if ("prod".equals(Context.environ(config))) {
- auroraProperties.put(AuroraField.TIER, "preferred");
- } else {
- auroraProperties.put(AuroraField.TIER, "preemptible");
- }
-
- String heronCoreReleasePkgURI = Context.corePackageUri(config);
- String topologyPkgURI = Runtime.topologyPackageUri(runtime).toString();
-
- auroraProperties.put(AuroraField.CORE_PACKAGE_URI, heronCoreReleasePkgURI);
- auroraProperties.put(AuroraField.TOPOLOGY_PACKAGE_URI, topologyPkgURI);
-
- return auroraProperties;
- }
-
- protected Map<String, String> createExtraProperties(Resource containerResource) {
- Map<String, String> extraProperties = new HashMap<>();
-
- if (config.containsKey(Key.SCHEDULER_PROPERTIES)) {
- String[] meta = config.getStringValue(Key.SCHEDULER_PROPERTIES).split(",");
- extraProperties.put(AuroraContext.JOB_TEMPLATE, meta[0]);
- for (int idx = 1; idx < meta.length; idx++) {
- extraProperties.put("AURORA_METADATA_" + idx, meta[idx]);
- }
- }
-
- return extraProperties;
- }
-}
diff --git a/heron/schedulers/tests/java/BUILD b/heron/schedulers/tests/java/BUILD
index 3ad9ffedf36..f3122052f0c 100644
--- a/heron/schedulers/tests/java/BUILD
+++ b/heron/schedulers/tests/java/BUILD
@@ -25,9 +25,6 @@ scheduler_deps_files = \
common_deps_files + \
spi_deps_files
-aurora_deps_files = [
- "//heron/schedulers/src/java:aurora-scheduler-java",
-]
yarn_deps_files = [
"//heron/packing/src/java:roundrobin-packing",
@@ -89,22 +86,6 @@ nomad_deps_files = \
"//heron/schedulers/src/java:scheduler-utils-java",
]
-java_library(
- name = "aurora-tests",
- srcs = glob(["**/aurora/*.java"]),
- deps = scheduler_deps_files + aurora_deps_files + ["@maven//:commons_cli_commons_cli"],
-)
-
-java_tests(
- size = "small",
- test_classes = [
- "org.apache.heron.scheduler.aurora.AuroraSchedulerTest",
- "org.apache.heron.scheduler.aurora.AuroraLauncherTest",
- "org.apache.heron.scheduler.aurora.AuroraCLIControllerTest",
- "org.apache.heron.scheduler.aurora.AuroraContextTest",
- ],
- runtime_deps = [":aurora-tests"],
-)
java_library(
name = "yarn-tests",
diff --git a/heron/schedulers/tests/java/org/apache/heron/scheduler/aurora/AuroraCLIControllerTest.java b/heron/schedulers/tests/java/org/apache/heron/scheduler/aurora/AuroraCLIControllerTest.java
deleted file mode 100644
index 4fad3998915..00000000000
--- a/heron/schedulers/tests/java/org/apache/heron/scheduler/aurora/AuroraCLIControllerTest.java
+++ /dev/null
@@ -1,200 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-package org.apache.heron.scheduler.aurora;
-
-import java.util.ArrayList;
-import java.util.Arrays;
-import java.util.Comparator;
-import java.util.HashMap;
-import java.util.List;
-import java.util.Map;
-import java.util.Set;
-import java.util.SortedSet;
-import java.util.TreeSet;
-
-import org.junit.After;
-import org.junit.AfterClass;
-import org.junit.Assert;
-import org.junit.Before;
-import org.junit.BeforeClass;
-import org.junit.Test;
-import org.mockito.Matchers;
-import org.mockito.Mockito;
-import org.mockito.invocation.InvocationOnMock;
-import org.mockito.stubbing.Answer;
-
-import org.apache.heron.spi.packing.PackingPlan;
-import org.apache.heron.spi.utils.PackingTestUtils;
-
-public class AuroraCLIControllerTest {
- private static final String JOB_NAME = "jobName";
- private static final String CLUSTER = "cluster";
- private static final String ROLE = "role";
- private static final String ENV = "gz";
- private static final String AURORA_FILENAME = "file.aurora";
- private static final String VERBOSE_CONFIG = "--verbose";
- private static final String BATCH_CONFIG = "--batch-size";
- private static final String JOB_SPEC = String.format("%s/%s/%s/%s", CLUSTER, ROLE, ENV, JOB_NAME);
- private static final boolean IS_VERBOSE = true;
-
- private AuroraCLIController controller;
-
- @BeforeClass
- public static void beforeClass() throws Exception {
-
- }
-
- @AfterClass
- public static void afterClass() throws Exception {
- }
-
- @Before
- public void setUp() throws Exception {
- controller = Mockito.spy(
- new AuroraCLIController(JOB_NAME, CLUSTER, ROLE, ENV, AURORA_FILENAME, IS_VERBOSE));
- }
-
- @After
- public void after() throws Exception {
- }
-
- @Test
- public void testCreateJob() throws Exception {
- Map<AuroraField, String> bindings = new HashMap<>();
- Map<String, String> bindings2 = new HashMap<>();
- List<String> expectedCommand = asList("aurora job create --wait-until RUNNING %s %s %s",
- JOB_SPEC, AURORA_FILENAME, VERBOSE_CONFIG);
-
- // Failed
- Mockito.doReturn(false).when(controller).runProcess(Matchers.anyListOf(String.class));
- Assert.assertFalse(controller.createJob(bindings, bindings2));
- Mockito.verify(controller).runProcess(Mockito.eq(expectedCommand));
-
- // Happy path
- Mockito.doReturn(true).when(controller).runProcess(Matchers.anyListOf(String.class));
- Assert.assertTrue(controller.createJob(bindings, bindings2));
- Mockito.verify(controller, Mockito.times(2)).runProcess(expectedCommand);
- }
-
- @Test
- public void testKillJob() throws Exception {
- List<String> expectedCommand = asList("aurora job killall %s %s %s %d",
- JOB_SPEC, VERBOSE_CONFIG, BATCH_CONFIG, Integer.MAX_VALUE);
-
- // Failed
- Mockito.doReturn(false).when(controller).runProcess(Matchers.anyListOf(String.class));
- Assert.assertFalse(controller.killJob());
- Mockito.verify(controller).runProcess(Mockito.eq(expectedCommand));
-
- // Happy path
- Mockito.doReturn(true).when(controller).runProcess(Matchers.anyListOf(String.class));
- Assert.assertTrue(controller.killJob());
- Mockito.verify(controller, Mockito.times(2)).runProcess(expectedCommand);
- }
-
- @Test
- public void testRestartJob() throws Exception {
- int containerId = 1;
- List<String> expectedCommand = asList("aurora job restart %s/%s %s %s %d",
- JOB_SPEC, containerId, VERBOSE_CONFIG, BATCH_CONFIG, Integer.MAX_VALUE);
-
- // Failed
- Mockito.doReturn(false).when(controller).runProcess(Matchers.anyListOf(String.class));
- Assert.assertFalse(controller.restart(containerId));
- Mockito.verify(controller).runProcess(Mockito.eq(expectedCommand));
-
- // Happy path
- Mockito.doReturn(true).when(controller).runProcess(Matchers.anyListOf(String.class));
- Assert.assertTrue(controller.restart(containerId));
- Mockito.verify(controller, Mockito.times(2)).runProcess(expectedCommand);
- }
-
- @Test
- public void testRemoveContainers() {
- class ContainerPlanComparator implements Comparator<PackingPlan.ContainerPlan> {
- @Override
- public int compare(PackingPlan.ContainerPlan o1, PackingPlan.ContainerPlan o2) {
- return ((Integer) o1.getId()).compareTo(o2.getId());
- }
- }
- SortedSet<PackingPlan.ContainerPlan> containers = new TreeSet<>(new ContainerPlanComparator());
- containers.add(PackingTestUtils.testContainerPlan(3));
- containers.add(PackingTestUtils.testContainerPlan(5));
-
- List<String> expectedCommand = asList("aurora job kill %s/3,5 %s %s %d",
- JOB_SPEC, VERBOSE_CONFIG, BATCH_CONFIG, Integer.MAX_VALUE);
-
- Mockito.doReturn(true).when(controller).runProcess(Matchers.anyListOf(String.class));
- controller.removeContainers(containers);
- Mockito.verify(controller).runProcess(Mockito.eq(expectedCommand));
- }
-
- @Test
- public void testAddContainers() {
- Integer containersToAdd = 3;
- List<String> expectedCommand = asList(
- "aurora job add --wait-until RUNNING %s/0 %s %s",
- JOB_SPEC, containersToAdd.toString(), VERBOSE_CONFIG);
-
- Mockito.doAnswer(new Answer<Boolean>() {
- @Override
- public Boolean answer(InvocationOnMock arg0) throws Throwable {
- final StringBuilder originalArgument = (StringBuilder) (arg0.getArguments())[2];
- originalArgument.append("Querying instance statuses: [1, 2, 3]");
- return true;
- }
- }).when(controller).runProcess(
- Matchers.anyListOf(String.class),
- Matchers.any(StringBuilder.class),
- Matchers.any(StringBuilder.class));
- Set<Integer> ret = controller.addContainers(containersToAdd);
- Assert.assertEquals(containersToAdd.intValue(), ret.size());
- Mockito.verify(controller)
- .runProcess(Matchers.eq(expectedCommand), Matchers.any(), Matchers.any());
- }
-
- @Test
- public void testAddContainersFailure() {
- Integer containersToAdd = 3;
- List<String> expectedCommand = asList(
- "aurora job add --wait-until RUNNING %s/0 %s %s",
- JOB_SPEC, containersToAdd.toString(), VERBOSE_CONFIG);
-
- Mockito.doAnswer(new Answer<Boolean>() {
- @Override
- public Boolean answer(InvocationOnMock arg0) throws Throwable {
- final StringBuilder originalArgument = (StringBuilder) (arg0.getArguments())[2];
- originalArgument.append("Querying instance statuses: x");
- return true;
- }
- }).when(controller).runProcess(
- Matchers.anyListOf(String.class),
- Matchers.any(StringBuilder.class),
- Matchers.any(StringBuilder.class));
- Set<Integer> ret = controller.addContainers(containersToAdd);
- Assert.assertEquals(0, ret.size());
- Mockito.verify(controller)
- .runProcess(Matchers.eq(expectedCommand), Matchers.any(), Matchers.any());
- }
-
- private static List<String> asList(String command, Object... values) {
- return new ArrayList<>(Arrays.asList(String.format(command, values).split(" ")));
- }
-}
diff --git a/heron/schedulers/tests/java/org/apache/heron/scheduler/aurora/AuroraContextTest.java b/heron/schedulers/tests/java/org/apache/heron/scheduler/aurora/AuroraContextTest.java
deleted file mode 100644
index 4f4017f41a7..00000000000
--- a/heron/schedulers/tests/java/org/apache/heron/scheduler/aurora/AuroraContextTest.java
+++ /dev/null
@@ -1,49 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-package org.apache.heron.scheduler.aurora;
-
-import org.junit.Assert;
-import org.junit.Test;
-
-import org.apache.heron.spi.common.Config;
-import org.apache.heron.spi.common.Key;
-
-public class AuroraContextTest {
-
- @Test
- public void testUsesConfigString() {
- final String auroraTemplate = "/dir/test.aurora";
- Config config = Config.newBuilder()
- .put(AuroraContext.JOB_TEMPLATE, auroraTemplate)
- .put(Key.HERON_CONF, "/test")
- .build();
- Assert.assertEquals("Expected to use value from JOB_TEMPLATE config",
- auroraTemplate, AuroraContext.getHeronAuroraPath(config));
- }
-
- @Test
- public void testFallback() {
- Config config = Config.newBuilder()
- .put(Key.HERON_CONF, "/test")
- .build();
- Assert.assertEquals("Expected to use heron_conf/heron.aurora", "/test/heron.aurora",
- AuroraContext.getHeronAuroraPath(config));
- }
-}
diff --git a/heron/schedulers/tests/java/org/apache/heron/scheduler/aurora/AuroraLauncherTest.java b/heron/schedulers/tests/java/org/apache/heron/scheduler/aurora/AuroraLauncherTest.java
deleted file mode 100644
index 76e94a5d9e9..00000000000
--- a/heron/schedulers/tests/java/org/apache/heron/scheduler/aurora/AuroraLauncherTest.java
+++ /dev/null
@@ -1,80 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-package org.apache.heron.scheduler.aurora;
-
-import org.junit.Assert;
-import org.junit.Test;
-import org.junit.runner.RunWith;
-import org.mockito.Mockito;
-import org.powermock.api.mockito.PowerMockito;
-import org.powermock.core.classloader.annotations.PowerMockIgnore;
-import org.powermock.core.classloader.annotations.PrepareForTest;
-import org.powermock.modules.junit4.PowerMockRunner;
-
-import org.apache.heron.scheduler.utils.LauncherUtils;
-import org.apache.heron.spi.common.Config;
-import org.apache.heron.spi.packing.PackingPlan;
-import org.apache.heron.spi.scheduler.IScheduler;
-
-@RunWith(PowerMockRunner.class)
-@PowerMockIgnore("jdk.internal.reflect.*")
-@PrepareForTest(LauncherUtils.class)
-public class AuroraLauncherTest {
- @Test
- public void testLaunch() throws Exception {
- Config config = Config.newBuilder().build();
- AuroraLauncher launcher = Mockito.spy(AuroraLauncher.class);
- launcher.initialize(config, config);
-
- LauncherUtils mockLauncherUtils = Mockito.mock(LauncherUtils.class);
- PowerMockito.spy(LauncherUtils.class);
- PowerMockito.doReturn(mockLauncherUtils).when(LauncherUtils.class, "getInstance");
-
- // Failed to schedule
- Mockito.when(mockLauncherUtils.onScheduleAsLibrary(
- Mockito.any(Config.class),
- Mockito.any(Config.class),
- Mockito.any(IScheduler.class),
- Mockito.any(PackingPlan.class))).thenReturn(false);
-
- Assert.assertFalse(launcher.launch(Mockito.mock(PackingPlan.class)));
- Mockito.verify(mockLauncherUtils).onScheduleAsLibrary(
- Mockito.any(Config.class),
- Mockito.any(Config.class),
- Mockito.any(IScheduler.class),
- Mockito.any(PackingPlan.class));
-
- // Happy path
- Mockito.when(mockLauncherUtils.onScheduleAsLibrary(
- Mockito.any(Config.class),
- Mockito.any(Config.class),
- Mockito.any(IScheduler.class),
- Mockito.any(PackingPlan.class))).thenReturn(true);
-
- Assert.assertTrue(launcher.launch(Mockito.mock(PackingPlan.class)));
- Mockito.verify(mockLauncherUtils, Mockito.times(2)).onScheduleAsLibrary(
- Mockito.any(Config.class),
- Mockito.any(Config.class),
- Mockito.any(IScheduler.class),
- Mockito.any(PackingPlan.class));
-
- launcher.close();
- }
-}
diff --git a/heron/schedulers/tests/java/org/apache/heron/scheduler/aurora/AuroraSchedulerTest.java b/heron/schedulers/tests/java/org/apache/heron/scheduler/aurora/AuroraSchedulerTest.java
deleted file mode 100644
index a73a7e0676d..00000000000
--- a/heron/schedulers/tests/java/org/apache/heron/scheduler/aurora/AuroraSchedulerTest.java
+++ /dev/null
@@ -1,372 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-package org.apache.heron.scheduler.aurora;
-
-import java.net.URI;
-import java.net.URISyntaxException;
-import java.util.HashMap;
-import java.util.HashSet;
-import java.util.List;
-import java.util.Map;
-import java.util.Set;
-
-import org.junit.After;
-import org.junit.AfterClass;
-import org.junit.Assert;
-import org.junit.Before;
-import org.junit.BeforeClass;
-import org.junit.Test;
-import org.junit.runner.RunWith;
-import org.mockito.Matchers;
-import org.mockito.Mockito;
-import org.powermock.api.mockito.PowerMockito;
-import org.powermock.core.classloader.annotations.PowerMockIgnore;
-import org.powermock.core.classloader.annotations.PrepareForTest;
-import org.powermock.modules.junit4.PowerMockRunner;
-
-import org.apache.commons.cli.CommandLine;
-import org.apache.heron.api.generated.TopologyAPI;
-import org.apache.heron.common.basics.ByteAmount;
-import org.apache.heron.common.utils.topology.TopologyTests;
-import org.apache.heron.proto.scheduler.Scheduler;
-import org.apache.heron.proto.system.PackingPlans;
-import org.apache.heron.scheduler.SubmitterMain;
-import org.apache.heron.spi.common.Config;
-import org.apache.heron.spi.common.Key;
-import org.apache.heron.spi.common.TokenSub;
-import org.apache.heron.spi.packing.PackingPlan;
-import org.apache.heron.spi.packing.Resource;
-import org.apache.heron.spi.statemgr.SchedulerStateManagerAdaptor;
-import org.apache.heron.spi.utils.PackingTestUtils;
-
-import static org.junit.Assert.assertEquals;
-import static org.junit.Assert.assertTrue;
-import static org.junit.Assert.fail;
-import static org.mockito.Matchers.anyString;
-import static org.mockito.Mockito.any;
-import static org.mockito.Mockito.doReturn;
-import static org.mockito.Mockito.eq;
-import static org.mockito.Mockito.mock;
-import static org.mockito.Mockito.when;
-
-@RunWith(PowerMockRunner.class)
-@PowerMockIgnore("jdk.internal.reflect.*")
-@PrepareForTest({TokenSub.class, Config.class})
-public class AuroraSchedulerTest {
- private static final String AURORA_PATH = "path.aurora";
- private static final String PACKING_PLAN_ID = "packing.plan.id";
- private static final String TOPOLOGY_NAME = "topologyName";
- private static final int CONTAINER_ID = 7;
-
- private static AuroraScheduler scheduler;
-
- @Before
- public void setUp() throws Exception {
- }
-
- @After
- public void after() throws Exception {
- }
-
- @BeforeClass
- public static void beforeClass() throws Exception {
- scheduler = Mockito.spy(AuroraScheduler.class);
- doReturn(new HashMap<String, String>())
- .when(scheduler).createAuroraProperties(Mockito.any(Resource.class));
- }
-
- @AfterClass
- public static void afterClass() throws Exception {
- scheduler.close();
- }
-
- /**
- * Tests that we can schedule
- */
- @Test
- public void testOnSchedule() throws Exception {
- AuroraController controller = Mockito.mock(AuroraController.class);
- doReturn(controller).when(scheduler).getController();
-
- SchedulerStateManagerAdaptor stateManager = mock(SchedulerStateManagerAdaptor.class);
- Config runtime = Mockito.mock(Config.class);
- when(runtime.get(Key.SCHEDULER_STATE_MANAGER_ADAPTOR)).thenReturn(stateManager);
- when(runtime.getStringValue(Key.TOPOLOGY_NAME)).thenReturn(TOPOLOGY_NAME);
-
- Config mConfig = Mockito.mock(Config.class);
- PowerMockito.mockStatic(Config.class);
- when(Config.toClusterMode(mConfig)).thenReturn(mConfig);
-
- when(mConfig.getStringValue(eq(AuroraContext.JOB_TEMPLATE),
- anyString())).thenReturn(AURORA_PATH);
-
- scheduler.initialize(mConfig, runtime);
-
- // Fail to schedule due to null PackingPlan
- Assert.assertFalse(scheduler.onSchedule(null));
-
- PackingPlan plan = new PackingPlan(PACKING_PLAN_ID, new HashSet<PackingPlan.ContainerPlan>());
- assertTrue(plan.getContainers().isEmpty());
-
- // Fail to schedule due to PackingPlan is empty
- Assert.assertFalse(scheduler.onSchedule(plan));
-
- // Construct valid PackingPlan
- Set<PackingPlan.ContainerPlan> containers = new HashSet<>();
- containers.add(PackingTestUtils.testContainerPlan(CONTAINER_ID));
- PackingPlan validPlan = new PackingPlan(PACKING_PLAN_ID, containers);
-
- // Failed to create job via controller
- doReturn(false).when(controller)
- .createJob(Matchers.anyMapOf(AuroraField.class, String.class),
- Matchers.anyMapOf(String.class, String.class));
- doReturn(true).when(stateManager)
- .updatePackingPlan(any(PackingPlans.PackingPlan.class), eq(TOPOLOGY_NAME));
-
- Assert.assertFalse(scheduler.onSchedule(validPlan));
-
- Mockito.verify(controller)
- .createJob(Matchers.anyMapOf(AuroraField.class, String.class),
- Matchers.anyMapOf(String.class, String.class));
- Mockito.verify(stateManager)
- .updatePackingPlan(any(PackingPlans.PackingPlan.class), eq(TOPOLOGY_NAME));
-
- // Happy path
- doReturn(true).when(controller)
- .createJob(Matchers.anyMapOf(AuroraField.class, String.class),
- Matchers.anyMapOf(String.class, String.class));
- assertTrue(scheduler.onSchedule(validPlan));
-
- Mockito.verify(controller, Mockito.times(2))
- .createJob(Matchers.anyMapOf(AuroraField.class, String.class),
- Matchers.anyMapOf(String.class, String.class));
- Mockito.verify(stateManager, Mockito.times(2))
- .updatePackingPlan(any(PackingPlans.PackingPlan.class), eq(TOPOLOGY_NAME));
- }
-
- @Test
- public void testOnKill() throws Exception {
- Config mockConfig = Mockito.mock(Config.class);
- PowerMockito.mockStatic(Config.class);
- when(Config.toClusterMode(mockConfig)).thenReturn(mockConfig);
-
- AuroraController controller = Mockito.mock(AuroraController.class);
- doReturn(controller).when(scheduler).getController();
- scheduler.initialize(mockConfig, Mockito.mock(Config.class));
-
- // Failed to kill job via controller
- doReturn(false).when(controller).killJob();
- Assert.assertFalse(scheduler.onKill(Scheduler.KillTopologyRequest.getDefaultInstance()));
- Mockito.verify(controller).killJob();
-
- // Happy path
- doReturn(true).when(controller).killJob();
- assertTrue(scheduler.onKill(Scheduler.KillTopologyRequest.getDefaultInstance()));
- Mockito.verify(controller, Mockito.times(2)).killJob();
- }
-
- @Test
- public void testOnRestart() throws Exception {
- Config mockConfig = Mockito.mock(Config.class);
- PowerMockito.mockStatic(Config.class);
- when(Config.toClusterMode(mockConfig)).thenReturn(mockConfig);
-
- AuroraController controller = Mockito.mock(AuroraController.class);
- doReturn(controller).when(scheduler).getController();
- scheduler.initialize(mockConfig, Mockito.mock(Config.class));
-
- // Construct the RestartTopologyRequest
- int containerToRestart = 1;
- Scheduler.RestartTopologyRequest restartTopologyRequest =
- Scheduler.RestartTopologyRequest.newBuilder().
- setTopologyName(TOPOLOGY_NAME).setContainerIndex(containerToRestart).
- build();
-
- // Failed to kill job via controller
- doReturn(false).when(
- controller).restart(containerToRestart);
- Assert.assertFalse(scheduler.onRestart(restartTopologyRequest));
- Mockito.verify(controller).restart(containerToRestart);
-
- // Happy path
- doReturn(true).when(
- controller).restart(containerToRestart);
- assertTrue(scheduler.onRestart(restartTopologyRequest));
- Mockito.verify(controller, Mockito.times(2)).restart(containerToRestart);
- }
-
- @Test
- public void testGetJobLinks() throws Exception {
- final String JOB_LINK_FORMAT = "http://go/${CLUSTER}/${ROLE}/${ENVIRON}/${TOPOLOGY}";
- final String SUBSTITUTED_JOB_LINK = "http://go/local/heron/test/test_topology";
-
- Config mockConfig = Mockito.mock(Config.class);
- when(mockConfig.getStringValue(AuroraContext.JOB_LINK_TEMPLATE))
- .thenReturn(JOB_LINK_FORMAT);
-
- PowerMockito.mockStatic(Config.class);
- when(Config.toClusterMode(mockConfig)).thenReturn(mockConfig);
-
- AuroraController controller = Mockito.mock(AuroraController.class);
- doReturn(controller).when(scheduler).getController();
- scheduler.initialize(mockConfig, Mockito.mock(Config.class));
-
- PowerMockito.spy(TokenSub.class);
- PowerMockito.doReturn(SUBSTITUTED_JOB_LINK)
- .when(TokenSub.class, "substitute", mockConfig, JOB_LINK_FORMAT);
-
- List<String> result = scheduler.getJobLinks();
-
- assertEquals(1, result.size());
- assertTrue(result.get(0).equals(SUBSTITUTED_JOB_LINK));
- }
-
-
- @Test
- public void testProperties() throws URISyntaxException {
- TopologyAPI.Topology topology = TopologyTests.createTopology(
- TOPOLOGY_NAME, new org.apache.heron.api.Config(),
- "spoutName", "boltName", 1, 1);
-
- Config runtime = mock(Config.class);
- when(runtime.get(Key.TOPOLOGY_DEFINITION)).thenReturn(topology);
- when(runtime.get(Key.TOPOLOGY_PACKAGE_URI)).thenReturn(new URI("http://foo/bar"));
-
- // This must mimic how SubmitterMain loads configs
- CommandLine commandLine = mock(CommandLine.class);
- when(commandLine.getOptionValue("cluster")).thenReturn("some_cluster");
- when(commandLine.getOptionValue("role")).thenReturn("some_role");
- when(commandLine.getOptionValue("environment")).thenReturn("some_env");
- when(commandLine.getOptionValue("heron_home")).thenReturn("/some/heron/home");
- when(commandLine.getOptionValue("config_path")).thenReturn("/some/config/path");
- when(commandLine.getOptionValue("topology_package")).thenReturn("jar");
- when(commandLine.getOptionValue("topology_defn")).thenReturn("/mock/defnFile.defn");
- when(commandLine.getOptionValue("topology_bin")).thenReturn("binaryFile.jar");
- Config config = Mockito.spy(SubmitterMain.loadConfig(commandLine, topology));
-
- AuroraScheduler testScheduler = new AuroraScheduler();
- testScheduler.initialize(config, runtime);
- Resource containerResource =
- new Resource(2.3, ByteAmount.fromGigabytes(2), ByteAmount.fromGigabytes(3));
- Map<AuroraField, String> properties = testScheduler.createAuroraProperties(containerResource);
-
- // this part is key, the conf path in the config is absolute to the install dir, but what
- // aurora properties get below is the relative ./heron-conf path to be used when run remotely
- assertEquals("Invalid value for key " + Key.HERON_CONF,
- "/some/config/path", config.getStringValue(Key.HERON_CONF));
-
- String expectedConf = "./heron-conf";
- String expectedBin = "./heron-core/bin";
- String expectedLib = "./heron-core/lib";
- String expectedDist = "./heron-core/dist";
- for (AuroraField field : AuroraField.values()) {
- boolean asserted = false;
- Object expected = null;
- Object found = properties.get(field);
- switch (field) {
- case CORE_PACKAGE_URI:
- expected = expectedDist + "/heron-core.tar.gz";
- break;
- case CPUS_PER_CONTAINER:
- expected = Double.valueOf(containerResource.getCpu()).toString();
- break;
- case DISK_PER_CONTAINER:
- expected = Long.valueOf(containerResource.getDisk().asBytes()).toString();
- break;
- case RAM_PER_CONTAINER:
- expected = Long.valueOf(containerResource.getRam().asBytes()).toString();
- break;
- case TIER:
- expected = "preemptible";
- break;
- case NUM_CONTAINERS:
- expected = "2";
- break;
- case EXECUTOR_BINARY:
- expected = expectedBin + "/heron-executor";
- break;
- case TOPOLOGY_PACKAGE_URI:
- expected = "http://foo/bar";
- break;
- case TOPOLOGY_ARGUMENTS:
- expected = "--topology-name=topologyName"
- + " --topology-id=" + topology.getId()
- + " --topology-defn-file=defnFile.defn"
- + " --state-manager-connection=null"
- + " --state-manager-root=null"
- + " --state-manager-config-file=" + expectedConf + "/statemgr.yaml"
- + " --tmanager-binary=" + expectedBin + "/heron-tmanager"
- + " --stmgr-binary=" + expectedBin + "/heron-stmgr"
- + " --metrics-manager-classpath=" + expectedLib + "/metricsmgr/*"
- + " --instance-jvm-opts=\"\""
- + " --classpath=binaryFile.jar"
- + " --heron-internals-config-file=" + expectedConf + "/heron_internals.yaml"
- + " --override-config-file=" + expectedConf + "/override.yaml"
- + " --component-ram-map=null"
- + " --component-jvm-opts=\"\""
- + " --pkg-type=jar"
- + " --topology-binary-file=binaryFile.jar"
- + " --heron-java-home=/usr/lib/jvm/default-java"
- + " --heron-shell-binary=" + expectedBin + "/heron-shell"
- + " --cluster=some_cluster"
- + " --role=some_role"
- + " --environment=some_env"
- + " --instance-classpath=" + expectedLib + "/instance/*"
- + " --metrics-sinks-config-file=" + expectedConf + "/metrics_sinks.yaml"
- + " --scheduler-classpath=" + expectedLib + "/scheduler/*:./heron-core"
- + "/lib/packing/*:" + expectedLib + "/statemgr/*"
- + " --python-instance-binary=" + expectedBin + "/heron-python-instance"
- + " --cpp-instance-binary=" + expectedBin + "/heron-cpp-instance"
- + " --metricscache-manager-classpath=" + expectedLib + "/metricscachemgr/*"
- + " --metricscache-manager-mode=disabled"
- + " --is-stateful=false"
- + " --checkpoint-manager-classpath=" + expectedLib + "/ckptmgr/*:"
- + expectedLib + "/statefulstorage/*:"
- + " --stateful-config-file=" + expectedConf + "/stateful.yaml"
- + " --checkpoint-manager-ram=1073741824"
- + " --health-manager-mode=disabled"
- + " --health-manager-classpath=" + expectedLib + "/healthmgr/*";
- break;
- case CLUSTER:
- expected = "some_cluster";
- break;
- case ENVIRON:
- expected = "some_env";
- break;
- case ROLE:
- expected = "some_role";
- break;
- case TOPOLOGY_NAME:
- expected = "topologyName";
- break;
- default:
- fail(String.format(
- "Expected value for Aurora field %s not found in test (found=%s)", field, found));
- }
- if (!asserted) {
- assertEquals("Incorrect value found for field " + field, expected, found);
- }
- properties.remove(field);
- }
-
- assertTrue("The following aurora fields were not set by the scheduler: " + properties,
- properties.isEmpty());
- }
-}
[incubator-heron] 03/07: [SPI] Removed key for Aurora scheduler class.
Posted by sa...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
saadurrahman pushed a commit to branch saadurrahman/3829-Deprecate-Apache-Aurora-dev
in repository https://gitbox.apache.org/repos/asf/incubator-heron.git
commit 0a56102b59aa1e407c47d52e3a6f2fc41defb233
Author: Saad Ur Rahman <sa...@apache.org>
AuthorDate: Mon May 9 17:19:33 2022 -0400
[SPI] Removed key for Aurora scheduler class.
---
heron/spi/src/java/org/apache/heron/spi/common/Key.java | 1 -
1 file changed, 1 deletion(-)
diff --git a/heron/spi/src/java/org/apache/heron/spi/common/Key.java b/heron/spi/src/java/org/apache/heron/spi/common/Key.java
index 94cd60483a6..9a06cb916dd 100644
--- a/heron/spi/src/java/org/apache/heron/spi/common/Key.java
+++ b/heron/spi/src/java/org/apache/heron/spi/common/Key.java
@@ -81,7 +81,6 @@ public enum Key {
PACKING_CLASS ("heron.class.packing.algorithm", Type.STRING),
REPACKING_CLASS ("heron.class.repacking.algorithm", Type.STRING),
STATE_MANAGER_CLASS ("heron.class.state.manager", Type.STRING),
- AURORA_CONTROLLER_CLASS ("heron.class.scheduler.aurora.controller.cli", Boolean.TRUE),
//keys for scheduler config
SCHEDULER_IS_SERVICE ("heron.scheduler.is.service", Boolean.TRUE),
[incubator-heron] 02/07: [Docs] Removed Aurora.
Posted by sa...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
saadurrahman pushed a commit to branch saadurrahman/3829-Deprecate-Apache-Aurora-dev
in repository https://gitbox.apache.org/repos/asf/incubator-heron.git
commit dcb68dafd357bbbbcd05d6193accc0f8d7efaa24
Author: Saad Ur Rahman <sa...@apache.org>
AuthorDate: Mon May 9 17:18:43 2022 -0400
[Docs] Removed Aurora.
---
website2/docs/schedulers-aurora-cluster.md | 138 ------------
website2/docs/schedulers-aurora-local.md | 329 -----------------------------
2 files changed, 467 deletions(-)
diff --git a/website2/docs/schedulers-aurora-cluster.md b/website2/docs/schedulers-aurora-cluster.md
deleted file mode 100644
index 43d79d61dec..00000000000
--- a/website2/docs/schedulers-aurora-cluster.md
+++ /dev/null
@@ -1,138 +0,0 @@
----
-id: schedulers-aurora-cluster
-title: Aurora Cluster
-sidebar_label: Aurora Cluster
----
-<!--
- Licensed to the Apache Software Foundation (ASF) under one
- or more contributor license agreements. See the NOTICE file
- distributed with this work for additional information
- regarding copyright ownership. The ASF licenses this file
- to you under the Apache License, Version 2.0 (the
- "License"); you may not use this file except in compliance
- with the License. You may obtain a copy of the License at
- http://www.apache.org/licenses/LICENSE-2.0
- Unless required by applicable law or agreed to in writing,
- software distributed under the License is distributed on an
- "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- KIND, either express or implied. See the License for the
- specific language governing permissions and limitations
- under the License.
--->
-
-Heron supports deployment on [Apache Aurora](http://aurora.apache.org/) out of
-the box. A step by step guide on how to setup Heron with Apache Aurora locally
-can be found in [Setting up Heron with Aurora Cluster Locally on Linux](schedulers-aurora-local). You can also run Heron on
-a [local scheduler](schedulers-local).
-
-## How Heron on Aurora Works
-
-Aurora doesn't have a Heron scheduler *per se*. Instead, when a topology is
-submitted to Heron, `heron` cli interacts with Aurora to automatically deploy
-all the [components](heron-architecture) necessary to [manage
-topologies](user-manuals-heron-cli).
-
-## ZooKeeper
-
-To run Heron on Aurora, you'll need to set up a ZooKeeper cluster and configure
-Heron to communicate with it. Instructions can be found in [Setting up
-ZooKeeper](state-managers-zookeeper).
-
-## Hosting Binaries
-
-To deploy Heron, the Aurora cluster needs access to the
-Heron core binary, which can be hosted wherever you'd like, so long as
-it's accessible to Aurora (for example in [Amazon
-S3](https://aws.amazon.com/s3/) or using a local blob storage solution). You
-can download the core binary from github or build it using the instructions
-in [Creating a New Heron Release](compiling-overview#building-all-components).
-
-Command for fetching the binary is in the `heron.aurora` config file. By default it is
-using a `curl` command to fetch the binary. For example, if the binary is hosted in
-HDFS, you need to change the fetch user package command in `heron.aurora` to use the
-`hdfs` command instead of `curl`.
-
-### `heron.aurora` example binary fetch using HDFS
-
-```bash
-fetch_heron_system = Process(
- name = 'fetch_heron_system',
- cmdline = 'hdfs dfs -get %s %s && tar zxf %s' % (heron_core_release_uri,
- core_release_file, core_release_file)
-)
-```
-
-Once your Heron binaries are hosted somewhere that is accessible to Aurora, you
-should run tests to ensure that Aurora can successfully fetch them.
-
-## Uploading the Topologies
-
-Heron uses an uploader to upload the topology to a shared location so that a worker can fetch
-the topology to its sandbox. The configuration for an uploader is in the `uploader.yaml`
-config file. For distributed Aurora deployments, Heron can use `HdfsUploader` or `S3Uploader`.
-Details on configuring the uploaders can be found in the documentation for the
-[HDFS](uploaders-hdfs) and [S3](uploaders-amazon-s3) uploaders.
-
-After configuring an uploader, the `heron.aurora` config file needs to be modified accordingly to
-fetch the topology.
-
-### `heron.aurora` example topology fetch using HDFS
-
-```bash
-fetch_user_package = Process(
- name = 'fetch_user_package',
- cmdline = 'hdfs dfs -get %s %s && tar zxf %s' % (heron_topology_jar_uri,
- topology_package_file, topology_package_file)
-)
-```
-
-## Aurora Scheduler Configuration
-
-To configure Heron to use Aurora scheduler, modify the `scheduler.yaml`
-config file specific for the Heron cluster. The following must be specified
-for each cluster:
-
-* `heron.class.scheduler` --- Indicates the class to be loaded for Aurora scheduler.
-You should set this to `org.apache.heron.scheduler.aurora.AuroraScheduler`
-
-* `heron.class.launcher` --- Specifies the class to be loaded for launching and
-submitting topologies. To configure the Aurora launcher, set this to
-`org.apache.heron.scheduler.aurora.AuroraLauncher`
-
-* `heron.package.core.uri` --- Indicates the location of the heron core binary package.
-The local scheduler uses this URI to download the core package to the working directory.
-
-* `heron.directory.sandbox.java.home` --- Specifies the java home to
-be used when running topologies in the containers.
-
-* `heron.scheduler.is.service` --- This config indicates whether the scheduler
-is a service. In the case of Aurora, it should be set to `False`.
-
-### Example Aurora Scheduler Configuration
-
-```yaml
-# scheduler class for distributing the topology for execution
-heron.class.scheduler: org.apache.heron.scheduler.aurora.AuroraScheduler
-
-# launcher class for submitting and launching the topology
-heron.class.launcher: org.apache.heron.scheduler.aurora.AuroraLauncher
-
-# location of the core package
-heron.package.core.uri: file:///vagrant/.herondata/dist/heron-core-release.tar.gz
-
-# location of java - pick it up from shell environment
-heron.directory.sandbox.java.home: /usr/lib/jvm/java-1.8.0-openjdk-amd64/
-
-# Invoke the IScheduler as a library directly
-heron.scheduler.is.service: False
-```
-
-## Working with Topologies
-
-After setting up ZooKeeper and generating an Aurora-accessible Heron core binary
-release, any machine that has the `heron` cli tool can be used to manage Heron
-topologies (i.e. can submit topologies, activate and deactivate them, etc.).
-
-The most important thing at this stage is to ensure that `heron` cli is available
-across all machines. Once the cli is available, Aurora as a scheduler
-can be enabled by specifying the proper configuration when managing topologies.
diff --git a/website2/docs/schedulers-aurora-local.md b/website2/docs/schedulers-aurora-local.md
deleted file mode 100644
index 7de3da7d37c..00000000000
--- a/website2/docs/schedulers-aurora-local.md
+++ /dev/null
@@ -1,329 +0,0 @@
----
-id: schedulers-aurora-local
-title: Setting up Heron with Aurora Cluster Locally on Linux
-sidebar_label: Aurora Locally
----
-<!--
- Licensed to the Apache Software Foundation (ASF) under one
- or more contributor license agreements. See the NOTICE file
- distributed with this work for additional information
- regarding copyright ownership. The ASF licenses this file
- to you under the Apache License, Version 2.0 (the
- "License"); you may not use this file except in compliance
- with the License. You may obtain a copy of the License at
- http://www.apache.org/licenses/LICENSE-2.0
- Unless required by applicable law or agreed to in writing,
- software distributed under the License is distributed on an
- "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- KIND, either express or implied. See the License for the
- specific language governing permissions and limitations
- under the License.
--->
-
-It is possible to setup Heron with a locally running Apache Aurora cluster.
-This is a step by step guide on how to configure and setup all the necessary
-components.
-
-## Setting Up Apache Aurora Cluster locally
-
-You first need to setup Apache Aurora locally. More detailed description of the
-following steps can be found in [A local Cluster with Vagrant](http://aurora.apache.org/documentation/latest/getting-started/vagrant/)
-
-### Step 1: Install VirtualBox and Vagrant
-
-Download and install VirtualBox and Vagrant on your machine. If vagrant is successfully
-installed in your machine the following command should list several common commands
-for this tool
-
-```bash
-$ vagrant
-```
-
-### Step 2: Clone the Aurora repository
-
-You can get the source repository for Aurora with the following command
-
-```bash
-$ dgit clone git://git.apache.org/aurora.git
-```
-
-Once the clone is complete cd into the aurora folder
-
-```bash
-$ cd aurora
-```
-
-### Step 3: Starting Local Aurora Cluster
-
-To start the local cluster all you have to do is execute the following command. It will install all the needed dependencies like Apache Mesos and Zookeeper in the VM.
-
-```bash
-$ vagrant up
-```
-
-Additionally to get rid of some of the warning messages that you get during up command execute the following command
-
-```bash
-$ vagrant plugin install vagrant-vbguest
-```
-
-You can verify that the Aurora cluster is properly running by opening the following links in your web-browser
-
-* Scheduler - http://192.168.33.7:8081
-* Observer - http://192.168.33.7:1338
-* Mesos Master - http://192.168.33.7:5050
-* Mesos Agent - http://192.168.33.7:5051
-
-If you go into http://192.168.33.7:8081/scheduler you can notice that the name of the default cluster that is setup in aurora is
-named `devcluster` this will be important to note when submitting typologies from heron.
-
-
-
-## Installing Heron within the Cluster VM
-
-Now that the Aurora cluster is setup you need to install heron within the cluster VM in order to be able to get the Heron
-deployment working. Since this is a fresh VM instance you will have to install the basic software such as "unzip" and set
-the JAVA_HOME path as an environmental variable ( Just need to add this to .bashrc file). After you have the basic stuff
-working follow the following steps to install Heron in the VM. You can ssh into the VM with the following command
-
-```bash
-$ vagrant ssh
-```
-
-### Step 1.a : Download installation script files
-
-You can download the script files that match your Linux distribution from
-https://github.com/apache/incubator-heron/releases/tag/{{% heronVersion %}}
-
-For example for the {{% heronVersion %}} release the files you need to download For Ubuntu will be the following.
-
-* `heron-install-{{% heronVersion %}}-ubuntu.sh`
-
-Optionally - You want need the following for the steps in the blog post
-
-* `heron-api-install-{{% heronVersion %}}-ubuntu.sh`
-* `heron-core-{{% heronVersion %}}-ubuntu.tar.gz`
-
-### Step 1.b: Execute the client and tools shell scripts
-
-
-```bash
-$ chmod +x heron-install-VERSION-PLATFORM.sh
-$ ./heron-install-VERSION-PLATFORM.sh --user
-Heron client installer
-----------------------
-
-Uncompressing......
-Heron is now installed!
-
-Make sure you have "/home/vagrant/bin" in your path.
-```
-
-After this you need to add the path "/home/vagrant/bin". You can just execute the following command
-or add it to the end of .bashrc file ( which is more convenient ).
-
-```bash
-$ export PATH=$PATH:/home/vagrant/bin
-```
-
-Install the following packages to make sure that you have all the needed dependencies in the VM.
-You might have to do sudo apt-get update before you execute the following.
-
-```bash
-$ sudo apt-get install git build-essential automake cmake libtool zip libunwind-setjmp0-dev zlib1g-dev unzip pkg-config -y
-```
-
-## Configuring State Manager ( Apache Zookeeper )
-
-Since Heron only uses Apache Zookeeper for coordination the load on the Zookeeper
-node is minimum. Because of this it is sufficient to use a single Zookeeper node or
-if you have an Zookeeper instance running for some other task you can simply use that.
-Since Apache Aurora already uses an Zookeeper instance you can directly use that instance
-to execute State Manager tasks of Heron. First you need to configure Heron to work with
-the Zookeeper instance. You can find meanings of each attribute in [Setting Up ZooKeeper
-State Manager](state-managers-zookeeper). Configurations for State manager are
-located in the directory `/home/vagrant/.heron/conf/aurora`.
-
-Open the file `statemgr.yaml` using vim ( or some other text editor you prefer )
-and add/edit the file to include the following.
-
-```yaml
-# local state manager class for managing state in a persistent fashion
-heron.class.state.manager: org.apache.heron.statemgr.zookeeper.curator.CuratorStateManager
-
-# local state manager connection string
-heron.statemgr.connection.string: "127.0.0.1:2181"
-
-# path of the root address to store the state in a local file system
-heron.statemgr.root.path: "/heronroot"
-
-# create the zookeeper nodes, if they do not exist
-heron.statemgr.zookeeper.is.initialize.tree: True
-
-# timeout in ms to wait before considering zookeeper session is dead
-heron.statemgr.zookeeper.session.timeout.ms: 30000
-
-# timeout in ms to wait before considering zookeeper connection is dead
-heron.statemgr.zookeeper.connection.timeout.ms: 30000
-
-# timeout in ms to wait before considering zookeeper connection is dead
-heron.statemgr.zookeeper.retry.count: 10
-
-# duration of time to wait until the next retry
-heron.statemgr.zookeeper.retry.interval.ms: 10000
-```
-
-## Creating Paths in Zookeeper
-
-Next you need to create some paths within Zookeeper since some of the paths
-are not created by Heron automatically. So you need to create them manually.
-Since Aurora installation already installed Zookeeper, you can use the Zookeeper
-cli to create the manual paths.
-
-```bash
-$ sudo ./usr/share/zookeeper/bin/zkCli.sh
-```
-
-This will connect to the Zookeeper instance running locally. Then execute the
-following commands from within the client to create paths `/heronroot/topologies`
-and `/heron/topologies`. Later in "Associating new Aurora cluster into Heron UI"
-you will see that you only need to create `/heronroot/topologies` but for now lets
-create both to make sure you don't get any errors when you run things.
-
-```bash
-create /heronroot null
-create /heronroot/topologies null
-```
-
-```bash
-create /heron null
-create /heron/topologies null
-```
-
-## Configuring Scheduler ( Apache Aurora )
-
-Next you need to configure Apache Aurora to be used as the Scheduler for our Heron
-local cluster. In order to do this you need to edit the `scheduler.yaml` file that is
-also located in `/home/vagrant/.heron/conf/aurora`. Add/Edit the file to include the
-following. More information regarding parameters can be found in [Aurora Cluster](schedulers-aurora-cluster)
-
-```yaml
-# scheduler class for distributing the topology for execution
-heron.class.scheduler: org.apache.heron.scheduler.aurora.AuroraScheduler
-
-# launcher class for submitting and launching the topology
-heron.class.launcher: org.apache.heron.scheduler.aurora.AuroraLauncher
-
-# location of the core package
-heron.package.core.uri: file:///home/vagrant/.heron/dist/heron-core.tar.gz
-
-# location of java - pick it up from shell environment
-heron.directory.sandbox.java.home: /usr/lib/jvm/java-1.8.0-openjdk-amd64/
-
-# Invoke the IScheduler as a library directly
-heron.scheduler.is.service: False
-```
-
-Additionally edit the `client.yaml` file and change the core uri to make it consistant.
-
-```yaml
-# location of the core package
-heron.package.core.uri: file:///home/vagrant/.heron/dist/heron-core.tar.gz
-```
-
-### Important Step: Change folder name `aurora` to `devcluster`
-
-Next you need to change the folder name of `/home/vagrant/.heron/conf/aurora` to
-`/home/vagrant/.heron/conf/devcluster`. This is because the name of your aurora
-cluster is `devcluster` as you noted in a previous step. You can do this with the
-following commands
-
-```bash
-$ cd /home/vagrant/.heron/conf/
-$ mv aurora devcluster
-```
-
-## Submitting Example Topology to Aurora cluster
-
-Now you can submit a topology to the aurora cluster. this can be done with the following command.
-
-```bash
-$ heron submit devcluster/heronuser/devel --config-path ~/.heron/conf/ ~/.heron/examples/heron-api-examples.jar org.apache.heron.examples.api.ExclamationTopology ExclamationTopology
-```
-
-Now you should be able to see the topology in the Aurora UI ( http://192.168.33.7:8081/scheduler/heronuser ) .
-
-
-
-### Understanding the parameters
-
-
-below is a brief explanation on some of the important parameters that are used in this command. the first
-parameter `devcluster/heronuser/devel` defines cluster, role and env ( env can have values `prod | devel | test | staging` ).
-The cluster is the name of the aurora cluster which is `devcluster` in our case. You can give something like your
-name for the role name and for env you need to choose from one of the env values.
-
-`--config-path` points to the config folder. the program will automatically look for a folder with the cluster name.
-This is why you had to change the name of the aurora conf folder to devcluster.
-
-Now that everything is working you need to perform one last step to be able to see the typologies that you can see in Aurora UI in Heron UI.
-
-## Associating new Aurora cluster into Heron UI
-
-Heron UI uses information that is gets from the heron tracker when displaying the information in the heron UI interface.
-So in-order to allow the Heron UI to show Aurora cluster information you need to modify configuration of the Heron tracker
-so that it can identify the Aurora Cluster.
-
-Heron Tracker configurations are located at `/home/vagrant/.herontools/conf` the configuration file is named `heron_tracker.yaml`.
-By default you should see the following in the file
-
-```yaml
-statemgrs:
- -
- type: "file"
- name: "local"
- rootpath: "~/.herondata/repository/state/local"
- tunnelhost: "localhost"
- -
- type: "zookeeper"
- name: "localzk"
- hostport: "localhost:2181"
- rootpath: "/heron"
- tunnelhost: "localhost"
-```
-
-You can see that there already two entries. Before, you had to create paths in Zookeeper for `/heron/topologies` this is
-because the entry named `localzk` in this file. If you remove this you will not need to create that path in Zookeeper.
-Now all you have to is to add a new entry for the aurora cluster into this file ( lets comment out `localzk` ).
-Then the file would look like below.
-
-```yaml
-statemgrs:
- -
- type: "file"
- name: "local"
- rootpath: "~/.herondata/repository/state/local"
- tunnelhost: "localhost"
- #-
- #type: "zookeeper"
- # name: "localzk"
- # hostport: "localhost:2181"
- # rootpath: "/heron"
- # tunnelhost: "localhost"
- -
- type: "zookeeper"
- name: "devcluster"
- hostport: "localhost:2181"
- rootpath: "/heronroot"
- tunnelhost: "localhost"
-```
-
-Now you can start Heron tracker and then Heron UI, Now you will be able to see the aurora cluster from the
-Heron UI ( http://192.168.33.7:8889/topologies ) as below
-
-```bash
-$ heron-tracker
-$ heron-ui
-```
-
-
[incubator-heron] 07/07: [MetricsMgr] Switched test to using Kubernetes in URI.
Posted by sa...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
saadurrahman pushed a commit to branch saadurrahman/3829-Deprecate-Apache-Aurora-dev
in repository https://gitbox.apache.org/repos/asf/incubator-heron.git
commit 8cce8b38d2b35132b859c6d22edc6254519e19b4
Author: Saad Ur Rahman <sa...@apache.org>
AuthorDate: Mon May 9 17:35:41 2022 -0400
[MetricsMgr] Switched test to using Kubernetes in URI.
---
.../java/org/apache/heron/metricsmgr/sink/PrometheusSinkTests.java | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/heron/metricsmgr/tests/java/org/apache/heron/metricsmgr/sink/PrometheusSinkTests.java b/heron/metricsmgr/tests/java/org/apache/heron/metricsmgr/sink/PrometheusSinkTests.java
index b73b1dde420..6a96c9c9c18 100644
--- a/heron/metricsmgr/tests/java/org/apache/heron/metricsmgr/sink/PrometheusSinkTests.java
+++ b/heron/metricsmgr/tests/java/org/apache/heron/metricsmgr/sink/PrometheusSinkTests.java
@@ -227,7 +227,7 @@ public class PrometheusSinkTests {
);
records = Arrays.asList(
- newRecord("shared-aurora-036:31/spout-release-1/container_1_spout-release-1_31",
+ newRecord("shared-kubernetes-036:31/spout-release-1/container_1_spout-release-1_31",
infos, Collections.emptyList())
);
PrometheusTestSink sink = new PrometheusTestSink();