You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@kafka.apache.org by ew...@apache.org on 2018/10/05 17:22:22 UTC
[kafka] branch 2.0 updated: MINOR: Switch to use AWS spot instances
This is an automated email from the ASF dual-hosted git repository.
ewencp pushed a commit to branch 2.0
in repository https://gitbox.apache.org/repos/asf/kafka.git
The following commit(s) were added to refs/heads/2.0 by this push:
new b49c721 MINOR: Switch to use AWS spot instances
b49c721 is described below
commit b49c721ba1b324492a53056c71980fae5d1c2470
Author: Max Zheng <ma...@gmail.com>
AuthorDate: Fri Oct 5 10:21:25 2018 -0700
MINOR: Switch to use AWS spot instances
Pricing for m3.xlarge: On-Demand is at $0.266. Reserved is at about $0.16 (40% discount). And Spot is at $0.0627 (76% discount relative to On-Demand, or 60% discount relative to Reserved). Insignificant fluctuation in the past 3 months.
Ran on branch builder and works as expected -- each worker is created using spot instances (https://jenkins.confluent.io/job/system-test-kafka-branch-builder/1982/console)
This can be safely backported to 0.10.2 (tested using https://jenkins.confluent.io/job/system-test-kafka-branch-builder/1983/)
Author: Max Zheng <ma...@gmail.com>
Reviewers: Ewen Cheslack-Postava <ew...@confluent.io>
Closes #5707 from maxzheng/minor-switch@trunk
(cherry picked from commit 50ec82940d4af61f12300235b7553bd5cf231894)
Signed-off-by: Ewen Cheslack-Postava <me...@ewencp.org>
---
Vagrantfile | 6 ++++++
tests/README.md | 1 +
vagrant/aws/aws-example-Vagrantfile.local | 1 +
vagrant/aws/aws-init.sh | 11 +++++++----
4 files changed, 15 insertions(+), 4 deletions(-)
diff --git a/Vagrantfile b/Vagrantfile
index 3636076..88f2028 100644
--- a/Vagrantfile
+++ b/Vagrantfile
@@ -42,6 +42,8 @@ ec2_region = "us-east-1"
ec2_az = nil # Uses set by AWS
ec2_ami = "ami-905730e8"
ec2_instance_type = "m3.medium"
+ec2_spot_instance = ENV['SPOT_INSTANCE'] ? ENV['SPOT_INSTANCE'] == 'true' : true
+ec2_spot_max_price = "0.113" # On-demand price for instance type
ec2_user = "ubuntu"
ec2_instance_name_prefix = "kafka-vagrant"
ec2_security_groups = nil
@@ -133,6 +135,10 @@ Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
else
aws.associate_public_ip = ec2_associate_public_ip
end
+ aws.region_config ec2_region do |region|
+ region.spot_instance = ec2_spot_instance
+ region.spot_max_price = ec2_spot_max_price
+ end
# Exclude some directories that can grow very large from syncing
override.vm.synced_folder ".", "/vagrant", type: "rsync", rsync__exclude: ['.git', 'core/data/', 'logs/', 'tests/results/', 'results/']
diff --git a/tests/README.md b/tests/README.md
index f0ffdf5..f42b28a 100644
--- a/tests/README.md
+++ b/tests/README.md
@@ -461,6 +461,7 @@ the test driver machine.
ec2_instance_type = "..." # Pick something appropriate for your
# test. Note that the default m3.medium has
# a small disk.
+ ec2_spot_max_price = "0.123" # On-demand price for instance type
enable_hostmanager = false
num_zookeepers = 0
num_kafka = 0
diff --git a/vagrant/aws/aws-example-Vagrantfile.local b/vagrant/aws/aws-example-Vagrantfile.local
index ee9db9a..23187a0 100644
--- a/vagrant/aws/aws-example-Vagrantfile.local
+++ b/vagrant/aws/aws-example-Vagrantfile.local
@@ -17,6 +17,7 @@
# To use it, move it to the base kafka directory and rename
# it to Vagrantfile.local, and adjust variables as needed.
ec2_instance_type = "m3.xlarge"
+ec2_spot_max_price = "0.266" # On-demand price for instance type
enable_hostmanager = false
num_zookeepers = 0
num_brokers = 0
diff --git a/vagrant/aws/aws-init.sh b/vagrant/aws/aws-init.sh
index 7517626..54092c8 100755
--- a/vagrant/aws/aws-init.sh
+++ b/vagrant/aws/aws-init.sh
@@ -31,15 +31,18 @@ base_dir=`dirname $0`/../..
if [ -z `which vagrant` ]; then
echo "Installing vagrant..."
- wget https://releases.hashicorp.com/vagrant/1.9.3/vagrant_1.9.3_x86_64.deb
- sudo dpkg -i vagrant_1.9.3_x86_64.deb
- rm -f vagrant_1.9.3_x86_64.deb
+ wget https://releases.hashicorp.com/vagrant/2.1.5/vagrant_2.1.5_x86_64.deb
+ sudo dpkg -i vagrant_2.1.5_x86_64.deb
+ rm -f vagrant_2.1.5_x86_64.deb
fi
# Install necessary vagrant plugins
# Note: Do NOT install vagrant-cachier since it doesn't work on AWS and only
# adds log noise
-vagrant_plugins="vagrant-aws vagrant-hostmanager"
+
+# Custom vagrant-aws with spot instance support. See https://github.com/mitchellh/vagrant-aws/issues/32
+wget -nv https://s3-us-west-2.amazonaws.com/confluent-packaging-tools/vagrant-aws-0.7.2.spot.gem -P /tmp
+vagrant_plugins="/tmp/vagrant-aws-0.7.2.spot.gem vagrant-hostmanager"
existing=`vagrant plugin list`
for plugin in $vagrant_plugins; do
echo $existing | grep $plugin > /dev/null