You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@kylin.apache.org by xx...@apache.org on 2022/02/11 04:03:18 UTC
[kylin] branch kylin4_on_cloud updated (8a93937 -> 7e59491)
This is an automated email from the ASF dual-hosted git repository.
xxyu pushed a change to branch kylin4_on_cloud
in repository https://gitbox.apache.org/repos/asf/kylin.git.
from 8a93937 # adapt initialized config for default aws account limit
new 3c72949 # update README.md
new 7e59491 # update README.md
The 2 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
README.md | 36 +++---
backup/properties/default/kylin.properties | 2 +-
.../properties/templates/kylin.properties.template | 2 +-
deploy.py | 1 +
images/cpulimit.png | Bin 0 -> 51155 bytes
readme/advanced_configs.md | 42 +++----
readme/commands.md | 49 ++++----
readme/cost_calculation.md | 76 +++++++++++++
readme/prerequisites.md | 123 ++++++++++++++-------
readme/quick_start.md | 33 ++++--
readme/quick_start_for_multiple_clusters.md | 8 +-
readme/trouble_shooting.md | 98 ++++++++++++++++
12 files changed, 353 insertions(+), 117 deletions(-)
create mode 100644 images/cpulimit.png
create mode 100644 readme/cost_calculation.md
create mode 100644 readme/trouble_shooting.md
[kylin] 01/02: # update README.md
Posted by xx...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
xxyu pushed a commit to branch kylin4_on_cloud
in repository https://gitbox.apache.org/repos/asf/kylin.git
commit 3c7294953db2a13b929f8ede3bb019e7c0c71059
Author: Mukvin <bo...@163.com>
AuthorDate: Tue Feb 8 18:20:08 2022 +0800
# update README.md
---
README.md | 36 +++---
backup/properties/default/kylin.properties | 2 +-
.../properties/templates/kylin.properties.template | 2 +-
images/cpulimit.png | Bin 0 -> 51155 bytes
readme/advanced_configs.md | 42 +++----
readme/commands.md | 32 +++---
readme/cost_calculation.md | 76 +++++++++++++
readme/prerequisites.md | 121 +++++++++++++++------
readme/quick_start.md | 28 +++--
readme/quick_start_for_multiple_clusters.md | 8 +-
readme/trouble_shooting.md | 98 +++++++++++++++++
11 files changed, 340 insertions(+), 105 deletions(-)
diff --git a/README.md b/README.md
index c400a83..6847bc2 100644
--- a/README.md
+++ b/README.md
@@ -1,8 +1,8 @@
-# Welcome To Kylin 4 On Cloud Deployment Solution !
+# Welcome To Kylin 4 On Cloud Deployment Solution!
-**Apache Kylin community** released Kylin 4.0 with a new architecture, which dedicated to building a high-performance and low-cost OLAP engine. The architecture of Kylin 4.0 supports the separation of storage and computing, which enables Kylin users to run Kylin 4.0 by adopting a more flexible and elastically scalable cloud deployment method.
+**Apache Kylin community** released Kylin 4.0 with a new architecture, which is dedicated to building a high-performance and low-cost OLAP engine. The architecture of Kylin 4.0 supports the separation of storage and computing, which enables Kylin users to run Kylin 4.0 by adopting a more flexible and elastically scalable cloud deployment method.
-For the best practices of Kylin4 on the cloud, **Apache Kylin community** contribute a **tool** to deploy kylin4 clusters on **AWS** cloud easily and conveniently.
+For the best practices of Kylin4 on the cloud, **Apache Kylin community contributes a **tool** to deploy kylin4 clusters on **AWS** cloud easily and conveniently.
# Introduction About This Tool
@@ -10,17 +10,17 @@ For the best practices of Kylin4 on the cloud, **Apache Kylin community** con
1. Deploy a Kylin4 cluster on Ec2 with Spark Standalone mode in `10` minutes.
2. Support to scale nodes (Kylin & Spark Worker) quickly and conveniently.
-3. Improve performance for query of Kylin4 in using `Local Cache + Soft Affinity` feature (`Experimental Feature`), please check the [details](https://kylin.apache.org/blog/2021/10/21/Local-Cache-and-Soft-Affinity-Scheduling/).
-4. Support to monitor status of cluster with `prometheus server` and `granfana`.
+3. Improve performance for the query of Kylin4 in using `Local Cache + Soft Affinity` feature (`Experimental Feature`), please check the [details](https://kylin.apache.org/blog/2021/10/21/Local-Cache-and-Soft-Affinity-Scheduling/).
+4. Support to monitor the status of the cluster with the `Prometheus server` and `Granfana`.
## Architecture
-When cluster(s) created, services and nodes will like below:
+When cluster(s) created, services and nodes will be like below:
- **Every stack module means related services will be controlled by a stack.**
-- **Read-write separated cluster will be easy created as same as image of architecture above.**
+- The **read-write separated cluster will be easily created as same as the image of architecture above.**
- Services are created as the number order from 1 to 5.
- Every machine node is presented by a white box.
- `Kylin Node` and `Spark Worker` Node can be easy to scale.
@@ -31,22 +31,24 @@ When cluster(s) created, services and nodes will like below:
## Quick Start
- Details about **`quick start`** of tool, please referer to [quick start](./readme/quick_start.md).
-- Details about **`quick start for mutilple clusters`** of tool, please referer to [quick start for mutilple clusters](./readme/quick_start_for_multiple_clusters.md).
+- Details about **`quick start for multiple clusters`** of tool, please referer to [quick start for multiple clusters](./readme/quick_start_for_multiple_clusters.md).
## Notes
-1. More details about `commands` of tool, see document [commands](./readme/commands.md).
-2. More details about `prerequisites` of tool, see document [prerequisites](./readme/prerequisites.md).
-3. More details about `advanced configs` of tool, see document [advanced configs](./readme/advanced_configs.md).
-4. More details about `monitor services` supported by tool, see document [monitor](./readme/monitor.md).
-5. Current tool already open the public port for some services. You can access the service by `public ip` of related EC2 instance.
+1. For more details about `cost` of tool, see document [cost calculation](./readme/cost_calculation.md).
+2. For more details about `commands` of tool, see document [commands](./readme/commands.md).
+3. For more details about the `prerequisites` of tool, see document [prerequisites](./readme/prerequisites.md).
+4. For more details about `advanced configs` of tool, see document [advanced configs](./readme/advanced_configs.md).
+5. For more details about `monitor services` supported by tool, see document [monitor](./readme/monitor.md).
+6. For more details about `troubleshooting`, see document [troubleshooting](./readme/trouble_shooting.md).
+7. The current tool has already opened the public port for some services. You can access the service by `public IP` of related EC2 instances.
1. `SSH`: 22.
2. `Granfana`: 3000.
- 3. `Prmetheus`: 9090, 9100.
+ 3. `Prometheus`: 9090, 9100.
4. `Kylin`: 7070.
5. `Spark`: 8080, 4040.
-6. More about cloudformation syntax, please check [aws website](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/Welcome.html).
-7. Current Kylin version is 4.0.0.
-8. Current Spark version is 3.1.1.
+8. More about cloudformation syntax, please check [aws website](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/Welcome.html).
+9. The current Kylin version is 4.0.0.
+10. The current Spark version is 3.1.1.
diff --git a/backup/properties/default/kylin.properties b/backup/properties/default/kylin.properties
index f800b5d..0bd043c 100644
--- a/backup/properties/default/kylin.properties
+++ b/backup/properties/default/kylin.properties
@@ -26,7 +26,7 @@ kylin.engine.spark-conf.spark.eventLog.dir=s3a:/{{ S3_BUCKET_PATH }}/working_dir
kylin.engine.spark-conf.spark.history.fs.logDirectory=s3a:/{{ S3_BUCKET_PATH }}/working_dir/spark-history
kylin.engine.spark-conf.spark.master=spark://{{ SPARK_MASTER }}:7077
-kylin.cube.cubeplanner.enabled=true
+kylin.cube.cubeplanner.enabled=false
kylin.engine.spark-conf.spark.executor.cores=3
kylin.engine.spark-conf.spark.executor.instances=20
kylin.engine.spark-conf.spark.executor.memory=12GB
diff --git a/backup/properties/templates/kylin.properties.template b/backup/properties/templates/kylin.properties.template
index 94dd2dd..41f4a96 100644
--- a/backup/properties/templates/kylin.properties.template
+++ b/backup/properties/templates/kylin.properties.template
@@ -24,7 +24,7 @@ kylin.engine.spark-conf.spark.eventLog.dir=s3a:/{{ S3_BUCKET_PATH }}/working_dir
kylin.engine.spark-conf.spark.history.fs.logDirectory=s3a:/{{ S3_BUCKET_PATH }}/working_dir/spark-history
kylin.engine.spark-conf.spark.master=spark://{{ SPARK_MASTER }}:7077
-kylin.cube.cubeplanner.enabled=true
+kylin.cube.cubeplanner.enabled=false
kylin.engine.spark-conf.spark.executor.cores=3
kylin.engine.spark-conf.spark.executor.instances=20
kylin.engine.spark-conf.spark.executor.memory=12GB
diff --git a/images/cpulimit.png b/images/cpulimit.png
new file mode 100644
index 0000000..972bb49
Binary files /dev/null and b/images/cpulimit.png differ
diff --git a/readme/advanced_configs.md b/readme/advanced_configs.md
index cd60193..cd36d3b 100644
--- a/readme/advanced_configs.md
+++ b/readme/advanced_configs.md
@@ -4,56 +4,56 @@
There are `9` modules params for tools. Introductions as below:
-- EC2_VPC_PARAMS: this params of module are for creating a vpc.
+- EC2_VPC_PARAMS: These params of the module are for creating a VPC.
-- EC2_RDS_PARAMS: this params of module are for creating a RDS.
+- EC2_RDS_PARAMS: These params of the module are for creating an RDS.
-- EC2_STATIC_SERVICES_PARAMS: this params of module are for creating a Prometheus Server and other static services.
+- EC2_STATIC_SERVICES_PARAMS: These params of the module are for creating a Prometheus Server and other static services.
-- EC2_ZOOKEEPERS_PARAMS: this params of module are for creating a Zookeeper Cluster.
+- EC2_ZOOKEEPERS_PARAMS: These params of the module are for creating a Zookeeper Cluster.
-- EC2_SPARK_MASTER_PARAMS: this params of module are for creating a Spark Master node.
+- EC2_SPARK_MASTER_PARAMS: These params of the module are for creating a Spark Master node.
-- EC2_KYLIN4_PARAMS: this params of module are for creating a Kylin4.
+- EC2_KYLIN4_PARAMS: These params of the module are for creating a Kylin4.
-- EC2_SPARK_WORKER_PARAMS: this params of module are for creating **Spark Workers**, default is **3** spark workers for all clusters.
+- EC2_SPARK_WORKER_PARAMS: These params of the module are for creating **Spark Workers**, the default is **3** spark workers for all clusters.
-- EC2_KYLIN4_SCALE_PARAMS: this params of module are for scaling **Kylin4 nodes**, the range of **Kylin4 nodes** is related to `KYLIN_SCALE_UP_NODES` and `KYLIN_SCALE_DOWN_NODES`.
+- EC2_KYLIN4_SCALE_PARAMS: these params of the module are for scaling **Kylin4 nodes**, the range of **Kylin4 nodes** is related to `KYLIN_SCALE_UP_NODES` and `KYLIN_SCALE_DOWN_NODES`.
> Note:
>
- > 1. `KYLIN_SCALE_UP_NODES` is for the range of kylin nodes to scale up.
- > 2. `KYLIN_SCALE_DOWN_NODES` is for the range of kylin nodes to scale down.
- > 3. The range of `KYLIN_SCALE_UP_NODES` must be contain the range of `KYLIN_SCALE_DOWN_NODES`.
- > 4. **They are effective to all clusters which is not only `default cluster` but also other cluster which index is in `${CLUSTER_INDEXES}`.**
+ > 1. `KYLIN_SCALE_UP_NODES` is for the range of Kylin nodes to scale up.
+ > 2. `KYLIN_SCALE_DOWN_NODES` is for the range of Kylin nodes to scale down.
+ > 3. The range of `KYLIN_SCALE_UP_NODES` must contain the range of `KYLIN_SCALE_DOWN_NODES`.
+ > 4. **They are effective to all clusters which are not only `default cluster` but also another cluster whose index is in `${CLUSTER_INDEXES}`.**
-- EC2_SPARK_SCALE_SLAVE_PARAMS: this params of module are for scaling **Spark workers**, the range of **Spark Workers ** is related to `SPARK_WORKER_SCALE_UP_NODES` and `SPARK_WORKER_SCALE_DOWN_NODES`.
+- EC2_SPARK_SCALE_SLAVE_PARAMS: these params of the module are for scaling **Spark workers**, the range of **Spark Workers is related to `SPARK_WORKER_SCALE_UP_NODES` and `SPARK_WORKER_SCALE_DOWN_NODES`.
> Note:
>
- > 1. `SPARK_WORKER_SCALE_UP_NODES` is for the range for spark workers to scale up. **It's effective to all clusters which is not only `default cluster` but also other cluster which index is in `${CLUSTER_INDEXES}`.**
- > 2. `SPARK_WORKER_SCALE_DOWN_NODES` is for the range for spark workers to scale down. **It's effective to all clusters which is not only `default cluster` but also other cluster which index is in `${CLUSTER_INDEXES}`.**
- > 3. The range of `SPARK_WORKER_SCALE_UP_NODES` must be contain the range of `SPARK_WORKER_SCALE_DOWN_NODES`.
- > 4. **They are effective to all clusters which is not only `default cluster` but also other cluster which index is in `${CLUSTER_INDEXES}`.**
+ > 1. `SPARK_WORKER_SCALE_UP_NODES` is for the range for spark workers to scale up. **It's effective to all clusters which are not only `default cluster` but also another cluster whose index is in `${CLUSTER_INDEXES}`.**
+ > 2. `SPARK_WORKER_SCALE_DOWN_NODES` is for the range for spark workers to scale down. **It's effective to all clusters which are not only `default cluster` but also another cluster whose index is in `${CLUSTER_INDEXES}`.**
+ > 3. The range of `SPARK_WORKER_SCALE_UP_NODES` must contain the range of `SPARK_WORKER_SCALE_DOWN_NODES`.
+ > 4. **They are effective to all clusters which are not only `default cluster` but also another cluster whose index is in `${CLUSTER_INDEXES}`.**
### Customize Configs
-User also can customize the params in `kylin_configs.yaml` to create an expected instances. Such as **the type of instance**, **the volume size of instance** and **the volumn type of instance** and so on.
+User also can customize the params in `kylin_configs.yaml` to create an expected instance. Such as **the type of instance**, **the volume size of the instance** and **the volume type of instance,** and so on.
1. If you want to customize configs for instances, you must modify the `EC2Mode` from `test` to `product` in `kylin_configs.yml`.
2. `Ec2Mode` is only in the parms of `EC2_STATIC_SERVICES_PARAMS`, `EC2_ZOOKEEPERS_PARAMS`, `EC2_SPARK_MASTER_PARAMS`, `EC2_KYLIN4_PARAMS`, `EC2_SPARK_WORKER_PARAMS`, `EC2_KYLIN4_SCALE_PARAMS` and `EC2_SPARK_SCALE_SLAVE_PARAMS`.
3. So instances can be customized to effect `Monitor Node`(`EC2_STATIC_SERVICES_PARAMS`), `Zookeeper Nodes`(`EC2_ZOOKEEPERS_PARAMS`), `Spark Master Node` ( `EC2_SPARK_MASTER_PARAMS`), `Kylin4 Node`( `EC2_KYLIN4_PARAMS`), `Spark workers `(`EC2_SPARK_WORKER_PARAMS`), `Kylin4 scale nodes`(`EC2_KYLIN4_SCALE_PARAMS`) and `Spark scale workers`(`EC2_SPARK_SCALE_SLAVE_PARAMS`).
4. Now`Ec2Mode` **only effect** the related params are `Ec2InstanceTypeFor*`,`Ec2VolumeSizeFor*` and `Ec2VolumnTypeFor`* in the params modules.
-5. If you don't change `ENABLE_LOCAL_CACHE_SOFT_AFFINITY` from `"false"` to `"true"` then cluster will created normally without `Local Cache + Soft Affinity` feature!
+5. If you don't change `ENABLE_LOCAL_CACHE_SOFT_AFFINITY` from `"false"` to `"true"` then the cluster will be created normally without the `Local Cache + Soft Affinity` feature!
#### Example
As an example in `EC2_STATIC_SERVICES_PARAMS`:
-- change `Ec2Mode` from `test`to `product`.
+- change `Ec2Mode` from `test` to `product`.
- change `Ec2InstanceTypeForStaticServices` from `m5.2xlarge` to `m5.4xlarge`.
- change `Ec2VolumeSizeForStaticServicesNode` from `'20'` to `'50'`.
- change `Ec2VolumnTypeForStaticServicesNode` from `gp2` to `standard`.
-- Then create the node of static service node will be a `m5.4xlarge` and it attach a volume which size is `50` and type is `standard`.
+- Then create the node of static service node will be an `m5.4xlarge` and it attaches a volume which size is `50` and type is `standard`.
diff --git a/readme/commands.md b/readme/commands.md
index 99e6306..99821ec 100644
--- a/readme/commands.md
+++ b/readme/commands.md
@@ -10,15 +10,15 @@ python deploy.py --type [deploy|destroy|list|scale] --scale-type [up|down] --nod
- destroy: destroy created cluster(s).
-- list: list alive nodes which are with stack name, instance id, private ip and public ip.
+- list: list alive nodes which are with stack name, instance id, private IP, and public IP.
- scale: Must be used with `--scale-type` and `--node-type`.
> Note:
>
- > 1. Current support to scale up/down `kylin` or `spark_worker` for specific cluster.
- > 2. Before scale up/down `kylin` or `spark_worker` nodes, Cluster services must be ready.
- > 3. If you want to scale a `kylin` or `spark_worker` node to a specify cluster, please add the `--cluster ${cluster num}` to specify the expected node add to the cluster `${cluster num}`.
+ > 1. Current support to scale up/down `kylin` or `spark_worker` for a specific cluster.
+ > 2. Before scaling up/down `kylin` or `spark_worker` nodes, Cluster services must be ready.
+ > 3. If you want to scale a `kylin` or `spark_worker` node to a specified cluster, please add the `--cluster ${cluster num}` to specify the expected node add to the cluster `${cluster num}`.
### Command for deploy
@@ -28,7 +28,7 @@ python deploy.py --type [deploy|destroy|list|scale] --scale-type [up|down] --nod
$ python deploy.py --type deploy [--cluster default]
```
-- Deploy a cluster with specific cluster index.
+- Deploy a cluster with a specific cluster index. <a name="deploycluster"></a>
```shell
$ python deploy.py --type deploy --cluster ${cluster num}
@@ -36,7 +36,7 @@ $ python deploy.py --type deploy --cluster ${cluster num}
> Note: the `${cluster num}` must be in the range of `CLUSTER_INDEXES`.
-- Deploy all cluster which contain default cluster and all cluster which index in the range of `CLUSTER_INDEXES`.
+- Deploy all clusters which contain the default cluster and all clusters whose index is in the range of `CLUSTER_INDEXES`.
```shell
$ python deploy.py --type deploy --cluster all
@@ -46,7 +46,7 @@ $ python deploy.py --type deploy --cluster all
> Note:
>
-> Destroy all cluster will not delete vpc, rds and monitor node. So if user don't want to hold the env, please set the `ALWAYS_DESTROY_ALL` to be `'true'`.
+> Destroy all clusters will not delete vpc, rds, and monitor node. So if user doesn't want to hold the env, please set the `ALWAYS_DESTROY_ALL` to be `'true'`.
- Destroy default cluster
@@ -54,7 +54,7 @@ $ python deploy.py --type deploy --cluster all
$ python deploy.py --type destroy [--cluster default]
```
-- Destroy a cluster with specific cluster index.
+- Destroy a cluster with a specific cluster index.
```shell
$ python deploy.py --type destroy --cluster ${cluster num}
@@ -62,7 +62,7 @@ $ python deploy.py --type destroy --cluster ${cluster num}
> Note: the `${cluster num}` must be in the range of `CLUSTER_INDEXES`.
-- Destroy all cluster which contain default cluster and all cluster which index in the range of `CLUSTER_INDEXES`.
+- Destroy all clusters which contain the default cluster and all clusters whose index is in the range of `CLUSTER_INDEXES`.
```shell
$ python deploy.py --type destroy --cluster all
@@ -70,7 +70,7 @@ $ python deploy.py --type destroy --cluster all
### Command for list
-- List nodes which are with **stack name**, **instance id**, **private ip** and **public ip** in **available stacks** .
+- List nodes that are with **stack name**, **instance id**, **private IP,** and **public IP** in **available stacks**.
```shell
$ python deploy.py --type list
@@ -81,14 +81,14 @@ $ python deploy.py --type list
> Note:
>
> 1. Scale command must be used with `--scale-type` and `--node-type`.
-> 2. If scale command not specify a cluster num, then the scaled node(kylin or spark worker) will be add to `default`cluster.
+> 2. If the scale command does not specify a cluster num, then the scaled node(Kylin or spark worker) will be added to the `default` cluster.
> 3. Scale command **not support** to **scale** node (kylin or spark worker) to **all clusters** at **one time**. It means that `python ./deploy.py --type scale --scale-type up[|down] --node-type kylin[|spark_worker] --cluster all` is invalid commad.
-> 4. Scale params which are `KYLIN_SCALE_UP_NODES`, `KYLIN_SCALE_DOWN_NODES`, `SPARK_WORKER_SCALE_UP_NODES` and `SPARK_WORKER_SCALE_DOWN_NODES` effect on all cluster. So if user want to scale node for a specify cluster, then modify the scale params before **every run time.**
-> 5. **(Important!!!)** Current cluster is created with default `3` spark workers and `1` kylin node. The `3` spark workers can not be scaled down. The `1` kylin node also can not be scaled down.
-> 6. **(Important!!!)** Cluster can only scale up or down the range of nodes which is in `KYLIN_SCALE_UP_NODES`, `KYLIN_SCALE_DOWN_NODES`, `SPARK_WORKER_SCALE_UP_NODES` and `SPARK_WORKER_SCALE_DOWN_NODES` . Not the default `3` spark workers and `1` kylin node in the cluster.
-> 7. **(Important!!!)** If user don't want to create a cluster with `3` default spark workers, then user can remove the useless node module in the `Ec2InstanceOfSlave0*` of `cloudformation_templates/ec2-cluster-spark-slave.yaml`. User need to know about the syntax of cloudformation as also.
+> 4. Scale params which are `KYLIN_SCALE_UP_NODES`, `KYLIN_SCALE_DOWN_NODES`, `SPARK_WORKER_SCALE_UP_NODES` and `SPARK_WORKER_SCALE_DOWN_NODES` effect on all cluster. So if user wants to scale a node for a specific cluster, then modify the scale params before **every run time.**
+> 5. **(Important!!!)** The current cluster is created with default `3` spark workers and `1` Kylin node. The `3` spark workers can not be scaled down. The `1` Kylin node also can not be scaled down.
+> 6. **(Important!!!)** The current cluster can only scale up or down the range of nodes which is in `KYLIN_SCALE_UP_NODES`, `KYLIN_SCALE_DOWN_NODES`, `SPARK_WORKER_SCALE_UP_NODES,` and `SPARK_WORKER_SCALE_DOWN_NODES`. Not the default `3` spark workers and `1` kylin node in the cluster.
+> 7. **(Important!!!)** If user doesn't want to create a cluster with `3` default spark workers, then user can remove the useless node module in the `Ec2InstanceOfSlave0*` of `cloudformation_templates/ec2-cluster-spark-slave.yaml`. User needs to know about the syntax of `cloudformation` as also.
-- Scale up/down kylin/spark workers in default cluster
+- Scale up/down Kylin/spark workers in default cluster
```shell
python deploy.py --type scale --scale-type up[|down] --node-type kylin[|spark_worker] [--cluster default]
diff --git a/readme/cost_calculation.md b/readme/cost_calculation.md
new file mode 100644
index 0000000..91691b5
--- /dev/null
+++ b/readme/cost_calculation.md
@@ -0,0 +1,76 @@
+## Cost Calculation
+
+### Cost about a cluster
+
+Full cost about a cluster by tool contains as follows:
+
+1. The type of EC2 instances.
+2. The volume of EBS for EC2 instances.
+3. The volume of S3 is in use.
+4. The data transfer for EC2 instances.
+5. The RDS.
+
+#### The type of EC2 instances
+
+A total of EC2 instances contains 3 `m5.large` (for zookeeper), 1 `m5.large`(for monitor services), 3 `m5.xlarge`(for spark slaves), 1 `m5.xlarge`(for spark master) and 1 `m5.xlarge` (for kylin).
+
+For example, if the current region is `ap-northeast-2` which is `Asia Pacific (Seoul)`, then user may cost **(3 + 1) * 0.118 + (3 + 1 + 1) * 0.236 = 1.652 USD/hourly**.
+
+> Note:
+>
+> 1. The type of EC2 instances for related services can be checked in the `cloudformation_templates/*.yaml`.
+> 2. Other scaled up/down nodes can also calculate in manually if users scaled.
+> 3. For more details about the cost of EC2 instances, please check [On-Demand Plans for Amazon EC2](https://aws.amazon.com/ec2/pricing/on-demand/).
+
+
+
+#### The volume of EBS for EC2 instances
+
+The default volume type for tool is `gp2`.
+
+A total volume size of a cluster contains 10 * 3 (for zookeeper), 20 * 1 (for monitor services), 30 * 3 (for spark slaves), 30 * 1 (for spark master) and 30 * 1(for kylin).
+
+For example, if the current region is `ap-northeast-2` which is `Asia Pacific (Seoul)`, then user may cost **[(10 * 3) + 20 * 1 + 30 * 3 + 30 * 1 + 30 * 1] * 0.114 = 22.8 USD/Per month which equals to 22.8 / (30d * 24h) = 0.0317 USD/hourly.**
+
+
+
+> Note:
+>
+> 1. The volume of EBS for EC2 instances for related services can be checked in the `cloudformation_templates/*.yaml`.
+> 2. Other scaled up/down nodes can also calculate manually if users scaled.
+> 3. For more details about the cost volume of EBS, please check [Amazon EBS pricing](https://aws.amazon.com/ebs/pricing/).
+
+
+
+#### The volume of S3 is in use
+
+the S3 volume pricing is a tiny fraction of the overall cost. **Data on S3 can be stored for the long term.**
+
+For example, if the current region is `ap-northeast-2` which is `Asia Pacific (Seoul)`, and a built data by Kylin such as `kylin_sales` and other needed files in the cluster will take about `2` GB size on the S3. So user may cost **2 * 0.025 GB = 0.05 USD.** **Note that there is no time limit.**
+
+> Note: For more details about the cost volume of S3, please check [Amazon S3 pricing](https://aws.amazon.com/s3/pricing/).
+
+#### The data transfer for EC2 instances
+
+**The tool will upload needed files for clusters to S3, and it's free.**
+
+**And Data transferred between Amazon EC2, Amazon RDS, Amazon Redshift, Amazon ElastiCache instances, and Elastic Network Interfaces in the same Availability Zone is free.**
+
+It will take a minimal fee for a user to login into EC2 instances by public IP. And it's hard to calculate. For more details please check [Data Transfer within the same AWS Region](https://aws.amazon.com/ec2/pricing/on-demand/#Data Transfer within the same AWS Region).
+
+#### The RDS
+
+The type of current RDS is `db.t3.micro`.
+
+The volume size of current RDS is `10` GB.
+
+For example, if the current region is `ap-northeast-2` which is `Asia Pacific (Seoul)`, and a user may cost **0.026 USD** for the type of RDS and 0.131 per GB-month which equals to 0.131/(30d * 24h) = **0.00018 USD/hourly**.
+
+**So it will total cost 0.026 + 0.00018 = 0.02618 USD/hourly.**
+
+
+
+### Total Cost
+
+**As discuss above, the total cost is about 1.652 + 0.0317 + 0.05 + 0.02618 = 1.75988 USD/hourly.**
+
diff --git a/readme/prerequisites.md b/readme/prerequisites.md
index 541737d..97ccdf9 100644
--- a/readme/prerequisites.md
+++ b/readme/prerequisites.md
@@ -1,6 +1,6 @@
## Prerequisites
-### Download source code & checkout to branch of `kylin4_on_cloud`
+### Download source code & checkout to a branch of `kylin4_on_cloud`
commands:
@@ -14,19 +14,32 @@ git clone https://github.com/apache/kylin.git && cd kylin && git checkout kylin4
> Note:
>
-> `IAM` role must have the access which contains `AmazonEC2RoleforSSM` , `AmazonSSMFullAccess` and `AmazonSSMManagedInstanceCore`.
+> `IAM` role must have the access which contains `AmazonEC2RoleforSSM`, `AmazonSSMFullAccess,` and `AmazonSSMManagedInstanceCore`.
>
-> This `IAM` Role will be used to initialize every ec2 instances which are for creating an kylin4 cluster on aws. And it will configure in `Initilize Env of Local Machine` part.
+> This `IAM` Role will be used to initialize every ec2 instances which is for creating a kylin4 cluster on AWS. And it will configure in `Initialize Env of Local Machine` part.
+
+
+
+> Q: Why should need an `IAM` role?
+>
+> A:
+>
+> 1. At first, the `IAM` role is an IAM identity that you can create in your account that has specific permissions.
+> 1. An IAM role does not have standard long-term credentials such as a password or access keys associated with it. Instead, when you assume a role, it provides you with temporary security credentials for your role session.
+> 1. The access will not cost money. The cost will depend on what you using on S3 and EC2 instances.
+> 1. For more details about `IAM`, please visit the [official website](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles.html) by aws.
+
+
#### II. Create a `User`
> Note:
>
-> The `User` who is created in aws will be used to execute some operations on EC2 instance. So the `User` must has some authorities as below.
+> The `User` who is created in aws will be used to execute some operations on an EC2 instance. So the `User` must have some authority as below.
| Services | Access level | Resources | Request condition |
| ------------------- | ------------------------------------------------------ | ------------- | ----------------- |
-| **Cloudformation** | **Limited**: List, Read,Write | All Resources | None |
+| **Cloudformation** | **Limited**: List, Read, Write | All Resources | None |
| **EC2** | **Limited**: Read | All Resources | None |
| **IAM** | **Limited**: List, Read, Write, Permissions management | All Resources | None |
| **RDS** | **Limited**: List, Write, Tagging | All Resources | None |
@@ -35,6 +48,19 @@ git clone https://github.com/apache/kylin.git && cd kylin && git checkout kylin4
| **STS** | **Limited**: Write | All Resources | None |
| **Systems Manager** | **Limited**: Write | All Resources | None |
+
+
+> Q: What's the relationship between the `User` and `IAM` role?
+>
+> A:
+>
+> 1. With `IAM` roles, you can establish trust relationships between your ***trusting*** account and other AWS ***trusted*** accounts.
+> 1. A `User` is an account.
+> 1. The establishment of the relationship will cost nothing.
+> 1. The cost majorly depends on what you are using on S3 and EC2 instances.
+
+
+
#### III. Login the created `User` to create key pair and Access Key<a name="keypair"></a>
> Note:
@@ -45,11 +71,11 @@ git clone https://github.com/apache/kylin.git && cd kylin && git checkout kylin4
-##### 2. Genrate a new `Access Key` which in the `My Security Credentials`
+##### 2. Generate a new `Access Key` in the `My Security Credentials`
> Note:
>
-> Please download the generated the csv file of `Access Key` immediately. Get the `Access Key` and `Secret Key` to initilize local mac to access aws.
+> Please download the generated CSV file of `Access Key` immediately. Get the `Access Key` and `Secret Key` to initialize local mac to access aws.
@@ -57,17 +83,29 @@ git clone https://github.com/apache/kylin.git && cd kylin && git checkout kylin4
+##### (!!!Important)IV. Make sure current account has 32 vCpu at least
+
+The current tool will deploy a cluster by EC2 instances, and `2 cores * 3(for zookeeper) + 2 cores(for monitor services) + 4 cores * 3(for spark slaves) + 4 cores(for spark master) + 4 cores (for kylin)=28 ` cpus to create instances in default.
+
+> Note: **32 vCPUS is the limit of an default aws account.**
+
+**User can check this quota limit in the limit of EC2 dashborad.**
+
+
+
+For more details about user current limit, please check [View your current limits](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-resource-limits.html).
+
### Initialize a S3 direcotry<a name="S3"></a>
#### I. Create a directory on S3
> Note:
>
-> This directory will store tars, scripts and jars below. And it will also be the root path of working dir for kylin4.
+> This directory will store tars, scripts, and jars below. And it will also be the root path of working dir for kylin4.
-Example: make a directory named `kylin4-aws-test` . You can also create a directory named what you like.
+Example: make a directory named `kylin4-aws-test`. You can also create a directory named what you like.
@@ -81,11 +119,10 @@ Example: make a directory named `kylin4-aws-test` . You can also create a direct
-> Create the directory named `tar` **in the path which was created by yourself**.
+> Create the directory named `tar` **in the path which was created by yourself**. For example, the full path would be `s3://.../kylin4-aws-test/tar`.
>
-> As example, the full path would be `s3://.../kylin4-aws-test/tar`.
-1. Download Kylin4 package by [official website](https://kylin.apache.org/download/).
+1. Download the Kylin4 package on the [official website](https://kylin.apache.org/download/).
2. Download Hadoop, [version 3.2.0](https://archive.apache.org/dist/hadoop/common/hadoop-3.2.0/hadoop-3.2.0.tar.gz).
3. Download Spark with hadoop3.2, [version 3.1.1](https://archive.apache.org/dist/spark/spark-3.1.1/spark-3.1.1-bin-hadoop3.2.tgz).
4. Download Hive, [version 2.3.9](https://archive.apache.org/dist/hive/hive-2.3.9/apache-hive-2.3.9-bin.tar.gz).
@@ -93,12 +130,12 @@ Example: make a directory named `kylin4-aws-test` . You can also create a direct
6. Download JDK, [version 1.8_301](https://www.oracle.com/java/technologies/javase/javase8u211-later-archive-downloads.html).
7. Download Node Exporter, [version 1.3.1.](https://github.com/prometheus/node_exporter/releases/download/v1.3.1/node_exporter-1.3.1.linux-amd64.tar.gz)
8. Download Prometheus Server, [version 2.31.1](https://github.com/prometheus/prometheus/releases/download/v2.31.1/prometheus-2.31.1.linux-amd64.tar.gz).
-9. Download Kylin4 package with local cache + soft affinity feature by [public website.](https://s3.cn-north-1.amazonaws.com.cn/public.kyligence.io/kylin/tar/apache-kylin-4.0.0-bin-spark3-soft.tar.gz)
+9. Download the Kylin4 package with local cache + soft affinity feature by the [public website.](https://s3.cn-north-1.amazonaws.com.cn/public.kyligence.io/kylin/tar/apache-kylin-4.0.0-bin-spark3-soft.tar.gz)
> Note:
-> If you want to use Kylin4 with local cache + soft affinity feature, please download the `experimental` package above.
+> If you want to use Kylin4 with a local cache + soft affinity feature, please download the `experimental` package above.
@@ -112,9 +149,8 @@ Example: make a directory named `kylin4-aws-test` . You can also create a direct
-> Create the directory named `jars` **in the path which was created by yourself**.
+> Create the directory named `jars` **in the path which was created by yourself**. For example, the full path would be `s3://.../kylin4-aws-test/jars`.
>
-> As example, the full path would be `s3://.../kylin4-aws-test/jars`.
Kylin4 needed extra jars
@@ -135,9 +171,8 @@ Kylin4 needed extra jars
-> Create the directory named `scripts` **in the path which was created by yourself**.
+> Create the directory named `scripts` **in the path which was created by yourself**. For example, the full path would be `s3://.../kylin4-aws-test/scripts`.
>
-> As example, the full path would be `s3://.../kylin4-aws-test/scripts`.
Scripts:
@@ -149,11 +184,11 @@ Scripts:
-### Initilize Env Of Local Machine
+### Initialize Env Of Local Machine
#### I. Initilize an aws account on local mac to access AWS<a name="localaws"></a>
-> Use `Access Key` and `Secret Key` above to Initialize an aws account on local mac.
+> Use `Access Key` and `Secret Key` above to Initialize an AWS account on a local mac.
```shell
$ aws configure
@@ -166,8 +201,7 @@ Default output format : json
> Note:
>
> 1. If this command got the response `-bash: aws: command not found`, please check in [Getting started with the AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-started.html).
-> 2. Region name can be in [Available Regions](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-regions-availability-zones.html#concepts-available-regions).
-> 3. Default output format is `json`.
+> 2. Region name can be in [Available Regions](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-regions-availability-zones.html#concepts-available-regions). The default output format is `json`.
#### II. Init a Python virtual env
@@ -188,29 +222,44 @@ $ bin/init.sh
**Required parameters**:
- `AWS_REGION`: Current region for EC2 instances.
-- `IAMRole`: IAM role which has the access to aws authority. This parameter will be set to created **name** of IAM role.
-- `S3_URI`: the prefix path of storing `jars/scripts/tar`. As example, this parameter will be set to `s3://.../kylin4-aws-test`.
-- `KeyName`: Security key name is a set of security credentials that you use to prove your identity when connecting to an instance. This parameter will be set to created **name** of `key pair` .
-- `CIDR_IP`: An inbound rule permits instances to receive traffic from the specified IPv4 or IPv6 CIDR address range, or from the instances associated with the specified security group.
-- `DB_IDENTIFIER`: this param should be only one in the `RDS -> Databases` . And it will be the name of created RDS database.
+- `IAMRole`: IAM role which has the access to aws authority. This parameter will be set to the created **name** of the IAM role.
+- `S3_URI`: the prefix path of storing `jars/scripts/tar`. For example, this parameter will be set to `s3://.../kylin4-aws-test`.
+- `KeyName`: Security key name is a set of security credentials that you use to prove your identity when connecting to an instance. This parameter will be set to the created **name** of key pair`.
+- `CIDR_IP`: An inbound rule permits instances to receive traffic from the specified IPv4 or IPv6 CIDR address range, or the instances associated with the specified security group.
+- `DB_IDENTIFIER`: this param should be only one in the `RDS -> Databases`. And it will be the name of created RDS database.
- `DB_PORT`: this param will be the port of created RDS database, default is `3306`.
-- `DB_USER`: this param will be a login ID for the master user of your DB instance, default is `root`.
-- `DB_PASSWORD`: this param will be the password of `DB_USER` to access DB instance. default is `123456test`, it's strongly suggest you to change it.
+- `DB_USER`: this param will be a login ID for the master user of your DB instance, the default is `root`.
+- `DB_PASSWORD`: this param will be the password of `DB_USER` to access the DB instance. default is `123456test`, it's strongly suggested you change it.
#### IV. Configure the `kylin.properties` in `backup/properties` directories.<a name="cluster"></a>
-1. The `kylin.properties` is for starting kylin instance in the cluster.
-2. Default cluster will check the `kylin.properties` in the `backup/properties/default`, and other specific cluster will check the related num directory such as `1`, `2` and `3`.
-3. User need to create new dir for the cluster num in `backup/properties`, and name it to the `${cluster num}`, such as `1`, `2` ,`3` and so on. The range of cluster num must be in `CLUSTER_INDEXES` which is configured in the `kylin_configs.yml`.
-4. Follow the `2.` step, copy the `kylin.properties.template` which is in `backup/properties/templates` to the related `${cluster num} ` directories， and rename the template to `kylin.properties`.
-5. The range of cluster nums must match the the config `CLUSTER_INDEXES`, such as `CLUSTER_INDEXES: (1, 3)` then the directories must be `1`, `2`,`3` in the `backup/properties`.
+> About the indexes of clusters:<a name="indexofcluster"></a>
+>
+> 1. **Tool will deploy a cluster which will mark to `default` if a user does not specify a cluster to deploy.**
+> 2. The range of cluster indexes must be in `CLUSTER_INDEXES` which is configured in the `kylin_configs.yml`.
+> 3. **`CLUSTER_INDEXES` is for users to create multiple clusters as READ-WRITE separation mode**, such as a user may want to create 2 query nodes of Kylin and 2 job nodes of Kylin.
+> 1. User can modify the `CLUSTER_INDEXES` to be `(1, 3)`, then as following steps of this module to modify the `kylin.properties` file for clusters.
+> 2. User can mark the Kylin node of the `default` cluster to be `query` mode and the Kylin node of the cluster whose index is `1` to be `query` mode. User can mark the Kylin node of the cluster which index is `2` to be `job` mode and the Kylin node of the cluster which index is `3` to job mode.
+> 3. User can also modify the `CLUSTER_INDEXES` to be `(1, 4)`, and mark the Kylin node of clusters whose index is `1` and `2` to be `query` mode and the Kylin node of clusters whose index is `3` and `4` to be `job` mode. Just don't use `default` to mark a cluster and execute exactly deploy cluster commands. For details about commands, please check to [deploy multiple clusters](./Commands.md#deploycluster).
+> 4. The list of mark name for clusters will be [`default`, `{cluster num}` ...] and `{cluster num}` is in the range of `CLUSTER_INDEXES`.
+> 5. For example, if `CLUSTER_INDEXES` is (1, 3) means that tool can deploy a cluster and it can be marked as `default` or `1` or `2` or `3`. And tool can execute to deploy total 1(`default`) + 3(clusters which mark name can be `1`, `2`, `3`) = 4 clusters.
+> 6. Every cluster contains 3 `Zookeepers Node`, 1 `Kylin Node`, 1 `Spark Master Node,` and 3 `Spark Slaves Node` after deployed and it can scale needed nodes of Kylin and Spark workers.
+> 7. **The difference between clusters only can be the index or customized configs of EC2 instances or customized properties of Kylin (and spark and zookeeper).**
+
+
+
+1. The `kylin.properties` is for starting kylin instance in the cluster.
+2. The default cluster will check the `kylin.properties` in the `backup/properties/default`, and other specific clusters will check the related num directory such as `1`, `2,` and `3`.
+3. User needs to create a new dir for the cluster num in `backup/properties`, and name it to the `${cluster num}`, such as `1`, `2` ,`3` and so on.
+4. Following the `2.` step, copy the `kylin.properties.template` which is in `backup/properties/templates` to the related `${cluster num} ` directories， and rename the template to `kylin.properties`.
+5. The range of cluster nums must match the config `CLUSTER_INDEXES`, such as `CLUSTER_INDEXES: (1, 3)` then the directories must be `1`, `2`,`3` in the `backup/properties`.
-6. Cusomize the config which you want to change except the config which contains `{{ }}` mark.
+6. Customize the config which you want to change except the config which contains `{{ }}` mark.
7. If the related `kylin.properties` was filled with cluster information, a file named `.rendered` will create in the directory.
-8. **(Important !!!)** If you want to re-fill the `kylin.properties` for a `kylin` node in specify cluster, you need to remove the `.rendered` file and re-copy the `kylin.propertie.template` . Redo steps from `3` to `6`.
\ No newline at end of file
+8. **(Important !!!)** If you want to re-fill the `kylin.properties` for a `kylin` node in a specific cluster, you need to remove the `.rendered` file and re-copy the `kylin.propertie.template`. Redo steps from `3` to `6`.
\ No newline at end of file
diff --git a/readme/quick_start.md b/readme/quick_start.md
index 7aae18c..76df90e 100644
--- a/readme/quick_start.md
+++ b/readme/quick_start.md
@@ -2,9 +2,14 @@
+- **Services are created as the number order from 1 to 4.**
+- **A cluster will be easily created as same as the image of architecture above.**
+
+
+
## Quick Start
-1. Initialize aws account credential on local machine, please check [details](./prerequisites.md#localaws).
+1. Initialize aws account credential on the local machine, please check [details](./prerequisites.md#localaws).
2. Download the source code:
@@ -16,11 +21,11 @@
1. Set the `AWS_REGION`.
- 2. Set the `IAMRole`,please check [details](./prerequisites.md#IAM).
+ 2. Set the `IAMRole`, please check the [details](./prerequisites.md#IAM).
3. Set the `S3_URI`, please check [details](./prerequisites.md#S3).
- 4. Set the `KeyName`,please check [details](./prerequisites.md#keypair).
+ 4. Set the `KeyName`, please check [details](./prerequisites.md#keypair).
5. Set the `CIDR_IP`, make sure that the `CIDR_IP` match the pattern `xxx.xxx.xxx.xxx/16[|24|32]`.
@@ -28,7 +33,7 @@
>
> 1. this `CIDR_IP` is the specified IPv4 or IPv6 CIDR address range which an inbound rule can permit instances to receive traffic from.
>
- > 2. In one word, it will let your mac which ip is in the `CIDR_IP` to access instances.
+ > 2. In one word, it will let your mac which IP is in the `CIDR_IP` to access instances.
4. Init local env.
@@ -38,21 +43,26 @@ $ bin/init.sh
> Note: Following the information into a python virtual env and get the help messages.
-5. Execute commands to deploy a `default` cluster, please check [details](./prerequisites.md#cluster).
+5. Execute commands to deploy a `default` cluster.
```shell
$ python deploy.py --type deploy
```
-After `default` cluster is ready, you will see the message `Kylin Cluster already start successfully.` in the console.
+After the `default` cluster is ready, you will see the message `Kylin Cluster already start successfully.` in the console.
+
+> Note:
+>
+> 1. For details about the properties of kylin4 in a cluster, please check [configure kylin.properties](./prerequisites.md#cluster).
+> 2. For details about the index of the cluster, please check [Indexes of clusters](./prerequisites.md#indexofcluster).
-6. Execute commands to list nodes of cluster.
+6. Execute commands to list nodes of the cluster.
```shell
$ python deploy.py --type list
```
-Then you can check the `public ip` of Kylin Node.
+Then you can check the `public IP` of Kylin Node.
You can access `Kylin` web by `http://{kylin public ip}:7070/kylin`.
@@ -66,5 +76,5 @@ $ python deploy.py --type destroy
-> Note: If you want to check about quick start for multiple clusters, please referer to [quick start for mutilple clusters](./quick_start_for_multiple_clusters.md).
+> Note: If you want to check about a quick start for multiple clusters, please referer to a [quick start for multiple clusters](./quick_start_for_multiple_clusters.md).
diff --git a/readme/quick_start_for_multiple_clusters.md b/readme/quick_start_for_multiple_clusters.md
index 86ef780..1b5d50b 100644
--- a/readme/quick_start_for_multiple_clusters.md
+++ b/readme/quick_start_for_multiple_clusters.md
@@ -1,14 +1,14 @@
## Quick Start For Multiple Clusters
-> Pre-steps is same as Quick Start steps which is from 1 to 5.
+> Pre-steps are the same as Quick Start steps which are from 1 to 5.
-1. Modify the config `CLUSTER_INDEXES` for multiple cluster.
+1. Modify the config `CLUSTER_INDEXES` for multiple clusters.
> Note:
>
> 1. `CLUSTER_INDEXES` means that cluster index is in the range of `CLUSTER_INDEXES`.
- > 2. If user create multiple clusters, `default` cluster always be created. If `CLUSTER_INDEXES` is (1, 3), there will be 4 cluster which contains the cluster 1, 2, 3 and `default` will be created if user execute the commands.
- > 3. Configs for multiple clusters always are same as the `default` cluster to read from `kylin_configs.yaml`
+ > 2. If a user creates multiple clusters, the `default` cluster always is created. If `CLUSTER_INDEXES` is (1, 3), there will be 4 cluster that contains cluster 1, 2, 3, and `default` will be created if a user executes the commands.
+ > 3. Configs for multiple clusters always are the same as the `default` cluster to read from `kylin_configs.yaml`
2. Copy `kylin.properties.template` for expecting clusters to deploy, please check the [details](./prerequisites.md#cluster).
diff --git a/readme/trouble_shooting.md b/readme/trouble_shooting.md
new file mode 100644
index 0000000..4f2f053
--- /dev/null
+++ b/readme/trouble_shooting.md
@@ -0,0 +1,98 @@
+## Trouble Shooting
+
+### Cloudformation stack is normal, but services can't access normally.
+
+Q: Sometimes users will found that the stack is normal in the cloudformation dashboard, but services aren't started normally. Such as the port of related service can not access by IP.
+
+A:
+
+1. Users can login into the instance of related services by the `credential file` and the public `IP` which is in the stack `Outputs`. Access command will be like `ssh -i xxx.pem ec2-user@{public ip}` on the terminal.
+2. Change to the `root` user by executing `sudo su` on the terminal.
+3. Check the `/var/log/cloud-init-output.log` to check any error happened in the processing of deployment.
+
+
+
+### The stack of related services is normal, but services aren't started normally.
+
+#### `Kylin` starts failing.
+
+Q: `Kylin` starts failing.
+
+A:
+
+1. Users login into the `Kylin` instances by the `credential file` and the public `IP` which is in the `Kylin` stack `Outputs`. Access command will be like `ssh -i xxx.pem ec2-user@{public ip}` on the terminal.
+2. Change to the `root` user by executing `sudo su` on the terminal.
+3. Refresh variables of env by executing `source ~/.bash_profile` on the terminal.
+4. Get the Kylin home by executing `echo $KYLIN_HOME` on the terminal.
+5. Check the logs of Kylin in `$KYLIN_HOME/logs/*`.
+6. More details about `Kylin` starting, user can check the scripts in `backup/scripts/prepare-ec2-env-for-kylin4.sh`.
+
+
+
+#### `Prometheus` starts failing.
+
+Q: `Prometheus` starts failing.
+
+A:
+
+1. Users login into the `Static Services` instances by the `credential file` and the public `IP` which is in the `Static Services` stack `Outputs`. Access command will be like `ssh -i xxx.pem ec2-user@{public ip}` on the terminal.
+2. Change to the `root` user by executing `sudo su` on the terminal.
+3. Refresh variables of env by executing `source ~/.bash_profile` on the terminal.
+4. Get the `Prometheus` home by executing `echo $PROMETHEUS_HOME` on the terminal.
+5. Check the logs of `Prometheus` in `$PROMETHEUS_HOME/output.log`.
+6. More details about `Prometheus` starting, user can check the scripts in `backup/scripts/prepare-ec2-env-for-static-services.sh`.
+
+
+
+#### `Granfana` starts failing.
+
+Q: `Granfana` starts failing.
+
+A:
+
+1. Users login into the `Static Services` instances by the `credential file` and the public `IP` which is in the `Static Services` stack `Outputs`. Access command will be like `ssh -i xxx.pem ec2-user@{public ip}` on the terminal.
+2. Change to the `root` user by executing `sudo su` on the terminal.
+3. `Granfana` is started as a docker service.
+ 1. Use `docker ps -a` to get the docker id of `Granfana`.
+ 2. User `docker exec -it ${granfana docker id}` to into `Granfana` service.
+ 3. Check the logs of `Grafana` in `/var/log/grafana/grafana.log`.
+4. More details about `Granfana` starting, user can check the scripts in `backup/scripts/prepare-ec2-env-for-static-services.sh`.
+
+
+
+#### `Spark` starts failing.
+
+Q: `Spark` starts failing.
+
+A:
+
+1. Users login into the `Spark Master/Spark Slave` instances by the `credential file` and the public `IP` which is in the `Spark Master / Spark Slave` stack `Outputs`. Access command will be like `ssh -i xxx.pem ec2-user@{public ip}` on the terminal.
+2. Change to the `root` user by executing `sudo su` on the terminal.
+3. Refresh variables of env by executing `source ~/.bash_profile` on the terminal.
+4. Get the `SPARK` home by executing `echo $SPARK_HOME` on the terminal.
+5. Check the logs of `SPARK` in `$SPARK_HOME/logs/*.log`.
+6. More details about `Spark` starting, user can check the scripts in `backup/scripts/prepare-ec2-env-for-spark-*.sh`.
+
+
+
+#### `Kylin starts` failing because can not connect to the Zookeeper.
+
+Q: `Kylin starts` failing because can not connect to the Zookeeper.
+
+A:
+
+1. Users login into the `Zookeeper` instances by the `credential file` and the public `IP` which is in the Zookeeper stack `Outputs`. Access command will be like `ssh -i xxx.pem ec2-user@{public ip}` on the terminal.
+2. Change to the `root` user by executing `sudo su` on the terminal.
+3. Refresh variables of env by executing `source ~/.bash_profile` on the terminal.
+4. Get the Zookeeper home by executing `echo $ZOOKEEPER_HOME` on the terminal.
+5. Check the Zookeeper config file in `$ZOOKEEPER_HOME/conf/zoo.cfg`.
+6. Check the logs of Zookeeper in `/tmp/zookeeper/zk{1..3}/log`.
+7. Check the data-dir of Zookeeper in ``/tmp/zookeeper/zk{1..3}/data`.
+8. More details about `Zookeeper` starting, user can check the scripts in `backup/scripts/prepare-ec2-env-for-zk.sh`.
+
+
+
+# TO BE CONTINUED...
+
+
+
[kylin] 02/02: # update README.md
Posted by xx...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
xxyu pushed a commit to branch kylin4_on_cloud
in repository https://gitbox.apache.org/repos/asf/kylin.git
commit 7e59491b870c473631566b674e082ffb4ecc0442
Author: Mukvin <bo...@163.com>
AuthorDate: Wed Feb 9 12:29:12 2022 +0800
# update README.md
---
README.md | 2 +-
deploy.py | 1 +
readme/commands.md | 25 +++++++++++++------------
readme/prerequisites.md | 14 +++++++-------
readme/quick_start.md | 19 +++++++++----------
readme/quick_start_for_multiple_clusters.md | 4 ++--
readme/trouble_shooting.md | 20 ++++++++++----------
7 files changed, 43 insertions(+), 42 deletions(-)
diff --git a/README.md b/README.md
index 6847bc2..c4704e4 100644
--- a/README.md
+++ b/README.md
@@ -2,7 +2,7 @@
**Apache Kylin community** released Kylin 4.0 with a new architecture, which is dedicated to building a high-performance and low-cost OLAP engine. The architecture of Kylin 4.0 supports the separation of storage and computing, which enables Kylin users to run Kylin 4.0 by adopting a more flexible and elastically scalable cloud deployment method.
-For the best practices of Kylin4 on the cloud, **Apache Kylin community contributes a **tool** to deploy kylin4 clusters on **AWS** cloud easily and conveniently.
+For the best practices of Kylin4 on the cloud, Apache Kylin community contributes a **tool** to deploy kylin4 clusters on **AWS** cloud easily and conveniently.
# Introduction About This Tool
diff --git a/deploy.py b/deploy.py
index 3fdb5e0..6085064 100644
--- a/deploy.py
+++ b/deploy.py
@@ -52,6 +52,7 @@ def deploy_on_aws(deploy_type: str, scale_type: str, node_type: str, cluster: st
if not cluster or cluster == Cluster.DEFAULT.value:
aws_engine.destroy_default_cluster()
aws_engine.refresh_kylin_properties_in_default()
+ aws_engine.destroy_rds_and_vpc()
if cluster and cluster.isdigit():
aws_engine.destroy_cluster(cluster_num=int(cluster))
diff --git a/readme/commands.md b/readme/commands.md
index 99821ec..ab55e32 100644
--- a/readme/commands.md
+++ b/readme/commands.md
@@ -18,11 +18,12 @@ python deploy.py --type [deploy|destroy|list|scale] --scale-type [up|down] --nod
>
> 1. Current support to scale up/down `kylin` or `spark_worker` for a specific cluster.
> 2. Before scaling up/down `kylin` or `spark_worker` nodes, Cluster services must be ready.
- > 3. If you want to scale a `kylin` or `spark_worker` node to a specified cluster, please add the `--cluster ${cluster num}` to specify the expected node add to the cluster `${cluster num}`.
+ > 3. If you want to scale a `kylin` or `spark_worker` node to a specified cluster, please add the `--cluster ${cluster ID}` to specify the expected node add to the cluster `${cluster ID}`.
+ > 4. For details about the index of the cluster, please check [Indexes of clusters](./prerequisites.md#indexofcluster).
### Command for deploy
-- Deploy default cluster
+- Deploy a default cluster
```shell
$ python deploy.py --type deploy [--cluster default]
@@ -31,10 +32,10 @@ $ python deploy.py --type deploy [--cluster default]
- Deploy a cluster with a specific cluster index. <a name="deploycluster"></a>
```shell
-$ python deploy.py --type deploy --cluster ${cluster num}
+$ python deploy.py --type deploy --cluster ${cluster ID}
```
-> Note: the `${cluster num}` must be in the range of `CLUSTER_INDEXES`.
+> Note: the `${cluster ID}` must be in the range of `CLUSTER_INDEXES`.
- Deploy all clusters which contain the default cluster and all clusters whose index is in the range of `CLUSTER_INDEXES`.
@@ -48,7 +49,7 @@ $ python deploy.py --type deploy --cluster all
>
> Destroy all clusters will not delete vpc, rds, and monitor node. So if user doesn't want to hold the env, please set the `ALWAYS_DESTROY_ALL` to be `'true'`.
-- Destroy default cluster
+- Destroy a default cluster
```shell
$ python deploy.py --type destroy [--cluster default]
@@ -57,10 +58,10 @@ $ python deploy.py --type destroy [--cluster default]
- Destroy a cluster with a specific cluster index.
```shell
-$ python deploy.py --type destroy --cluster ${cluster num}
+$ python deploy.py --type destroy --cluster ${cluster ID}
```
-> Note: the `${cluster num}` must be in the range of `CLUSTER_INDEXES`.
+> Note: the `${cluster ID}` must be in the range of `CLUSTER_INDEXES`.
- Destroy all clusters which contain the default cluster and all clusters whose index is in the range of `CLUSTER_INDEXES`.
@@ -81,14 +82,14 @@ $ python deploy.py --type list
> Note:
>
> 1. Scale command must be used with `--scale-type` and `--node-type`.
-> 2. If the scale command does not specify a cluster num, then the scaled node(Kylin or spark worker) will be added to the `default` cluster.
+> 2. If the scale command does not specify a `cluster ID`, then the scaled node(Kylin or spark worker) will be added to the `default` cluster.
> 3. Scale command **not support** to **scale** node (kylin or spark worker) to **all clusters** at **one time**. It means that `python ./deploy.py --type scale --scale-type up[|down] --node-type kylin[|spark_worker] --cluster all` is invalid commad.
> 4. Scale params which are `KYLIN_SCALE_UP_NODES`, `KYLIN_SCALE_DOWN_NODES`, `SPARK_WORKER_SCALE_UP_NODES` and `SPARK_WORKER_SCALE_DOWN_NODES` effect on all cluster. So if user wants to scale a node for a specific cluster, then modify the scale params before **every run time.**
> 5. **(Important!!!)** The current cluster is created with default `3` spark workers and `1` Kylin node. The `3` spark workers can not be scaled down. The `1` Kylin node also can not be scaled down.
-> 6. **(Important!!!)** The current cluster can only scale up or down the range of nodes which is in `KYLIN_SCALE_UP_NODES`, `KYLIN_SCALE_DOWN_NODES`, `SPARK_WORKER_SCALE_UP_NODES,` and `SPARK_WORKER_SCALE_DOWN_NODES`. Not the default `3` spark workers and `1` kylin node in the cluster.
+> 6. **(Important!!!)** The current cluster can only scale up or down the range of nodes which is in `KYLIN_SCALE_UP_NODES`, `KYLIN_SCALE_DOWN_NODES`, `SPARK_WORKER_SCALE_UP_NODES,` and `SPARK_WORKER_SCALE_DOWN_NODES`. Not the default `3` spark workers and `1` kylin node in a cluster.
> 7. **(Important!!!)** If user doesn't want to create a cluster with `3` default spark workers, then user can remove the useless node module in the `Ec2InstanceOfSlave0*` of `cloudformation_templates/ec2-cluster-spark-slave.yaml`. User needs to know about the syntax of `cloudformation` as also.
-- Scale up/down Kylin/spark workers in default cluster
+- Scale up/down Kylin/spark workers in the default cluster
```shell
python deploy.py --type scale --scale-type up[|down] --node-type kylin[|spark_worker] [--cluster default]
@@ -97,7 +98,7 @@ python deploy.py --type scale --scale-type up[|down] --node-type kylin[|spark_wo
- Scale up/down kylin/spark workers in a specific cluster
```shell
-python deploy.py --type scale --scale-type up[|down] --node-type kylin[|spark_worker] --cluster ${cluster num}
+python deploy.py --type scale --scale-type up[|down] --node-type kylin[|spark_worker] --cluster ${cluster ID}
```
-> Note: the `${cluster num}` must be in the range of `CLUSTER_INDEXES`.
\ No newline at end of file
+> Note: the `${cluster ID}` must be in the range of `CLUSTER_INDEXES`.
\ No newline at end of file
diff --git a/readme/prerequisites.md b/readme/prerequisites.md
index 97ccdf9..361077d 100644
--- a/readme/prerequisites.md
+++ b/readme/prerequisites.md
@@ -75,7 +75,7 @@ git clone https://github.com/apache/kylin.git && cd kylin && git checkout kylin4
> Note:
>
-> Please download the generated CSV file of `Access Key` immediately. Get the `Access Key` and `Secret Key` to initialize local mac to access aws.
+> Please download the generated CSV file of `Access Key` immediately. Get the `Access Key` and `Secret Key` to initialize local machine to access aws.
@@ -186,9 +186,9 @@ Scripts:
### Initialize Env Of Local Machine
-#### I. Initilize an aws account on local mac to access AWS<a name="localaws"></a>
+#### I. Initilize an aws account on local machine to access AWS<a name="localaws"></a>
-> Use `Access Key` and `Secret Key` above to Initialize an AWS account on a local mac.
+> Use `Access Key` and `Secret Key` above to Initialize an AWS account on a local machine.
```shell
$ aws configure
@@ -241,7 +241,7 @@ $ bin/init.sh
> 1. User can modify the `CLUSTER_INDEXES` to be `(1, 3)`, then as following steps of this module to modify the `kylin.properties` file for clusters.
> 2. User can mark the Kylin node of the `default` cluster to be `query` mode and the Kylin node of the cluster whose index is `1` to be `query` mode. User can mark the Kylin node of the cluster which index is `2` to be `job` mode and the Kylin node of the cluster which index is `3` to job mode.
> 3. User can also modify the `CLUSTER_INDEXES` to be `(1, 4)`, and mark the Kylin node of clusters whose index is `1` and `2` to be `query` mode and the Kylin node of clusters whose index is `3` and `4` to be `job` mode. Just don't use `default` to mark a cluster and execute exactly deploy cluster commands. For details about commands, please check to [deploy multiple clusters](./Commands.md#deploycluster).
-> 4. The list of mark name for clusters will be [`default`, `{cluster num}` ...] and `{cluster num}` is in the range of `CLUSTER_INDEXES`.
+> 4. The list of mark name for clusters will be [`default`, `{cluster ID}` ...] and `{cluster ID}` is in the range of `CLUSTER_INDEXES`.
> 5. For example, if `CLUSTER_INDEXES` is (1, 3) means that tool can deploy a cluster and it can be marked as `default` or `1` or `2` or `3`. And tool can execute to deploy total 1(`default`) + 3(clusters which mark name can be `1`, `2`, `3`) = 4 clusters.
> 6. Every cluster contains 3 `Zookeepers Node`, 1 `Kylin Node`, 1 `Spark Master Node,` and 3 `Spark Slaves Node` after deployed and it can scale needed nodes of Kylin and Spark workers.
> 7. **The difference between clusters only can be the index or customized configs of EC2 instances or customized properties of Kylin (and spark and zookeeper).**
@@ -250,9 +250,9 @@ $ bin/init.sh
1. The `kylin.properties` is for starting kylin instance in the cluster.
2. The default cluster will check the `kylin.properties` in the `backup/properties/default`, and other specific clusters will check the related num directory such as `1`, `2,` and `3`.
-3. User needs to create a new dir for the cluster num in `backup/properties`, and name it to the `${cluster num}`, such as `1`, `2` ,`3` and so on.
-4. Following the `2.` step, copy the `kylin.properties.template` which is in `backup/properties/templates` to the related `${cluster num} ` directories， and rename the template to `kylin.properties`.
-5. The range of cluster nums must match the config `CLUSTER_INDEXES`, such as `CLUSTER_INDEXES: (1, 3)` then the directories must be `1`, `2`,`3` in the `backup/properties`.
+3. User needs to create a new dir for the cluster ID in `backup/properties`, and name it to the `${cluster ID}`, such as `1`, `2` ,`3` and so on.
+4. Following the step `2`, copy the `kylin.properties.template` which is in `backup/properties/templates` to the related `${cluster ID} ` directories， and rename the template to `kylin.properties`.
+5. The range of cluster IDs must match the config `CLUSTER_INDEXES`, such as `CLUSTER_INDEXES: (1, 3)` then the directories must be `1`, `2`,`3` in the `backup/properties`.
diff --git a/readme/quick_start.md b/readme/quick_start.md
index 76df90e..3c44d55 100644
--- a/readme/quick_start.md
+++ b/readme/quick_start.md
@@ -3,8 +3,6 @@
- **Services are created as the number order from 1 to 4.**
-- **A cluster will be easily created as same as the image of architecture above.**
-
## Quick Start
@@ -43,18 +41,18 @@ $ bin/init.sh
> Note: Following the information into a python virtual env and get the help messages.
-5. Execute commands to deploy a `default` cluster.
+5. Execute commands to deploy a cluster quickly.
```shell
$ python deploy.py --type deploy
```
-After the `default` cluster is ready, you will see the message `Kylin Cluster already start successfully.` in the console.
+After this cluster is ready, you will see the message `Kylin Cluster already start successfully.` in the console.
> Note:
>
-> 1. For details about the properties of kylin4 in a cluster, please check [configure kylin.properties](./prerequisites.md#cluster).
-> 2. For details about the index of the cluster, please check [Indexes of clusters](./prerequisites.md#indexofcluster).
+> 1. For more details about the properties of kylin4 in a cluster, please check [configure kylin.properties](./prerequisites.md#cluster).
+> 2. For more details about the index of the clusters, please check [Indexes of clusters](./prerequisites.md#indexofcluster).
6. Execute commands to list nodes of the cluster.
@@ -68,13 +66,14 @@ You can access `Kylin` web by `http://{kylin public ip}:7070/kylin`.
-7. Destroy the `default` cluster.
+7. Destroy the cluster quickly.
```shell
$ python deploy.py --type destroy
```
-
-
-> Note: If you want to check about a quick start for multiple clusters, please referer to a [quick start for multiple clusters](./quick_start_for_multiple_clusters.md).
+> Note:
+>
+> 1. If you want to check about a quick start for multiple clusters, please referer to a [quick start for multiple clusters](./quick_start_for_multiple_clusters.md).
+> 2. **Current destroy operation will remain some stack which contains `RDS` and so on**. So if user want to destroy clearly, please modify the `ALWAYS_DESTROY_ALL` in `kylin_configs.yml` to be `true` and re-execute `destroy` command.
diff --git a/readme/quick_start_for_multiple_clusters.md b/readme/quick_start_for_multiple_clusters.md
index 1b5d50b..14ee9a7 100644
--- a/readme/quick_start_for_multiple_clusters.md
+++ b/readme/quick_start_for_multiple_clusters.md
@@ -7,8 +7,8 @@
> Note:
>
> 1. `CLUSTER_INDEXES` means that cluster index is in the range of `CLUSTER_INDEXES`.
- > 2. If a user creates multiple clusters, the `default` cluster always is created. If `CLUSTER_INDEXES` is (1, 3), there will be 4 cluster that contains cluster 1, 2, 3, and `default` will be created if a user executes the commands.
- > 3. Configs for multiple clusters always are the same as the `default` cluster to read from `kylin_configs.yaml`
+ > 2. Configs for multiple clusters are also from `kylin_configs.yaml`.
+ > 3. For more details about the index of the clusters, please check [Indexes of clusters](./prerequisites.md#indexofcluster).
2. Copy `kylin.properties.template` for expecting clusters to deploy, please check the [details](./prerequisites.md#cluster).
diff --git a/readme/trouble_shooting.md b/readme/trouble_shooting.md
index 4f2f053..1cc3d2c 100644
--- a/readme/trouble_shooting.md
+++ b/readme/trouble_shooting.md
@@ -14,9 +14,9 @@ A:
### The stack of related services is normal, but services aren't started normally.
-#### `Kylin` starts failing.
+#### `Kylin` web can not access.
-Q: `Kylin` starts failing.
+Q: `Kylin` web can not access.
A:
@@ -29,9 +29,9 @@ A:
-#### `Prometheus` starts failing.
+#### `Prometheus` can not access.
-Q: `Prometheus` starts failing.
+Q:`Prometheus` can not access.
A:
@@ -44,9 +44,9 @@ A:
-#### `Granfana` starts failing.
+#### `Granfana` can not access.
-Q: `Granfana` starts failing.
+Q: `Granfana` can not access.
A:
@@ -60,9 +60,9 @@ A:
-#### `Spark` starts failing.
+#### `Spark` can not access.
-Q: `Spark` starts failing.
+Q: `Spark` can not access.
A:
@@ -75,9 +75,9 @@ A:
-#### `Kylin starts` failing because can not connect to the Zookeeper.
+#### `Kylin` can not access and exception "Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect." is in kylin.log.
-Q: `Kylin starts` failing because can not connect to the Zookeeper.
+Q: `Kylin` can not access and exception "Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect." is in kylin.log.
A: