You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by xi...@apache.org on 2019/11/05 06:51:19 UTC
[incubator-pinot] 01/01: Adding example for aws
This is an automated email from the ASF dual-hosted git repository.
xiangfu pushed a commit to branch aws-quickstart
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git
commit 84c76dafb4a34dd07ed952ee90d72075c9502a4f
Author: Xiang Fu <fx...@gmail.com>
AuthorDate: Mon Nov 4 21:02:16 2019 -0800
Adding example for aws
---
kubernetes/examples/helm/README.md | 81 ++++++++++++++++++++--
.../examples/helm/pinot-realtime-quickstart.yml | 36 ++++++++--
2 files changed, 108 insertions(+), 9 deletions(-)
diff --git a/kubernetes/examples/helm/README.md b/kubernetes/examples/helm/README.md
index 2c8c280..f1648cf 100644
--- a/kubernetes/examples/helm/README.md
+++ b/kubernetes/examples/helm/README.md
@@ -28,6 +28,69 @@
- Configure kubectl to connect to the Kubernetes cluster.
- Skip to [Section: How to setup a Pinot cluster for demo](#How to setup a Pinot cluster for demo) if a k8s cluster is already setup.
+
+## (Optional) Setup a Kubernetes cluster on Amazon Elastic Kubernetes Service (Amazon EKS)
+
+### (Optional) Create a new k8s cluster on AWS EKS
+
+- Install AWS CLI (<https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-install.html#install-tool-bundled>)
+- Install AWS-IAM-AUTHENTICATOR (<https://docs.aws.amazon.com/eks/latest/userguide/install-aws-iam-authenticator.html>)
+- Install eksctl (<https://docs.aws.amazon.com/eks/latest/userguide/eksctl.html#installing-eksctl>)
+
+- Login to your AWS account.
+
+```bash
+aws configure
+```
+
+Note that environment variables `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` will override the aws configuration in file `~/.aws/credentials`.
+
+- Amazon EKS Prerequisites
+
+It requires Role ARN, VPC, Subnets, Security Groups to create an EKS cluster.
+
+Please follow wiki [Amazon EKS Prerequisites](https://docs.aws.amazon.com/eks/latest/userguide/getting-started-console.html#eks-prereqs) to
+create Amazon Resource Name (ARN) and Amazon EKS Cluster VPC
+
+- Create an EKS cluster
+
+Please modify the parameters in the example command below:
+
+```bash
+eksctl create cluster \
+--name pinot-quickstart \
+--version 1.14 \
+--region us-west-2 \
+--nodegroup-name standard-workers \
+--node-type t3.small \
+--nodes 3 \
+--nodes-min 3 \
+--nodes-max 4 \
+--node-ami auto
+```
+You can monitor cluster status by command:
+
+```bash
+EKS_CLUSTER_NAME=pinot-quickstart
+aws eks describe-cluster --name ${EKS_CLUSTER_NAME}
+```
+
+Once the cluster is in `ACTIVE` status, it's ready to be used.
+
+### (Optional) How to connect to an existing cluster
+
+Simply run below command to get the credential for the cluster you just created or your existing cluster.
+
+```bash
+EKS_CLUSTER_NAME=pinot-quickstart
+aws eks update-kubeconfig --name ${EKS_CLUSTER_NAME}
+```
+
+To verify the connection, you can run
+```bash
+kubectl get nodes
+```
+
## (Optional) Setup a Kubernetes cluster on Google Kubernetes Engine(GKE)
### (Optional) Create a new k8s cluster on GKE
@@ -129,18 +192,26 @@ helm dependency update
### Start Pinot with Helm
+If cluster is just initialized, ensure helm is initialized by running:
+
+```bash
+helm init --service-account tiller
+```
+
+Then deploy pinot cluster by:
+
```bash
helm install --namespace "pinot-quickstart" --name "pinot" .
```
#### Troubleshooting
-Error: Please run below command if encountering issue:
+- Error: Please run below command if encountering issue:
```
Error: could not find tiller".
```
-Resolution:
+- Resolution:
```bash
kubectl -n kube-system delete deployment tiller-deploy
@@ -148,11 +219,11 @@ kubectl -n kube-system delete service/tiller-deploy
helm init --service-account tiller
```
-Error: Please run below command if encountering permission issue:
+- Error: Please run below command if encountering permission issue:
```Error: release pinot failed: namespaces "pinot-quickstart" is forbidden: User "system:serviceaccount:kube-system:default" cannot get resource "namespaces" in API group "" in the namespace "pinot-quickstart"```
-Resolution:
+- Resolution:
```bash
kubectl apply -f helm-rbac.yaml
@@ -196,7 +267,7 @@ Please use below script to do local port-forwarding and open Pinot query console
### How to clean up Pinot deployment
```bash
-kubectl delete -f pinot-example-loader.yml
+kubectl delete -f pinot-realtime-quickstart.yml
helm del --purge kafka
helm del --purge pinot
```
diff --git a/kubernetes/examples/helm/pinot-realtime-quickstart.yml b/kubernetes/examples/helm/pinot-realtime-quickstart.yml
index 7b47bb2..a121b11 100644
--- a/kubernetes/examples/helm/pinot-realtime-quickstart.yml
+++ b/kubernetes/examples/helm/pinot-realtime-quickstart.yml
@@ -414,21 +414,34 @@ data:
apiVersion: batch/v1
kind: Job
metadata:
- name: pinot-realtime-quickstart
+ name: pinot-realtime-quickstart-pinot-schema-creation
namespace: pinot-quickstart
spec:
template:
spec:
containers:
- - name: loading-data-to-kafka
- image: winedepot/pinot:0.1.13-SNAPSHOT
- args: [ "StreamAvroIntoKafka", "-avroFile", "sample_data/airlineStats_data.avro", "-kafkaTopic", "flights-realtime", "-kafkaBrokerList", "kafka:9092", "-zkAddress", "kafka-zookeeper:2181" ]
- name: pinot-add-example-schema
image: winedepot/pinot:0.1.13-SNAPSHOT
args: [ "AddSchema", "-schemaFile", "/var/pinot/examples/airlineStats_schema.json", "-controllerHost", "pinot-controller", "-controllerPort", "9000", "-exec" ]
volumeMounts:
- name: examples
mountPath: /var/pinot/examples
+ restartPolicy: OnFailure
+ volumes:
+ - name: examples
+ configMap:
+ name: examples
+ backoffLimit: 100
+---
+apiVersion: batch/v1
+kind: Job
+metadata:
+ name: pinot-realtime-quickstart-pinot-table-creation
+ namespace: pinot-quickstart
+spec:
+ template:
+ spec:
+ containers:
- name: pinot-add-example-realtime-table
image: winedepot/pinot:0.1.13-SNAPSHOT
args: [ "AddTable", "-filePath", "/var/pinot/examples/airlineStats_realtime_table_config.json", "-controllerHost", "pinot-controller", "-controllerPort", "9000", "-exec" ]
@@ -440,6 +453,21 @@ spec:
- name: examples
configMap:
name: examples
+ backoffLimit: 100
+---
+apiVersion: batch/v1
+kind: Job
+metadata:
+ name: pinot-realtime-quickstart-load-data-into-kafka
+ namespace: pinot-quickstart
+spec:
+ template:
+ spec:
+ containers:
+ - name: loading-data-to-kafka
+ image: winedepot/pinot:0.1.13-SNAPSHOT
+ args: [ "StreamAvroIntoKafka", "-avroFile", "sample_data/airlineStats_data.avro", "-kafkaTopic", "flights-realtime", "-kafkaBrokerList", "kafka:9092", "-zkAddress", "kafka-zookeeper:2181" ]
+ restartPolicy: OnFailure
backoffLimit: 3
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org